Time Series as Images: Vision Transformer for Irregularly Sampled Time Series
Authors: Zekun Li
Shiyang Li
Xifeng Yan
Venue: NeurIPS 2023 poster (2023)
Keywords: irregularly sampled time series
vision transformer
healthcare
time series classification
TL;DR: We introduce a new perspective for irregularly sampled time series modeling by transforming time series data into line graph images and adapting vision transformers for time series modeling.

Abstract: Irregularly sampled time series are increasingly prevalent, particularly in medical domains. While various specialized methods have been developed to handle these irregularities, effectively modeling their complex dynamics and pronounced sparsity remains a challenge. 
This paper introduces a novel perspective by converting irregularly sampled time series into line graph images, then utilizing powerful pre-trained vision transformers for time series classification in the same way as image classification. This method not only largely simplifies specialized algorithm designs but also presents the potential to serve as a universal framework for time series modeling. Remarkably, despite its simplicity, our approach outperforms state-of-the-art specialized algorithms on several popular healthcare and human activity datasets. Especially in the rigorous leave-sensors-out setting where a portion of variables is omitted during testing, our method exhibits strong robustness against varying degrees of missing observations, achieving an impressive improvement of 42.8% in absolute F1 score points over leading specialized baselines even with half the variables masked. Code and data are available at https://github.com/Leezekun/ViTST.

Time Series as Images: Vision Transformer for Irregularly Sampled Time Series
Zekun Li, Shiyang Li, Xifeng Yan University of California, Santa Barbara (zekunli, shiyangli, xyan) @cs. ucsb. edu
Abstract
Irregularly sampled time series are increasingly prevalent, particularly medical domains. While various specialized methods have been developed to handle these irregularities, effectively modeling their complex dynamics and pronounced spar- sity remains challenge. This paper introduces novel perspective by converting irregularly sampled time series into line graph images, then utilizing powerful pre- trained vision transformers for time series classification in the same way as image classification. This method not only largely simplifies specialized algorithm designs but also presents the potential to serve universal framework for time series mod- eling. Remarkably, despite its simplicity, our approach outperforms statc-of-the of-the-art specialized algorithms several popular healthcare and human activity datasets. Especially in the rigorous ceave-sensors-out setting where a portion of variables is omitted during testing, our method exhibits strong robustness against varying degrees of missing observations, achieving an impressive improvement 42.8% absolute Fl score points over leading specialized baselines even with half the vari- ViTST
ables masked. Code and data are available at httpp://gituub..om/l.eee
Introduction
Time series data ubiquitous in wide range of domains, including healthcare, finance, traffic, and climate science. With the advances in deep learning architectures such as LSTM [13), Temporal Convolutional Network (TCN) (18), and Transformer (38), numerous algorithms have been developed for time series modeling. However, these methods typically assume fully observed data at regular intervals and fixed-size numerical inputs. Consequently, these methods encounter difficulties when faced with irregularly sampled time series, which consist of a sequence of samples with irregular intervals between their observation times. To address this challenge, highly specialized models have been developed, which require substantial prior knowledge and efforts in model architecture selection and algorithm design 24 20 3
In parallel. pre- treitrd transformer- based vision models, most notably vision transformers have emerged and demonstrated strong abilities in various vision tasks such image classification and object detection, nearly approaching buman-level performance. Motivated by the flexible and effective manner in which humans analyze complex numerical time series data through visualization, raise the question: Can these powerful pre-trained vision transformers capture temporal patterns in visualized time series data, similar how humans do?
To investigate this question, we propose a minimalist approach called VITST (Vision Time Series Transformer), which involves transforming irregularly sampled multivariate time series into line graphs organizing them into a standard RGB image format, and finetuning a pre-trained vision
In this paper, we use the term "vision transformers" to denote category of pre-trained transformer-basel vision models, such ViT 9 Swin Transformer [22). DeiT (36] etc.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Multivariate Series
Time
00:30
00:34
02:13
03:25
Prediction Head
HCT
PeC02
SWABB
Value
155.0
23.1
33.0
142.0
Vislon Transformer Encoder (e.g. Swin)
with Aojun
PsDO2
Systap
lICT
Ww
An illustration of our approach ViTST. The example is from healthcare dataset P12 [12. which provides the irregularly sampled observations of 36 variables for patients (we only show variables here for simplicity). Each column in the table an observation of variable, with the observed time and value. We plot separate line graphs for each variable and arrange them into single image, which is then fed into the vision transformer for classification.
Figure I:
transformer for classification using the resulting image as input. An illustration of our method can be found in Figure !
Line graphs serve as an effective and efficient visualization technique for time series data, regardless of imegularity, structures, and scales. They can capture crucial patterns, such as the temporal dynamics represented within individual line graphs and interrelations between variables throughout separate graphs. Such visualization technique benefits approach because it is both simple and intuitively comprehensible to humans, enabling straightforward decisions for time series-to-image transformations. Leveraging vision models for time series modeling in this manner mirrors the concept of prompt engineering, wherein individuals can intuitively comprehend and craft prompts potentially enhance the processing efficiency of language models.
We conduct a comprehensive investigation and validation of the proposed approach, VITST, which has demonstrated its superior performance for irregularly sampled time series. Specifically, ViTST exceeded prior SoTA by 2.2% and 0.7% in absolute AUROC points, and 1.3% and 2.9% in absolute AUPRC points for healthcare datasets P19 [29] and P12 [12), respectively. For the human activity dataset, PAM (28). we observed improvements of 7.3% in accuracy, 6.3% in precision, 6.2% in recall, and 6.7% in FI score (absolute points) over existing SoTA methods. Our approach also exhibits strong robustness to missing observations, surpassing previous leading solutions by notable 42.8% in absolute Fl score points under the leave-sensors-out scenario, where half of the variables in the test set are masked during testing. Furthermore, when tested on regular time series data including those with varying numbers of variables and extended sequence lengths, VITST still achieves excellent results comparable to the specialized algorithms designed for regular time series modeling. This underscores the versatility of our approach, as traditional methods designed for regularly sampled time series often struggle with irregularly sampled data, and vice versa.
ofte-off-the-rr (SoTA) methods specifically designed
In summary, the contributions of this work are three-fold: (1) We propose a simple yet highly effective approach for multivariate irregularly sampled time series classification. Despite its sim- plicity, our approach achieves strong results against the highly specialized SoTA methods. (2) Our approach demonstrates excellent results on both irregular and regular time series data, showcasing its versatility and potential as ceneral-purposs framework for time series modeling. It offers a robust solution capable of handling diverse time series datasets with varying characteristics. (3) Our work demonstrates the successful transfer of knowledge from vision transformers pre-trained on natural images to synthetic visualized time series line graph images. We anticipate that this will facilitate the utilization of fast-evolving and well-studied computer vision techniques in the time series domains, such better model architecture (33, data augmentation 32] interpretability 4 and self- supervised pre-training [15].
Related work
Irregularly sampled time series. An irregularly sampled time series is a sequence of observations with varying time intervals between them. In multivariate setting, different variables within the

---

same time series may not align. These characteristics present significant challenge to standard time series modeling methods, which usually assume fully observed and regularly sampled data points. A common approach to handling irregular sampling is to convert continuous-time observations into fixed time intervals [4 20]. To account for the dynamies between observations, several models have been proposed, such as GRU-D (3, which decays the hidden states based on gated recurrent units (GRU) [6]. Similarly, [46] proposed an approach based on multi-directional RNN, which can capture the inter- and intra-steam patterns. Besides the recurrent and differential equation- based model architectures, recent work has explored attention-base models. Transformer [38] is naturally able to handle arbitrary sequences of observations. ATTAIN [50] incorporates attention mechanism with LSTM to model time irregularity. SeFT [16] maps the irregular time series into set of observations based on differentiable set functions and utilizes an attention mechanism for classification. mTTAND 34 learns continuous- embeddings coupled with a multi-time attention mechanism deal with continuous-time inputs. UTDE [88] integrates embeddings from mTAND and classical imputed time series with learnable gates to tackle complex temporal patterns. Raindrop [49] models irregularly sampled time series as graphs and utilizes graph neural networks to model relationships between variables. While these methods are specialized for irregular time series, our work explores simple and general vision transformer-basee approach for irregularly sampled time series modeling.
Numerical time series modeling with transformers. Transformers have gained significant attention in time series modeling due to their exceptional ability to capture long-range dependencies in sequential data. A surge of seassfommrrbbseed methods have been proposed and successfully applied to various time series modeling tasks, such as forecasting [9 51 41 52 classification [47]. and anomaly detection 45). However, these methods are usually designed for regular time series settings, where multivariate numerical values at the same timestamp are viewed as a unit, and temporal interactions across different units are explicitly modeled. A recent work (25) suggests dividing each univariate time series into a sequence of sub-series and modeling their interactions independently. These methods all operate on numerical values and assume fully observed input, while our proposed method deals with time series data in the visual modality. By transforming the time series into visualized line graphs, we can effectively handle irregularly sampled time series and harness the strong visual representation learning abilities of pre-trained vision transformers.
Imaging time series. Previous studies have explored transforming time series data into different types of images, such as Gramian fields [3) recurring plots 14 37). and Markov transition fields [40]. These approaches typically employ convolutional neural networks (CNNs) for classification tasks. However, they often require domain expertise to design specialized imaging techniques and are not universally applicable across domains. Another approach (35) involves utilizing convolutional autoencoders to complete images transformed from time series, specifically for forecasting purposes. Similarly. [1] utilized CNNs to encode images converted from time series and use regressor for numerical forecasting. These approaches, however, still need plenty of specialized designs and modifications to adapt to time series modeling. Furthermore. they still lag behind the current leading numerical techniques. In contrast, our proposed method leverages the strong abilities of pre-trained vision transformer to achieve superior results by transforming the time series into line graph images, sidestepping the need of prior knowledge and specific modifications and designs.
3 Approach
As illustrated in Fig. D ViTST consists of two main steps: (1) transforming multivariate time series into a concatenated line graph image, and (2) utilizing a pre-trained vision transformer as an image classifier for the classification task. To begin with, we present some basic notations and problem formulation.
Notation. Let D ((St,
= 1, ;=)li= Every data sample is associated with a label yi E {I..... .C), where C is the number of classes. Each multivariate time series S, consists of observations of D variables at most (some might have no observations). The observations for each variable d are given by a sequence of tuples with observed )}, where n the number of observations for variable
N) denote a time series dataset containing N samples.
time and value [(44 u4), (tg, ug),-- ... (th., vd_)], d If the intervals between observation times [1g, 18. _d__] are different across variables or samples, S, is an irregularly sampled time series. Otherwise, it is regular time series.
3

---

Problem formulation. Given the dataset D = {(S,, - .Ny containing N multivariate time series, we aim to predict the label D. E {1, ... , C}} for each time series S, There are mainly two components in our framework: (1) function that transforms the time series S; into an image x;; (2) an image classifier that takes the line graph image x, as input and predicts the label ys-
3.1
Time Series to Image Transformation
Time series line graph. The line graph is a prevalent method for visualizing temporal data points. In this representation, each point signifies an observation marked by its time and value: the horizontal axis captures timestamps, and the vertical axis denotes values. Observations are connected with straight lines in chronological order, with any missing values interpolated seamlessly. This graphing approach allows for flexibility for users in plotting time series as images, intuitively suited for the processing efficiency of vision transformers. The practice mirrors prompt engineering when using language models, where users can understand and adjust natural language prompts to enhance the model performance.
In our practice, we use marker symbols "*" to indicate the observed data points in the line graph. Since the scales of different variables may vary significantly, we plot the observations of each variable in an individual line graph, as shown in Fig. T The scales of each line graph gi.d are kept the same across different time series S,. We employ distnnet colors for each line graph for differentiation. Our initial experiments indicated that tick labels and other graphical components are superfluous, as an observation's 's position inherently signals its relative time and value magnitude. We investigated the influences of different choices time series-to-image transformation in Section 4.3 Image Creation. Given a set of time series line graphs G: 82. "., Eg for a time series S,, we arrange them in a single image x; using pre-defined grid layout. We adopt a square grid by default, following [10]. Specifically, we arrange the D time series line graphs in a grid of size I x / if I 1x (L-1) 1) D < X I, and grid of size 1 x (I + 1) if < < (I + 1). For example, the P19, P12, and PAM datasets contain 34, 36, and 17 variables, respectively, and the corresponding default grid layouts are 6 x 6, 6 6, and 4 5. Any grid spaces not occupied by line graph remain empty. Figure @arreeessoocsee examples of the resulting images. As for the order of variables, we sort them according to the missing ratios for irregularly sampled time series. We explored the effects of different grid layouts and variable orders in Section 4.3
3.2 Vision Transformers for Time Series Modeling
Given the image x; transformed from time series S, we leverage an image classifier to perceive the image and perform the classification task. The time series patterns a line graph image involve both local (i.e., the temporal dynamics of a single variable in line graph) and global (the correlation among variables across different line graphs) contexts. To better capture these patterns, we choose the recently developed vision transformers. Unlike the predominant CNNs, vision transformers are proven to excel at maintaining spatial information and have stronger abilitics to capture local and global dependencies 99, 22).
originally adapted from NLP. The input image is split into fixed-sized patches, and each patch is linearly embedded and augmented with position embed- dings. The resulting sequence of vectors then fed into a standard Transformer encoder consisting of stack of multi-head attention modules (MSA) and MLP to obtain patch representations. An extra classification token is added to the sequence to perform classification or other tasks. ViT models global inter-unit interactions between all pairs of patches, which can be computationally expensive for high-resolution images. Swin Transformer, on the other hand, adopts a hierarchical architccture with multi-level feature maps and performs self-attention locally within non-overlapping windows, reducing computational complexity while improving performance. We use Swin Transformer as the default backbone vision model unless otherwise specified, but any other vision model can also be used within this framework.
Preliminary. Vision Transformer (ViT)
Swin Transformer captures the local and global information by constructing a hierarchical rep- resentation starting from small-sized patches earlier layers and gradually merging neighboring patches in deeper layers. Specifically, in the W-MSA block, self-attention is calculated within each non-overlapping window, allowing for the capture of local intra-variable interactions and temporal dynamics of a single line graph for variable d. The shifted window block SW-MSA then enables

---

connections between different windows, which span across different line graphs, to capture global interactions. Fig. iillustrates this process. Mathematically, the consecutive Swin Transformer blocks are calculated as:
z' W-MSA (LN (z'-1)
2'-1
z' MLP (LN (2) + +2',
2'1 SW-MSA (LN (z') z
1+1 = MLP (LN (z"+)) + 741
(I)
where z' and z' denote the output features of the (S)W-MSA module and the MLP module for block 1, respectively; IN stands for the layer normaliza- tion [. After multiple stages of blocks, the global interactions among patches from all line graphs can be captured, enabling the modeling of correlations be- tween different variables. We have also explored the use of additional positional embeddings, including local positional embeddings to indicate the position of each patch within its corresponding line graph, and global positional embeddings to represent the index of the associated line graph in the entire image. How- ever, we didn't It observe consistent improvement over the already highly competitive performance, which might suggest that the original pre-trained positional embeddings have already been able to capture the in- formation regarding local and global patch positions.
pu mongy
ull
ehilled
Figure Illustration of the shifted window approach in Swin Transformer. Self-attention is calculated within each window (grey box). When the window is contained within single line graph, it captures local interactions. After shifting, the window includes patches from different line graphs, allowing for the model- ing of global cross- vross-variable interactions.
Inference. We use vision transformers predict the labels of time series in the same way as image classification. The outputs of Swin Transformer blocks at the final stage are used as the patch representations, upon which a flattened layer with linear head is applied to obtain the prediction ÿ:-
4 Experiments
4.1
Experimental Setup
Table I: Statistics of the irregularly sampled time series datasets used in our experiments.
Datasets
#Samples
#Variables
#Avg. obs.
#Classes
Demographic info
Imbalanced
Missing ratio
P19
38,803
401
2
True
True
94.9%
P12
11,988
233
2
True
True
88.4%
PAM
5.333
17
4,048
8
False
False 60.0%
Datasets and metrics. We conducted experiments using three widely used healthcare and human activity datasets, as outlined in Table [] The P19 dataset [29] contains information from 38,80s patients, with 34 sensor variables and a binary label indicating sepsis. The P12 dataset [12] comprises data from 11,988 patients, including 36 sensor variables and binary label indicating survival during hospitalization. Lastly, the PAM dataset (28) includes 5.333 samples from distinct human activities, with 17 sensor variables provided for each sample. We used the processed data provided by Raindrop [49]. To ensure consistency, we employed identical data splits across all comparison baselines, and evaluated the performance using standard metrics such Area Under ROC Curve (AUROC) and Area Under Precision- Reccall Curve (AUPRC) for the imbalanced P P22 and P19 datasets. For the more balanced PAM dataset, we reported Accuracy, Precision, Recall, and Fl score.
Implementation. We utilized the Matplotlib package to plot line graphs and save them as standard RGB images. For the P19, P12, and PAM datasets, we implemented grid layouts of 6 x 6, 6 x 6. and x 5, respectively. For a fair comparison, we assigned a fixed size of 64 x 64 to each grid cell (line graph), resulting in image sizes of 384 x 384, 384 x 384, and 256 x 320, respectively. It is important to note that image sizes can also be directly set to any size, irrespective of grid cell dimensions. We plot the images according to the value scales on training sets. We employed the checkpoint of Swin
5

---

Table 2: Comparison with the baseline methods irregularly sampled time series classification task. Bold indicates the best performer, while underline represents the second best.
P19
P12
FAM
Methods
AUROC
AUPRC
AUROC
AUPRC
Accuracy
Precision
Recall
Fl
Transformer
80.7
42.7 1.7
83.3
47.9
83.5
84.8
86.0
85.0
Trans-mean
83.7 +
3.2
2.0
4..0
83.7 2.3
61.922.
8..1=2.
85. 2.4
GRU-D
83.9
26.9
21.9
46.1
83.3116
8.
84.8
SefT
81.2 2.2
4..
73.9 1.5
31.1144
67.1 122
2.4
68.3 1.8
mTAND
84.4
50.6120
84.2
48.2
74.6
74.3
79.5
76.8
IP-Net
84.6 1.3
38. 137
82.6114
47.6
74.3 3.8
75.6121
77.9
76.6 2.8
DGM
82.4+28
MIGNN
6.2
8.9
7.77
3..5
83
85.2
86.1=1.0
85.9
Raindrop
97.0+23
82.8
44.0+3.0
+.6 88.3
89.9
89.9
89.8
VITST
89.21
53.1.134
8..1100
41.1
1:
1
96.5
Transformer pre-trained on the ImageNet-21K, datase The default patch size and window size are 4 and 7, respectively.
Training. Given the highly imbalanced nature of the P12 and P19 datasets, we employed upsampling of the minority class to match the size of the majority class. We finetuned the Swin Transformer for 2 and 4 epochs on the upsampled P19 and P12 datasets. respectively, and for 20 epochs on the PAM dataset. The batch sizes used for training were 48 for P19 and P12, and 72 for PAM, while the learning rate was set to 2e-5. The models were trained using A6000 GPUs with 48G memory.
Incorporating static features. In real-world applications, especially in the healtheare domain, irregular time series data often accompanies additional information such as categorical or textual features. In the P12 and P19 datasets, each patient's demographic information, including weight, height, and ICU type, is provided. This static information remains constant over time and can be expressed using natural language. To incorporate this information into our framework, employed a template to convert it into natural language sentences and subsequently encoded the resulting text using a RoBERTa-base (2T) text encoder. The resulting text embeddings were concatenated with the image embeddings obtained from the vision transformer to perform classification. The static features were also applied to all compared baselines.
4.2 Main Results
Comparison to state-of-the-art. We compare our approach with several state-of-the-art methods that specifically designed for irregularly sampled time series, including Transformer (38), Trans-mean (Transformer with an imputation method that replaces the missing value with the average observed value of the variable), GRU-D 3 SeFT [16,, mTTAND 34) IP-Net (33, and Raindrop [49]. In
addition, we also compared our method with two approaches initially designed for forecasting tasks, namely DGM 42) and MTGNN 4K). The implementation and hyperparameter settings of these baselines were kept consistent with those used in Raindrop [49]. Specifically, batch size of 128 was employed, and all compared models were trained for 20 epochs. To ensure fairness in our evaluation. we averaged the performance of each method over 5 data splits that were kept consistent across all compared approaches.
As shown in Table our proposed approach demonstrates strong performance against the specialized tfate-of-the-an algorithms all three datasets. Specifically, the P19 and P12 datasets, ViTST achieves improvements of 2.2% and 0.7% in absolute AUROC points, and 1.3% and 2.9% in absolute AUPRC points over the state-of-the-art results, respectively. For the PAM dataset, the improvement is even more significant, with an increase of 7.3% in Accuracy, 6.3% in Precision, 6.2% in Recall, and 6.7% absolute Fl score points.
Leaving-sensorseee settings. We conducted additional evaluations to assess the performance of our model under more challenging scenarios, where the observations of a subset of sensors (variables) were masked during testing. This setting simulates real-world scenarios when some sensors fail or become unreachable. Following the approach adopted in [49]. we experimented with two setups using the PAM dataset: (1) feave-fixed-seeson
which drops a fixed set of sensors across all

---

Transformers
Accurily (50)
Recall (N)
(9g
Trans-mean
Procaion
SeFT
Raindrop
20
30
20
30
20
50
20
Missing (9))
Missing (9))
Missing (96)
Missing ratio
(a) Leave-fixed sensors-out
seave-fied--ensooss-oul
Transformers
Acturssy (96)
(96)
scar (86)
Trans-mean
Predsion
GRU-D
SeFT
mTAND
Raindrop
VITST
30 Micsing ng (%)
30
20
30
20
Missing (14)
Missing (9)
Missing
(b) Leave-aaddom--seesss--oo
Figure 3: Performance in leave-fixel fixed-sensors-out and saweeerrndom--sssoooee settings PAM dataset. The x-axis is the "missing ratio" which denotes the ratio of masked variables.
samples and compared methods, and (2) leave-random weaee-rrandddnssooooeeee sensors-out, which drops sensors randomly. is important to note that only the observations in the validation and test sets were dropped, while the training set was kept unchanged. To ensure a fair comparison, we dropped the same set of sensors the leave-fixed-- www.-ddeeesoos-ouuu setting as 449
The results are presented in Fig, from which we observe that our approach consistently achieves the best performance and outperforms all the specialized baselines by large margin. With the missing ratio increasing from 10% to 50%, our approach maintains strong performance, staying above 80%. contrast, the comparison baseline shows marked drop. The advantage of ViTST over the comparison baselines becomes increasingly significant. Even when half of the variables were dropped, our approach was still able achieve acceptable performance over 80, surpassing the best-perfone hest-performing baseline Raindrop by 33.1 1% in Accuracy, 40.9% in Precision, 39.4 39.4% in Recall, and 42.8% F1 score in the leave-fixed sensorssos setting (all absolute points). We also notice that the variances in our results are notably lower compared to the baselines. These results suggest that our approach is highly robust to varying degrees of missing observations in time series.
4.3
Additional Analysis
90
100
90
ResNet
RanNe:
VI
80
80
70
Swin scratch
70
Swin scretch
80
60
VIT
50
60
Swin
50
40
40
40
30
AUROC
AUPRC
20
AUROC
20
Precision
Recall
Score
(a) P19
(b) P12
(c) PAM
Figure 4: Performance of different backbone vision models on P19, P12, and PAM datasets. We do not use static features for our approach here to exclude their influence.
Where does the performance come from? Our approach transforms time series into line graph images, allowing the use of vision transformers for time series modeling. hypothesize that vision transformers can leverage their general-purpose image recognition ability acquired from large-scale pre-training on natural images (such as ImageNet (30) capture informative patterns in the line graph images. To validate that, compared the performance of pre-trained Swin

---

Table 3: Ablation studies on different strategies of time series-to-image transformation.
P19
P12
PAM
Methods
AUROC
AUPRC
AUROC
AUPRC
Accuracy
Precision
Recall
FI score
Delault
89.2 2.0
$3.1 3.4
85.1 +
51.1+
96.2 +
96.2 1.3
96.5 1.2
w/o interpolation
89.6 2.1
52.9
85.7
11.9
95.6 1.1
96.6 0.9
1.10
96.2
wio markers
89.0 2.1
51.7=2.
0..
3..
95.0 95.8 1.1
0.7
96.0
96.4 0.9
wio colors
88.8 1
1..41.4
0.7
47.0.22
96.2
9
95.7 1.99
wlo
2.3
52.7
47.8
Transformer with Swin Transformer trained from scratch, as shown in Fig. 4 The significant drop in performance without pre-training provides evidence that Swin transformer could transfer the knowledge obtained from pre-training on natural images to our synthetic time series line graph images, achieving impressive performance. Nevertheless, the underlying mechanism needs further exploring and probing in future studies.
How do different vision models perform? We benchmarked several backbone vision models within our framework. Specifically, we tried another popular pre-trained vision transformer Vin and a pre-trained CNN-based model, R The results are presented in Fig. The pre-trained transformer-basee ViT and Swin Transformer demonstrate comparable performance, both outperform- ing the previous state-of-the-art method, Raindrop. In contrast, the pre-trained CNN-based ResNet lagged considerably behind the vision transformer models. This performance gap is consistent with observations in image classification tasks on datasets like ImageNet, where vision transformers have been shown excel preserving spatial information compared to conventional CNN models. This advantage enables vision transformers to effectively capture the positions of patches within each line graph sub-image and the entire image and facilitates the modeling of complex dynamics and relationships between variables.
How to create time series line graph images? Using line graphs to visualize time series offers us an intuitive way to interpret the data and adjust the visualization strategy for enhanced clarity and potentially augment the performance. To offer insights on effective time series-to-image transforma- tion, we analyze the effects of several key elements in practice: (1) the default linear interpolation to connect partially observed data points on the line graphs; (2) markers to indicate observed data points; (3) variable-specife colors to differentiate between line graphs representing different variables; (4) the order determined by missing ratios when organizing multiple line graphs in a single image.
The results are presented in Table 3 Given the balanced missing ratios in the PAM dataset, we excluded results without the sorting order. Interestingly. plotting only observed data points without linear interpolation led to better results on P19 and P12 datasets. This could be attributed to the potential inaccuracies introduced by interpolation, blurring distinctions between observed and interpolated points. Additionally, omitting markers complicates the model's task of discerning observed data points from interpolated ones, degrading its performance. The absence of distinctive colors led to the most significant performance drop, underlining the necessity of using varied hues for individual line graphs help the model to distinguish them. While specific sorting order may not ensure optimal outcomes across all datasets, it does provide relatively stable results over multiple datasets and evaluation metrics. For the PAM dataset, these nuances seem have minimal impact, indicating the robustness of our approach against these variations in some scenarios.
Effects of grid layouts and image sizes. We explored the influence of grid layouts and image dimensions on our approach's efficacy. For a fair comparison across grid layouts, we fixed the size of each grid cell as 64 x 64 and altered the grid layouts. As shown in Figure [5] we observed our approach's 's robustness to variations in grid layouts, with square layout consistently yielding good results across different datasets and metrics, which was particularly evident for the P12 dataset. Regarding image size, when we maintained the grid layout but reduced the overall image dimensions, noticeable performance decline was observed on the P12 and PAM datasets, which complies with our intuition.
Robustness against varied plotting parameters. To gauge the robustness of our approach against different plotting parameters, we assessed aspects including line style/width and marker style/size,
hhapss//hugiingfacecc htg.://ggg....toooooooooooiiiiaa 1--24-21kk
8

---

100
1250x576)
2x9 (128x576)
435 1250:200
5x7 1320x4
(390x512)
360 (192x384)
525 1224x224)
80
1364x384)
80
6xb 130-1x3011
(224x224)
96
60
92
50
AUROC
AUPRC
40
AUROC
AUPRC
90 Accuracy Precision
Recall
Fl
Score
(a) P19
PAM
Figure 5: Ablation study of the influence of grid layouts and image sizes. For instance, 4x9 (256x576) denotes a grid layout xx9 with an image size 256x576 pixels.
primarily using the P19 dataset. As shown in Table 4 our approach demonstrates robustness against changes these parameters, maintaining strong performance across different plotting configurations.
What does VITST capture? To gain insights into the patterns captured by ViTST in the time series line graph images, we analyzed the aver- aged attention map of a VITST model with ViT as the backbone, as depicted in Fig. The at- tention map reveals that the model consistently attends to the informative part, i.e., line graph contours within the image. Furthermore, we ob- served that the model appropriately focuses on observed data points and areas where the slopes of the lines change. Conversely, flat line graphs that lack dynamic patterns seem to receive less attention. This demonstrates that VITST might be able to discern between informative and informative features in the line graph images, enabling to extract meaningful patterns.
Table 4: Robustness regarding the style and size of lines and markers. the brackets, the first element denotes style, and the second represents size.
Linc
Marker
AUROC
AUPRC
(solid,
2)
89.2
53. 1
(dashed,1)
(+,2)
2.1
53.7
(dotted.1)
(+,2)
2.1
52.8
(solid,0.5)
(+,2)
88.6
53.0
(solid,
(+,2)
8.0
53. 1
(solid,2)
(+,2)
88.5
53.6
(solid,1)
(+,2)
8.0
53.
(solid,
(A, 2)
89.3
52.6
(solid,
(o,
89.1
51.3
(solid,
(+,
88.2
52.
(solid,1)
(+,2)
89.2
53.
(solid,) I)
(+, 3)
88.9
52.8 12.888..
M
Input
w
Attention
P19
P12
Figure 6: Illustration of the averaged attention map of VITST.
4.4 Regular Time Series Classification
advantage of our approach its ability to model time series of diverse shapes and scales, whether they regular irregular. To evaluate the performance of our approach regular time series data,

---

Table 5: Performance comparison on regular multivariate time series datasets, Bold indicates the best performer, while underline represents the second best.
Datasets
EC
SCPI
SCP2
JV
SAD
HB
FD
PS
FW
Average
Datasel statistics
#Variables
3
6
12
13
61
144
963
Length
1,751
315
896
1,152
29
93
405
62
144
17984
Model performances
DTWp
0.323
0.903
0.775
0.539
0.949
0,963
0.717
0.529
0.711
0.618
0.717
LSTM
0.323
0.412
0.689
0.466
0.797
0,319
0.722
0,577
0.399
0,523
XGBoost
0.437
0.759
0,846
0.489
0.865
0.696
0.732
0.633
0,983
0.727
Rocket
0.452
0.944
0.908
0,533
0.962
0.712
0.756
0.647
0.751
0.741
TST
0.326
0.913
0.922
0.6044
0.997
0.998
0.776
0.689
0.896
0.748
0.791
VITST
0.456
0.862
0.898
0.561
0.946
0,985
0.766
0.632
0.913
0.878
0.780
representative multivariate time series datasets from the UEA Time Series Classification Archive (2). These datasets exhibit diverse characteristics, as summarized in Table It is worth noting that the PS dataset in evaluation contains exceptionally high number of variables (963), while the EW dataset has extremely long time series (17984). We specifically
conducted experiments
selected these two datasets to assess the effectiveness of our approach in handling large numbers of variables and long time series. We follow (47] to use these baselines for comparison: DTW D which stands for dimension- dimension-Dependdnn DTW combined with dilation-CNN M. LSTM B
XGBoost (3), Rocket 77, and transformer-basee TST 47 which operates on fully observed numerical time series.
The performance of our approach on regular time series datasets is consistently strong, demon- strated in Table 5 With an average accuracy that is second-hest and closely aligned with the top-performing baseline method TST, our approach showcases its competitive capabilities. Notably, excels on challenging datasets PS and EW with massive variables and observation length. These results were achieved using the same image resolution (384 x 384) the other datasets, indicating the effectiveness and efficiency of our approach. The ability of our approach to handle both irregular and regular time series data further emphasizes its versatility and broad applicability.
5 Conclusion
In this paper, we introduce a novel perspective on modeling irregularly sampled time series. By transforming time series data into line graph images, we could effectively leverage the strengths of pre-trained vision transformers. This approach is straightforward yet effective and versatile, enabling the modeling of time series of diverse characteristics, regardless of irregularity, different structures, and scales. Through extensive experiments, state-of-the thettrrrrt methods designed for irregular time series and maintains strong robustness to varying degrees of missing observations. Additionally, our approach achieves promising results regular time series data. We envision its potential as a general general-purpose purpose framework for various time series tasks. Our results underscore the potential of adapting rapidly advancing computer vision techniques to time series modeling.
demonstrate that approach surpasses
6 Limitations and Future Work
In this work, we utilized a straightforward method image multivariate time series by converting them into line graph images using matplotii and then saving them as RGB images. While our results are promising and exhibit robustness against variations in the time series-to-image transformation process, there may be alternative ways to visualize the data. This includes potentially more controllable and accurate plotting methods or different image representations beyond line graphs. Our findings also highlight the efficacy of pre-trained vision transformers for time series classification, suggesting that these models might leverage knowledge acquired from pre-training on natural images. Yet. the underlying reasons for their remarkable success still need deeper exploration and investigation. This research serves as a promising starting point this domain, suggesting various potential directions. We leave these further explorations and investigations for future work.
10

---

References
[1] Ba, J., Kiros, J. R., and Hinton, G. E. Layer normalization. ArXiv, abs/1607.064500 2016.
[2]
Bagnall, A., Dau, H. A., Lines, J., Flynn, M., Large, J., Bostrom, A., Southam, P., and Keogh, The uea multivariate time series classification archive, 2018, arXiv preprint arXiv:1811.00075 2018.
[3] Che, Z., Purushotham, S., Cho, K., Sontag, D., and Liu, Y. Recurrent neural networks for multivariate time series with missing values. Scientific reports, 8(1):1-12, 2018.
[4] Chefer, H.,, Gur, S., and Wolf, L. Transformer interpretability beyond attention visualization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 782-791, 2021.
[5] Chen, T. and Guestrin, C. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pp. 785-794, 2016.
[6] Chung, J., Gulcehre, C., Cho. K., and Bengio, Y. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv: 1412.355, 2014.
[7] Dempster, A., Petitjean, F., and Webb, G. I. Rocket: exceptionally fast and accurate time series classification using random convolutional kemels.. Data Mining and Knowledge Discovery, 34 (5):1454-1495, 2020.
[8] DeVries, T. and Taylor, G. W. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.0000 045552, 2017.
[9] Dosovitskiy, A., Beyer, I.., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., al. An image worth 16x 16 words: Transformers for image recognition at scale. International Conference on Learning Representations, 2020.
[10] Fan, Q., Chen, C.-F., and Panda, R. Can an image classifier suffice for action recognition? In International Conference on Learning Representations, 2021.
[11] Franceschi, J.-Y., Dieuleveut, A., and Jaggi, M. Unsupervised scalable representation learning for multivariate time series. Advances neural information processing systems, 32, 2019.
[12] Goldberger, L., Amaral, A., Glass, L., Hausdorff, J. M., Ivanov, P. C., Mark, G., Mietus, J. E., Moody, G. B., Peng, C.-K., and Stanley, H. E. Physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals. circulation, 101(23): e215 e220, 2000.
[13] Graves, A. Long short-term memory. Supervised sequence labelling with recurrent neural networks, pp. 37-45, 2012.
[14] Hatami. N., Gavet, Y., and Debayle, J. Classification of time-series images using deep convo- lutional neural networks. In Tenth international conference on machine vision (ICMV 2017), volume 10696, pp. 242-249. SPIE, 2018.
[15] He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16000-16009, June 2022.
[16] Horn, M., Moor, M., Bock, C., Rieck, B., and Borgwardt, Set functions for time series. In International Conference on Machine Learning, pp.
PMLR, 2020.
[17] Langley, P. Crafting papers on machine learning. In I Langley, P. (ed.), Proceedings of the 77th International Conference on Machine Learning (ICMI. 2000), pp- 1207-1216, Stanford, CA, 2000. Morgan Kaufmann.
[18] Lea, C., Flynn, M. D., Vidal, R., Reiter, A., and Hager, G. D. Temporal convolutional networks for action segmentation and detection. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 156-165, 2017. 2017.

---

[19] Li. S., Jin, X., Xuan, Y., Zhou, X., Chen, W., Wang, Y.-X., and Yan, X. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting, Advances in neural information processing systems, 32, 2019.
[20] Lipton, Z. C., Kale, D., and Wetzel, R. Directly modeling missing data in sequences with mns: Improved classification of clinical time series. In Machine learning for healthcare conference, pp. 253-25322220 233-270. PMLR, 2016.
[21] Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M.. Zettlemoyer, L.,, and Stoyanov, V. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1692, 2019.
[22] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012-10022, 2021.
[23] Liu, Z., Hu, H., Lin, Y., Yao, Z., Xie, Z., Wei, Y., Ning, J., Cao. Y., Zhang, Z., Dong, L., et al. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 12009-12019, 2022.
[24] Marlin, B. M., Kale, D. C., Khemani, R. G., and Wetzel, R. C. Unsupervised pattern discovery in electronic health care data using probabilistic clustering models. In Proceedings of the 2nd ACM SIGHIT international health informatics symposium, pp. 389-398, 2012.
[25] Nie, Y.. Nguyen, N. H., Simthong, P., and Kalagnanam, J. A time series is worth 64 words: Long-term forecasting with transformers. arXiv preprint arXiv:2211.14730, 2022.
[26] OpenAl. Gpt-4 technical report, 2023.
[27] Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 7748-8765. PMLR, 2021.
[28] Reiss, A. and Stricker, D. Introducing new benchmarked dataset for activity monitoring. In 2012 16th international symposium on wearable computers, pp. 108-109. IEEE, 2012.
[29] Reyna, M. A., Josef, C., Seyedi, S., Jeter, R., Shashikumar, S. P., Westover, M. B., Sharma, A., Nemati, S., and Clifford, G. D. Early prediction of sepsis from clinical data: the phys- ionet/computing in cardiology challenge 2019. In 20/9 Computing Cardiology (CinC), pp- Page-1. IEEE, 2019.
[30] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma. S., Huang, Z.. Karpathy, A.. Khosla, A., Bernstein, M., et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-2533 2015.
[31] Semenoglou, A.-A., Spiliotis, F.,, and Assimakopoulos, V. Image-based time series forecasting: A deep convolutional neural network approach. Neural Networks, 157:39-53, 2023.
[32] Shorten, C. and Khoshgoftaar, T. M. A survey on image data augmentation for deep learning. Journal of big data, 6(1): 2019.
[33] Shukla, S. N. and Marlin, B. Interpolation series. In International Conference on Learning Representations, 2018.
networks for irregularly sampled time
[34] Shukla, S. N. and Marlin, B. Multi-time attention networks for irregularly sampled time series. In International Conference on Learning Representations, 2020.
[35] Sood, S., Zeng, Z., Cohen, N.. Balch, T., and Veloso, M. Visual time series forecasting: an image-driven approach. In Proceedings of the Second ACM International Conference on Al in Finance, pp. 1-9, 2021.
[36] Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jégou, H. Training data- efficient image transformers distillation through attention, In International Conference on Machine Learning, pp. 10347-10357. PMLR, 2021.
12

---

[37] Tripathy, R. and Acharya, U. R. Use of features from IT- ri-time series and eeg signals for automated classification of sleep stages in deep neural network framework. Biocybernetics and Biomedical Engineering, 38(4):890-90) 902, 2018.
[38] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, Attention is all you need. Advances neural information processing systems, 30, 2017.
[39] Wang, Z. and Oates, T. Imaging time-series to improve classification and imputation. In Twenty-Fourth International Joint Conference on Artificial Intelligence, 2015.
[40] Wang, Z. and Oates, T. Spatially encoding temporal correlations to classify temporal data using convolutional neural networks. arXiv preprint arXiv: 159.0081,,, 2015.
[41] Wu, H., Xu, J., Wang, J., and Long, M. Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting. Advances in Neural Information Processing Systems, 34:22419-22438 2021.
[42] Wu, Y. Ni, J., Cheng, W. Zong, B., Song, D., Chen, Z., Liu, Y., Zhang, X., Chen, H., and Davidson, S. B. Dynamic gaussian mixture based deep generative model for robust forecasting on sparse multivariate time series. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 651-659, 2021.
[43] Wu, Z., Pan, S., Long, Jiang, J., Chang, X., and Zhang, C. Connecting the dots: Multivariate time series forecasting with graph neural networks. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining. pp. 753-763, 2020.
[44] Xie, Z., Zhang, Z., Cao, Y., Lin, Y., Bao, J., Yao, Z., Dai, Q., and Hu. Simmim: simple framework for masked image modeling. In Proceedings of the IEEF/CVF Conference Computer Vision and Puttern Recognition, pp. 9653-9663, 2022.
[45] Xu, J., Wu, H., Wang, J., and Long, M. Anomaly transformer: Time series anomaly detection with association discrepancy. arXiv preprint arXiv:2110.02642, 2021.
[46] Yoon, J., Zame, W. R., and van der Schaar, M. Multi-directional recurrent neural networks: A novel method for estimating missing data. In Time series workshop in international conference on machine learning, 2017.
[47] Zerveas, G., Jayaraman, S., Patel, D., Bhamidipaty, A., and Eickhoff, C. A transformer-basse framework for multivariate time series representation learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 2114 2124, 2021.
[48] Zhang, X., LI, S., Chen, 7.,, Yan, X., and Petzold, L.. Improving medical predictions by irregular multimodal electronic health records modeling. ArXiv, abs/2210. 12156, 2022.
[49] Zhang, X.. Zeman, M., Tsiligkaridis, T., and Zitnik, M. Graph-guided network for irregularly sampled multivariate time series. In International Conference on Learning Representations, 2022.
[50] Zhang. Y. Attain: Attention-based time-aware Istm networks for disease progression modeling. In In Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI- 2019), pp. 4369-4375, Macao, China., 2019.
[51] Zhou, H., Zhang. Peng, J., Zhang, S., Li, J., Xiong, H., and Zhang, W. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 11106-11115, 2021.
[52] Zhou, T., Ma, Z., Wen, Q., Wang, X., Sun, L., and Jin, R. Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting, arXiv preprint 1:Xiv:2201.12744 2022.
13

---

A More Details on Time Series Line Graph Image Creation
The teme--sries-to-inee to-image transformation can be implemented using the Matplotlib package with the following few lines of code.
Implementation
def IS2Image (t, D, colors, saraag_heeihht,.. image_widee grid height, grid_width) plt
3 import chhppootiib.ppplee
(figsize- (image_height/2)
image_width/100) dpi-100)
4
for á in range (D): enumerate the multiple variables
5
(grid_height, grid_width, d+1) # position in the grid # plot line graph of variable d plt.plot (t [d], v[d], colcr=colorsid), linestyle="=", marker=*x")
Axis Limits of Line Graphs The axis limits determine the plot area of the line graphs and the range of displayed timestamps and values. By default, set the limits of the x-axis and y-axis the ranges of all the observed timestamps and values. However, we found that some extreme observed values for some variables can largely expand the range of the y-axis, causing most plotted points of observations to cluster in small area and resulting in flat line graphs. Common normalization and standardization methods will not solve this issue, as the relative magnitudes remain unchanged in the created images. We thus tried the following strategies to remove extreme values and narrow the range of the y-axis in our preliminary experiments:
Interquartile Range (IQR): IQR is one of the most extensively used methods for outlier detection and removal. The interquartile range is calculated based on the first and third quartiles of all the observed values of each variable in the dataset and then used to calculate the upper and lower limits.
Standard Deviation (SD): The upper and lower boundaries are calculated by taking 3 standard deviations from the mean of observed values for each variable across the dataset. This method usually assumes the data is normally distributed.
Modified Z-score (MZ): A z-score measures how many standard deviations away a value is from the mean and is similar to the standard deviation method to detect outliers. However, z- scorees can be influenced by extreme values, which modified z-scores can better handle. We set the upper and lower limits as the values whose modified z-scores are 3.5 and -3.5,
We show examples of the created images with these strategies in Figure 7
Table 6: Preliminary experiments on different strategies to decide the line graph limit. The default strategy is to directly set the axis limit as the range of all observed values on the dataset. "IQR". "SD", and "MZS' denote three strategies remove extreme value, i.e., Interqurtile Range, Standard Deviation, and Modified 7-score. The reported numbers are averaged on data splits.
P19
P12
PAM
Strategies
AUROC
AUPRC
AUROC
AUPRC
Accuracy
Precision
Recall
Fl score
Default
89.4
52.8 +
85.6 1.1
49.8
96.1
96.8
96.5 0.7
96.6 0.9
IQR SD MZS
8..
1.7
84.5
95.9 9.6
96.8 a
96.1
96.4
1.4
51.2 13.6
8..6
1.99
96.6 6 1.9
97.1 1 0.8
97.0 1 0.6
97. 0 0
8.0
50.8
84.3
47.1
96.0
96.8
96.4
96.6
The performance comparison of models trained on images created with different strategies is shown in Table 6 We observe that the methods that remove extreme values hurt the performance, except for SD on the PAM dataset. Although these methods narrow the value range and highlight the dynamic patterns of line graphs, they discard the extreme values that might be informative themselves. Therefore, we stick with the default way that sets the axis limits as the range of all observed values.
Computation cost We list the inference time (in seconds) of different methods the test sets of three datasets in Table All the inferences are made in a single Nvidia A6000 GPU. It is observed that our vision-based method consumes more inference time than the non-vision baselines. However, we believe this cost remains within an acceptable range in the context of today's 's ML practice and medical applications, considering the inference each sample only costs around 0.01 seconds.
hntpss//mapllotibborr//
14

---

Defaalt
Lo
m
IQR
M
W
Stamdard Deviation
Yit
Mpx
m
M
W
Modified Z score
nv
W
(a) P19
(b) P12
(c) PAM
Figure 7: The images created with different strategies for three samples from P19, P12, and PAM dataset, respectively (sample "p000019" for P19, "132548" for P12, and "0" for PAM).
Table 7: The inference time (in seconds) of different methods the test sets.
Datasets
Trasformers
m'AND
SeFT
Raindrop
MTGNN
DGM
GRU-D
VITST
P19
0.21
0.52
2.72
3.05
3.62
2.47
31.04
44.51
P12
0.12
0.44
0.97
1.27
1.46
2,80
10.13
12.14
PAM
0.06
0.23
0.89
0.67
1.16
2.98
4.55
5.30
15

---

B More Experimental Details
B.1 Datasets
We used the datasets processed by [49], whose details given below.
P19: PhysioNet Sepsis Early Prediction Challenge 2019. The P19 dataset consists of clinical data for 38,803 patients, and aims to predict whether sepsis will occur within the next 6 hours. The dataset includes 34 irregularly sampled sensors with 8 vital signs and 26 laboratory values for each patient, as well as 6 demographic features. To process the static features, we use templates outlined T and utilize a pre-trained Roberta-base model extract textual features. These textual features are then combined with visual features obtained from the vision transformer to perform binary classification. The dataset is highly imbalanced with only 4% of samples being positive, and has missing ratio of 94.9%.
P12: PhysioNet Mortality Prediction Challenge 2012. P12 dataset [2] includes clinical data from 11,988 ICU patients, with 36 irregularly sampled sensor observations and 6 static demographic features provided for each patient. The goal is predict patient mortality, which is a binary classification task. The dataset is highly imbalanced, with around 86% of samples being negative. The missing ratio of the dataset is 88.4%.
PAM: PAMAP2 Physical Activity Monitoring. The PA. dataset originally contains data of 18 physical activities with 9 subjects wearing inertial measurement units. However, make it suitable for irregular time series classification, [49 excluded the ninth subject due to its short length of sensor readouts, and 10 out of the 18 activities that had less than 500 samples were also excluded. As result, the task is an 8-way classification with 5,333 samples, each with 600 continuous observations. To simulate the irregular time series setting, 60% of the observations are randomly removed. No static features are provided, and the 8 categories are approximately balanced. The missing ratio is 60.0%.
Table 8:
Templates for transforming static features natural language sentences.
Dataset
Static features
Template
Example
Age, Gender, Unitl (medi- cal ICU), Unit2 (surgery ICU), HospAdmTime; ICULOS (ICU length-of- stay)
A patient (Age) years old, [Gender), went [Unirl Unit2) (HospAdmTime) hours after hospi- tal admit, had stayed there for IICU- LOS) hours.
patient is 65 years old, female, went the med- ical ICU 10 hours after hospital admit, had stayed there for 20 hours.
P19
RecordlD. Age. Gender. Height (em), ICUType, Weight (kg)
A patient is [Age] years old, [Gen- derl. (Height) em, (Weight) kg,
A patient is 48 years old, male, 171 em. 78 kg, stayed in surgical ICU,
P12
stayed in (ICUType).
Table 9: Ablation studies on different methods to encode static features.
P19
P12
Methods
AUROC
AUPRC
AUROC
AUPRC
Raindrop
87.0 à 2.3
51.8
82.8
44.0 1: 3.0
Swin
89.4
50.2
84.3
49.3
Swin-MI.D
88.6 :: 1.3
31.4
84.6
48.7 2
Swin- Roberta
89.4 + 1.9
52.8 3.8
85.6
49.8 2.5
B.2 Experiments on Static Features
Time series data often associated with information from other modalities, such as the textual clinical notes in electronic health records (EHRs) in the healthcare domain. Our approach is naturally suitable for incorporating such information since we convert time series data to images, and thus various vision-language and multi-mode techniques can be utilized to incorporate the visual (time series) information and information from other modalities. For example, the CLIP 27 learns a shared
16

---

hidden feature space where the paired image and text stay close. Under our framework, such shared space can also be learned for the paired visual time series images and textual clinical notes, which is our future direction. It also paves the way for the application of multi-modal models such as GPT- 4 [26] to handle the visualized time series data and the clinical notes simultaneously. In our current experiments, we used text encoder, Roberta-band baare, to encode textual demographic information in the P19 and P12 datasets. We also experimented with normalizing the original categorical features and encoding them using MLP as in previous work, and compare with the strong bascline, Raindrop. The results are shown in Table [ already outperformed Raindrop. In addition, utilizing Roberta to encode and incorporate the textual feature is more effective than applying MLP over categorical features.
We observe that even without using static features, our method has
Table 10: Preliminary experiments on two settings to fuse the line graph images for different variables. The default setting is first arrange all line graph images into single image and then learn the representation for classification. ViT-subimage stands for that we first learn the representation for each line graph subimage separately and then concatenate their representation for classification.
P19
P12
PAM
Stratopies
AUROC
AUPRC
AUROC
AUPRC
Accuracy
Procision
Recall
Fi score
VIT ViT-subimage
2..
61.613.
84.8
48.1 138
93.4
94.7=
94.1 2.7
0.
8..
47.
17.6492
35.5
90.4
92.9=
91.1=0.
91.9
B.3 Experiments on Image Fusion
In our preliminary experiments, we examined two distinct approaches fusing the line graph sub- images in each multivariate time series data. First, we processed each sub-image independently to learn the patch/image representations and then concatenated their respective patch representations to input into the final prediction layer. In contrast, our default method aggregates all sub-images into single image to learn the patch embeddings. The key distinction between these strategies is whether a patch attend to those from other sub-images and how position embedding factors in during the representation learning phase. Aside from this, other parameters were consistent across both methods. We tested on ViT and the performance comparisons with these two settings are shown in Table [0] It evidenced that attending to patches from other sub-images offers advantages. This likely allows for capturing cross variable correlations granular level within the self-attention layers, opposed to only at the final linear prediction layer.
Table 11: Statistics and hyperparameter settings of evaluated regular multivariate time series datasets.
Datasets
Variables
Classes
Length
Train size
Grid layout
Image size
Learning rale
Epochs
EC
1,751
261
2
256 x 256
Ic-4
315
120
2
256 x
lc-4
100
SCPI
896
268
2
256 x
le-4
100
SCP2
1.152
200
3 3
384 384
5c-5
100
IV
12
29
270
4
384 x 384
lc-4
100
SAD
13
10
93
6599
4
384 x 384
lc-5
20
HB
61
2
405
204
4 4
384 x 384
le.4
100
FD
144
2
62
5890
12 12
384 x 384
5c-4
100
PS
963
7
144
267
32 x 32
384 X 384
5e-4
100
FW
17984
128
2x3
256 x 384
2e-5
100
B.4
Experiment on Regular Time Series
We selected ten representative multivariate time series datasets from the UEA Time Series Classifica- tion Archive 2 with diverse characteristics, including the number of classes, variables. and time series length. The datasets we chose are EthanolConcentration (EC), Handwriting (HW), UWaveGes- trreLibrary (UW), SelfRegulationSCCII (SCP1), SeliRegulationSCPP (SCP2), lapanese (JV), SpokenAraaicDiggiss (SAD), Heartbeat (HB). FaceDetection (FD), PEMS SF (PS), and EigenWorms (EW). Notably, the PS dataset has an exceptionally high number of variables (963), while the EW dataset has extremely long time series (17984). These two datasets allow us to assess the effectiveness of our approach when dealing with large numbers of variables and long time series. We applied different image sizes according to the grid layouts for these datasets. The hyperparameter settings
17

---

provided in Table [1 and we applied cutout 8 data augmentation methods to SCPI, SCP2, and JV datasets due to the small size of their training sets.
B.5 Self-supervised Learning
We preliminary explored masked image modeling self-supervised pre-training on the time series line graph images. We randomly mask columns patches with a width of 32 each line graph within grid cell. The masking ratio is set as 50%. We finetuned the Swin Transformer model for 10 epochs with batch size 48. The learning rate is 2e-5. Following [44] we use linear layer reconstruct the pixel values and employ an & loss on the masked pixels:
L 2((M) |P PMll1
where Ри and PM are the masked and reconstructed pixels, respectively; Q(-) denotes the number of elements. With self-supervised masked image modeling, the performance improves by 1.0 in AUPRC points (percentage) from 52.8 (# 3.8) to 53.8 (= 3.2). The AUROC points (percentage) slightly dropped from 89.4 (# 1.9) to 88.9 (+: 2.1).
B.6 Full Experimental Results
We presented the full experimental results in the leave- sou settings in Table 12
Table 12: Full results in the leave-sensors-out settings on PAM dataset. The "missing ratio" denotes the ratio of masked variables.
Missing
Methods
PAM [laave-frodf-semorreee
PAM (I (wemoo-rmmon---nneeeeeeoe (on)ioommm-mnnnooonn)))))
Acconsey
Precision
Recall
El spore
Apcuracy
Precision
Recall
scnec
57.2
Tran-mon
60.4 11:2
61.8 14.9
10.2
58.0 15.2
62.4 3.5
50.6
63.7
62.7 :
GRL-D
65.4+
+.66+26
6..3+
63.6 0.4
6
+.2+1.0
70.8
72.0
10%
SeFT
58.9
1.8
5..6
59.8=
4..0
40,8
41.0+0.7
39.9
6.8
54.6177
Raindrop
7.
1.1
1.
75
76.7 1.1
70.9117
77.9199
78.6
VITT
92.811.0
94.2111
93.4 93.41
98.7 1.6
98.1 1.0.9
94.3 n.0.9
94.0
94.1 1.1.1
6..1
71
6.2
12.77
11.4
Trans-meun
31.2++3.0
74.2+ 1.8
63.5 4.4
64.1
36.8
59.4
53.2+3.
56.3 3.5
GRL-D
64.6+11
1..3+
63.5+
64.8 5.8
618+04
69.8
65.8
6..2+
20%
SeFT
35.7
12.1
38.1
35.0
312428
31.9
31.6211
2.
49.0+16
Raindrop
4..5+4..
5.
67.9
6
7L3+22
75,8+12
72.5+20
2314+111
VITT
89.7
91.$
9..
90.8 1.8
1..
93
92.8
93.0
Transfomer
31.62
2.74
24.0 10.0
19.0 12.8
12.0
16.2
50.1 13.4
48-4 18.2
Trans-esn
4..52.8.
4..31
37.0
33.9
65.1 119
63.8112
67
64.9 1.7
30%
GRL-D
45.112.
61
42.1
47.2
58.0
63.1177
18.2111
1..
ScFT
31.5
Raindrop
52.4
5..9
51.3
48.4=18
60.
88.1
60.3
61.9
VITT
88.3
1.7
88.5 0.77
89.8 1.0.
90.1 110
89.8 1.0.9
Transfommrr
2..0
7.4166
2..5
6.9
43.8
44.6
40.5 16.9
40.2
Trans-eem
25.71
9.1.99
18.5114
9.9
48.7 1.9.
56.8
64.2.1.
56.1 2.2.9
GRU-D
16.4
64.5
4.
44.3
47.7
63.411
44.5
47.5
26.310.
1..4
1..1
13.7
33.9
26.416
29.3
Kaindrop
3.7
63.4
4..
44.7 334
57.0
65.7
56.7 1:31
58.9 2.5
VIST
2..0+2.6
8..7+
8.3++24
812 2.7
83.7 1.3
85.5 15.
85.6
85.1 +.3
Transformer
2..4
2.7
12.5+
0.4
43.2+2.5
52,0+2.5
36,9
4..9
Trans mon
21.3
2.8.2.44
12.5
1.6002
46.1
59.1
43.1222
36.5
GRU-D
37.
29.6
32.8
28.6
40.7
32.403
42.5117
47.5
SeFT
23.5
mTAND
16.9
12.6
17.0=1
13.9=4.
20.9
36.1
23,0#32
27.7 2:72:3
Raindrop
46.6222.
44.52.20
42.4
38.02:40
47.2144
30.4
44.8258
47.6
VITST
79.
83441.3
81.8110
80.8
8..8114
84.9
84.9
84.4
18


Reviews:
[{"review_id": "jImcB6O0Sx", "reviewer": "Reviewer_7PQE", "summary": "The authors propose a method to model irregularly sampled time series. The method is based on transforming numerical time series data to line graphs and then applying pretrained vision transformers to that data. For multivariate datasets, every variable is plotted separately, and plots are aggregated in a grid. The authors provide exhaustive evaluation on different datasets, including leaving-sensors-out settings and regularly sampled time series data.", "questions": "-\tFor the PS dataset, the authors must have used much smaller (I assume 12x12) sizes for their plots, am I correct?\n-\tWhat is the computational cost of the proposed method?\n-\tDid the authors conduct experiments on whether the order of the plots in the grid matters? \n-\tBased on the ablation studies provided by the authors it seems likely that most of the performance comes from pretraining of the transformer models. Wouldn’t a large pretrained transformer for numerical time series modeling be likely to perform similar or better than the method proposed by the authors? If the authors agree, then I would like to see a brief discussion of this in an updated version of the manuscript.", "limitations": "Unfortunately, the authors do not comment on limitations of their work. I see the following possible limitations and would like to see an open discussion of those in an updated version of the manuscript:\n-\tAbility to handle extremely large number of variables\n-\tComputational costs of the method", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is well written and easy to understand. The method proposed by the authors is extremely simple, yet surprisingly effective on many time series modeling tasks, as shown by exhaustive experimental evaluation.", "weaknesses": "-\tI don’t see the authors claims made in the abstract and introduction backed by their empirical results. It is difficult to judge, whether their proposed method indeed outperforms previous approaches “significantly” (L. 10, L. 43, L. 67) as such an analysis is missing. On most of the metrics for the P19 and P12 data at least, improvements do not look significant to me. On the PAM dataset results are likely significant (analysis is also missing here).\n-\tThe PAM dataset (the only one where we likely actually see a significant improvement over Raindrop) is also the one with the fewest number of variables (17 vs 34 & 36 for P19 and P12, respectively). I assume that the proposed method performs much worse for datasets with many variables, as also shown in the authors experiments on the FD, and PS datasets, where ViTST performs indeed much worse than other methods. \n-\tThe authors do not comment on the computational cost of their method, which is likely much higher than for most non-vision based approaches.\n-\tRecently, Semenoglou et al. proposed a very similar method, albeit limited to single-variable time series. They plot a univariate time series and feed this plot into a CNN to predict future observations of the same series. While I think that the present work improves on the work by Semenoglou et al. in several regards (multivariate, pretrained ViT outperforming pretrained ResNet, …), the authors should include a reference to Semenoglou et al. in their related work. Additionally, I don't think statements such as \"This paper studies the problem from a whole new perspective by transforming irregularly sampled time series into line graph images and leveraging powerful vision transformers for time series classification in the same way as image classification.\" (L. 4-7) are valid, given that the idea to transform time series to images and then applying image classification networks is not novel.\n-\tThe authors do not comment on limitations of their method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable feedback. Below are our responses to your concerns:\n\n**Response to W1: Concerns on “significant” improvement**\n\nWe base our claim of \"significant\" improvement on extensive comparisons across three datasets and their associated metrics. In all the evaluated datasets and associated metrics, our method consistently shows superior performance.\n- P19 dataset: Our method outperforms the second-best approach, Raindrop, by 2.4 points in AUROC and 1.0 points in AUPRC.  In addition, our approach has a lower variance than Raindrop and the third-best approach DGM2-O. \n- PAM dataset: The advantage of our approach over the compared baselines are especially signification.\n- P12 dataset: While Raindrop is considered a strong baseline on P19 and PAM datasets, it lags behind our method by 2.8 points in AUROC and 5.8 in AUPRC. DGM2-O and mTAND are respectively the second-best in AUROC and AUPRC for this dataset. However, neither of the baselines achieves reasonable performance on both AUROC and AUPRC metrics.\n\nIn conclusion, none of the specialized methods we evaluated consistently matched or surpassed our method across all metrics and datasets. Beyond the numbers, what we want to show is that our simple approach can match or surpass specialized methods regardless of the margin.\n\n**Response to W2: Performance on datasets with many variables**\n\nWhile the number of variables may play a role, other factors could also influence why our method's advantage on the P12 and P19 datasets isn't as significant as on the PAM dataset, such as the number of observations and/or the length of the time series.\nRegarding the PS dataset with the largest number of variables (963), our approach achieves better performance than the strong baseline TST and is second-best only behind XGBoost. On the FD dataset, it is comparable to XGBoost and ROCKET. It's worth noting that different models may be optimized for specific data types. For instance, our method excels in handling long sequences (EW dataset with a sequence length of 17984). When evaluating performance across all datasets, our method is second-best.  It's important to emphasize that most methods tailored for irregularly sampled time series often struggle with regularly sampled data, and vice versa.\n\n**Response to W3: Computational cost**\n\nOur experiments were conducted on Nvidia A6000 GPUs. It takes around 24 mins to run 20 epochs on the PAM dataset with a batch size of 72,  32 mins to run 4 epochs on the P12 dataset, and 58 mins to run 2 epochs (with upsampling) on the P19 dataset—all on a single GPU. While our computational costs might be higher than some non-vision methods, they appear to be within the range that could be considered acceptable in the context of current ML practices, considering the widespread use of large language and vision models nowadays.\n\n**Response to W4/5: Related work and limitations**\n\nWe will add the work of Semenoglou et al. in the related work and add the limitation section in the updated version.\n\n**Response to Q1: For the PS dataset, the authors must have used much smaller (I assume 12x12) sizes for their plots, am I correct?**\n\nOne can choose any size for the plot. For example, we tried the size of each plot (grid cell) as 24x24, where the image size increased to 768x768. With this adjustment, the performance of ViTST improves from 91.3 to 92.4, compared with using a 12x12 plot size.\n\n**Response to Q2: Computation cost**\n\nPlease refer to the response to W3.\n\n**Response to Q3: Impact of grid order on results**\n\nWe have conducted experiments to examine the impact of the grid order. By default, the grid order was sorted based on the number of observations. We also conducted tests with grids in random orders. The results from these tests are outlined below:\n| **P19** | **AUROC** | **AUPRC** |\n| --- | --- | --- |\n| Sorted order | 89.4$\\pm$1.9 | 52.8$\\pm$3.8 |\n| Random order 1 | 88.3$\\pm$1.8 | 49.9$\\pm$3.2 |\n| Random order 2 | 88.2$\\pm$1.6 | 51.0$\\pm$3.9 |\n\n| **P12** | **AUROC** | **AUPRC** |\n| --- | --- | --- |\n| Sorted order | 85.6$\\pm$1.1 | 49.8$\\pm$2.5 |\n| Random order 1 | 84.3$\\pm$2.2 | 48.0$\\pm$4.5 |\n| Random order 2 | 84.2$\\pm$1.7 | 47.6$\\pm$3.7 |\n\n| **PAM** | **Accuracy** | **Precision** | **Recall** | **F1 Score** | \n| --- | --- | --- | --- | --- |\n| Default order | 96.1$\\pm$0.7 | 96.8$\\pm$1.1 | 96.5$\\pm$0.7 | 96.6$\\pm$0.9 |\n| Random order 1 | 95.5$\\pm$1.0 | 96.8$\\pm$0.8 | 95.7$\\pm$1.0 | 96.2$\\pm$0.9 |\n| Random order 2 | 95.2$\\pm$0.7 | 96.6$\\pm$0.6 | 95.6$\\pm$0.8 | 96.0$\\pm$0.7 |\n\nThe order does affect performance, most notably in the P19 and P12 datasets, which have varying observations across different variables. On the other hand, the PAM dataset, with a uniform observation count for every variable, displays less variability in the performance. A possible explanation for this could be that clustering regions with higher observation plots, which are also more informative, allows the model to better capture their correlations and allocate attention effectively.\n\n**Response to Q3: Effectiveness of a large pretrained transformer for numerical time series modeling**\n\nA large pre-trained transformer for numerical time series modeling might offer comparable or superior performance to ViTST. However, such a transformer would necessitate extensive time series data for pretraining, requiring significant data collection efforts. It's important to emphasize that our intention is not to advocate that our vision transformer-based approach is the optimal option for performance. Instead, we aim to demonstrate that the publicly available pre-trained vision transformer can easily adapt to time series modeling and achieve satisfying results. This method largely simplifies model design and data collection efforts and could also bridge the time series and computer vision domain, opening up the possibility of adapting established vision techniques to the time series domain and facilitating multi-modal research.\n\n\n\n"}, {"review_id": "n7v9ttoUjW", "reviewer": "Reviewer_QWW4", "summary": "For the classification task based on irregular time series data, the authors introduced Vision Time Series Transformer (ViTST) approach where irregular time series data is displayed as line graph then fed to pretrained transformer type models. The authors tested this approach using several time series data from medical domain and human activity domain, and the result shows confident advantages of the proposed approach.", "questions": "Have you tested multi-class classification tasks and/or regression tasks?", "limitations": "Since this approach could have a potential to expand widely, it would be better to state \"Limitation\" of this paper.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Simple idea but very interesting. Could contribute to many domains dealing with time series data.", "weaknesses": "Time series data is everywhere but the authors tested only a few dataset from medical domain and human activity domain. Thus \"any shape\" in line 357 sounds too strong at this time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for valuing our work and recognizing its strengths. We aim to address your concerns with our response below.\n\n**Response to W1: Time series data is everywhere but the authors tested only a few dataset from medical domain and human activity domain. Thus \"any shape\" in line 357 sounds too strong at this time.**\n\nIn addition to the three irregularly sampled datasets P19, P12, and PAM, we also evaluated our approach on ten representative multivariate time series datasets from the UEA Time Series Classification Archive [1], as outlined in **Section 4.4**. These datasets have diverse characteristic in terms of number of variables, length, training size, and number of classes. Table 4 shows both the number of variables, length of different datasets and the performance comrasion on these datasets. More details on the datasets can be found in the Table 7 in the Appendix. For clarity, we also list it here:\n| Dataset | #Variables | #Classes | Length | Train Size |\n| --- | --- | --- | --- | --- |\n| EC | 3 | 4 | 1,751 | 261 |\n| UW | 3 | 8 | 315 | 120 |\n| SCP1 | 6 | 2 | 896 | 268 |\n| SCP2 | 7 | 2 | 1,152 | 200 |\n| JV | 12 | 9 | 29 | 270 |\n| SAD | 13 | 10 | 93 | 6599 |\n| HB | 61 | 2 | 405 | 204 |\n| FD | 144 | 2 | 62 | 5890 |\n| PS | **963** | 7 | 144 | 267 |\n| EW | 6 | 5 | **17984** | 128 |\n\nTable 4 in our paper presents the performance comparison of ViTST against six baseline methods designed for regular time series classification. From the table, we can see that our approach achieves consistently strong performance. Its average performance over the ten datasets are second-best, only slightly lower than TST. Notably, PS has the highest number of variables at 963, while EW boasts a longest sequence length of 17,984. Our approach still achieves competitive results on both, suggesting its effectiveness in handling time series with massive variables and/or long sequence.\nIn summary, we assessed our technique on three datasets for irregularly sampled classification and ten for regular time series, covering a range of ''shapes''. Our approach consistently achieve competitive results. It's important to emphasize that most methods tailored for irregularly sampled time series often struggle with regularly sampled data, and vice versa.\n\n**Response to Q1: Have you tested multi-class classification tasks and/or regression tasks?**\n\nYes, we tested multi-class classification tasks on multiple datasets, as shown in table above. EC, UW, JV, SAD, PS, EW, and PAM datatsets are all used for multi-class classification.\n\n**References**\n\n[1] Bagnall, Anthony, et al. \"The UEA multivariate time series classification archive, 2018.\" arXiv preprint arXiv:1811.00075 (2018)."}, {"review_id": "BLise30WQU", "reviewer": "Reviewer_GiK3", "summary": "This paper describes a surprisingly simple and effective approach for applying computer vision Transformer models to time series classification. Multivariate time series inputs are plotted in a grid to produce images that are used to fine-tune pretrained vision Transformer models. This approach achieves state-of-the-art results on irregularly sampled datasets and competitive results on regularly sampled datasets. Aside from the empirical efficacy, the results are relevant to understanding the capabilities of Transformers and the effectiveness of cross-domain pretraining.", "questions": "How were line colours selected? If random, what distribution of colours was used? Is there any justification for the specific choice made?\n\nFor your self-supervised learning experiments, did you start with a pretrained model?", "limitations": "Limitations are not explicitly discussed. No societal impact concerns come to mind, but I think technical limitations should have been addressed. These could include comparing training and inference resource requirements to other models and acknowledging points like those I raised in the Weaknesses section if the authors agree they are limitations.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "I think the basic finding of the paper - that pre-trained vision Transformers are effective at performing time series classification - is original and significant. It raises very interesting questions about the general efficacy of pre-trained Transformers and the relationship between time series and visual data that could inform future work. In a practical sense, I have some concerns detailed below but I generally appreciate the simplicity and efficacy of the proposed approach.\n\nThe basic results are convincing and the ablation results are extensive and informative. Most of the questions I would have had about the model are already addressed in the main paper or Appendix.\n\nThe paper was well-written and clear. Results are reproducible since full code was provided.", "weaknesses": "To use the proposed method requires making a number of somewhat arbitrary decisions on how to convert time series to images: what layout to use, what order to put the time series in, what colours to use, line width/style, etc. The results in Table 3 and the Appendix show that decisions like these can affect performance even though they don't strictly change the information contained in the input. While these factors can be empirically tuned, the lack of robustness to seemingly arbitrary choices is concerning.\n\nIn particular, the use of matplotlib to generate inputs makes the input construction somewhat poorly controlled. While I appreciate the simplicity of using an existing library, this leaves many implementation details about how exactly plots are rendered up to the library. Again, given that purely visual variations can affect model performance, it's not clear how important these implementation details are. I would have preferred to see a more precise approach that formally determined what numeric value is assigned to each pixel, making these details apparent.\n\nThe range of irregularly sampled time series datasets is somewhat small to draw general conclusions from, but this is in line with existing work in this area.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's recognition of the value and strengths of our work, as well as the constructive feedback and suggestions. We aim to address your concerns in our response as follows:\n\n**Response to W1: Robustness to seemingly arbitrary choices**\n\nOur approach does involve several decisions during the image transformation. We have conducted ablation studies on the choice of layout. While we have already conducted ablation studies concerning the layout choice, we've further expanded our investigation to cover additional parameters.\n\n**Order and Color:**\nOur default approach orders the variables based on the number of observations within each. For comparison, we shuffled this order three times and assessed the performance. We manually assigned colors, with the principle of making adjacent line graphs have distinguishable colors. Likewise, we randomly selected colors over 3 times for comparison. The results are provided below. \n\nOn the P19 and P12 datasets which have varying observations across different variables, the sorted order outperforms randomly shuffled orders. On the PAM dataset, where each variable has a consistent number of observations, the order doesn't have a significant impact.\nRegarding color selection, our manual choices weren't optimal for the P19 dataset; random color selections yielded superior results, boosting the AUPRC score to 54.5. The PAM dataset, however, remained less affected by changes in line color.\n\n| P19 | AUROC | AUPRC |\n| --- | --- | --- |\n| Default | 89.4$\\pm$1.9 | 52.8$\\pm$3.8 |\n| Random order 1 | 88.3$\\pm$1.8 | 49.9$\\pm$3.2 |\n| Random order 2 | 88.2$\\pm$1.6 | 51.0$\\pm$3.9 |\n| Random order 3 | 88.5$\\pm$2.3 | 51.9$\\pm$2.6 |\n| Random color 1 | 89.3$\\pm$1.4 | 53.6$\\pm$1.9 |\n| Random color 2 | 89.1$\\pm$1.9 | 54.5$\\pm$3.5 |\n| Random color 3 | 88.9$\\pm$2.1 | 52.9$\\pm$2.7 |\n\n| P12 | AUROC | AUPRC |\n| --- | --- | --- |\n| Default | 85.6$\\pm$1.1 | 49.8$\\pm$2.5 |\n| Random order 1 | 84.3$\\pm$2.2 | 48.0$\\pm$4.5 |\n| Random order 2 | 84.2$\\pm$1.7 | 47.6$\\pm$3.7 |\n| Random order 3 | 83.9$\\pm$1.5 | 46.9$\\pm$4.0 |\n| Random color 1 | 85.0$\\pm$1.6 | 49.3$\\pm$3.7 |\n| Random color 2 | 83.9$\\pm$1.5 | 48.6$\\pm$2.7 |\n| Random color 3 | 85.2$\\pm$1.9 | 49.2$\\pm$3.2 |\n\n| PAM | Accuracy | Precision | Recall | F1 Score |\n| --- | --- | --- | --- | --- |\n| Default | 96.1$\\pm$0.7 | 96.8$\\pm$1.1 | 96.5$\\pm$0.7 | 96.6$\\pm$0.9 |\n| Random color 1 | 94.5$\\pm$1.3 | 95.8$\\pm$0.8 | 95.0$\\pm$1.2 | 95.3$\\pm$1.0 | \n| Random color 2 | 95.2$\\pm$1.5 | 96.5$\\pm$1.1 | 95.2$\\pm$1.4 | 95.8$\\pm$1.3 |\n| Random color 3 | 95.4$\\pm$1.7 | 96.6$\\pm$0.9 | 95.9$\\pm$1.5 | 95.9$\\pm$1.1 |\n| Random order 1 | 95.5$\\pm$1.0 | 96.8$\\pm$0.8 | 95.7$\\pm$1.0 | 96.2$\\pm$0.9 |\n| Random order 2 | 95.2$\\pm$0.7 | 96.6$\\pm$0.6 | 95.6$\\pm$0.8 | 96.0$\\pm$0.7 |\n| Random order 3 | 95.4$\\pm$0.8 | 96.6$\\pm$0.7 | 95.8$\\pm$1.1 | 96.1$\\pm$0.7 |\n\n**Line width/style and marker size/style:**\nWe configured line style to solid, and marker as '*'. For the P12 and P19 datasets, the line width and the marker size are set to 1 and 2, while for the PAM dataset, they are set to 0.5 and 0.5. The guiding principle is to optimize the line graphs for human visualization, ensuring that the lines effectively show trends and markers clearly indicate observed data without obscuring the lines. We also tested the other configuration. Due to limited space, we only show the results on P19 here.\n\n| Line style | Line width | Marker | Marker size | AUROC | AUPRC |\n| --- | --- | --- | --- | --- | --- |\n| Solid | 1 | * | 2 | 89.4$\\pm$1.9 | 52.8$\\pm$3.8 |\n| Dotted | 1 | * | 2 | 88.7$\\pm$2.0 | 53.3$\\pm$3.4 |\n| Dashed | 1 | * | 2 | 88.5$\\pm$2.8 | 53.3$\\pm$4.2 |\n| Solid | 2 | * | 2 | 88.5$\\pm$2.3 | 53.6$\\pm$3.4 |\n| Solid | 3 | * | 2 | 88.9$\\pm$1.9 | 52.8$\\pm$3.2 |\n| Solid | 1 | o | 2 | 88.8$\\pm$2.6 | 53.2$\\pm$3.8 |\n| Solid | 1 | ^ | 2 | 88.8$\\pm$2.2 | 53.1$\\pm$4.0 |\n| Solid | 1 | * | 1 | 88.6$\\pm$2.4 | 52.6$\\pm$3.6 |\n| Solid | 1 | * | 3 | 89.2$\\pm$2.1 | 53.1$\\pm$4.2 |\n\nFrom the result, we can see that the line style has a significant impact on the performance, while markers do not. As for other factors, a good combination of line width and marker size might lead to better performance. Our approach can achieve competitive results in most settings.\n\n**Response to W2: more precise approach for image rendering.**\n\nWe greatly value your insightful feedback. Our current study is a pivotal first step in investigating the potential of vision transformers applied to images transformed from time series. Currently, we've adopted a straightforward approach using matplotlib for image rendering, and our results suggest the feasibility and potential of this method. In light of your suggestion, we are considering directly assigning pixel values for image rendering, mirroring prior work on synthetic images used in vision model pre-training [1]. We leave it as future work.\n\n**Response to Q1: How were line colours selected?**\n\nAs highlighted in our response to W1, for datasets with a limited number of variables, we manually selected the colors, ensuring that adjacent line graphs had distinguishable colors. For datasets with a larger number of variables, we divided the RGB value range into different slices and shuffled them, ensuring neighboring time series have distinct RGB values.\n\n**Response to Q2: For your self-supervised learning experiments, did you start with a pretrained model?**\n\nYes, we start with the pre-trained checkpoint. It is an exploration of the potential of integrating vision techniques into the time series domain. Similar to masked image modeling work [2] in CV, we start with the pre-trained checkpoint and apply self-supervised learning.\n\n**References**\n\n[1] Kataoka, Hirokatsu, et al. \"Pre-training without natural images.\" Proceedings of the Asian Conference on Computer Vision. 2020.\n\n[2] Xie, Zhenda, et al. \"Simmim: A simple framework for masked image modeling.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n"}, {"review_id": "e0EIQnerUM", "reviewer": "Reviewer_iUCn", "summary": "The paper focuses on learning from irregularly sampled time series data. The paper presents a simple approach that converts irregularly sampled time series into an image where different input channels are line graphs. The converted images are then modeled using a standard Transformer model. Experiments show that the proposed approach outperforms the SOTA approaches on several datasets. \n", "questions": "I have mentioned my concerns in the weaknesses section.", "limitations": "No.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper focuses on the task of learning from irregularly sampled data which is important in many domains.\n- Experimental results show the effectiveness of the approach when compared to baselines and other recent approaches.\n- The paper is clear and well written.\n", "weaknesses": "- The irregularly sampled time series data are represented by a line graph where observed points are connected using a straight line which is a very adhoc way to deal with missing values and irregularity.\n- The novelty is marginal as the proposed approach mainly uses a standard Transformer model where time series data is fed as images.\n- The standard deviation of results in Table 2 seems very high. It is not immediately clear if the proposed approach achieves statistically significant performance or it's just noise.\n- The proposed approaches uses static features present with the data. It is not immediately clear how these static features were used with the baseline approaches. \n- The paper notes that all the methods were trained for 20 epochs. Did they all converge in 20 epochs? The baseline approach should be trained till convergence. \n- The paper is missing experiments on MIMIC-III dataset, most commonly used dataset in this space.\n- The paper is missing comparisons with recent ODE based approaches which achieve SOTA performance. \n- Does the size of image change if the sequence length changes?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable feedback. Below is our response addressing your concerns.\n\n**Response to W1: adhoc way to deal with missing values and irregularity**\n\nWhile our method may seem ad-hoc, it is simple and notably effective. It largely simplifies model design for irregular time series modeling and bridges time series analysis with the computer vision domain. As an initial exploration, we demonstrate such an approach is feasible and promising. Further exploration of better way to handle missing data is encouraged.\n\n**Response to W2: marginal novelty as it mainly uses a standard Transformer model**\n\nOur paper primarily demonstrates that the pre-trained vision transformers, prevalent in computer vision tasks, can be easily adapted for time series classification by representing these series as images. Our approach is simple, effective, and general. It not only largely simplifies dedicated model design for irregular time series modeling but also bridges the time series domain with the computer vision domain. \nWe recognize the potential of developing vision transformers tailored for time series modeling as a promising direction for future improvement. However, our current work represents the critical first step and demonstrates the feasibility and promise of this approach.\n\n**Response to W3: standard deviation of results in Table 2**\n\nAs shown in Table 2, our approach exhibits the lowest standard deviation among all compared methods, except for Transformer-mean on the P19 dataset.\n\n**Response to W4: how the static feature is used in the baseline approaches**\n\nThe static feature is converted into a vector and incorporated with the time series data for modeling in the baseline approaches. In our experiment, we convert the static feature into natural language sentences and use Roberta to decode it. \n\nAs mentioned in B2 in the Appendix, we have also tried to convert the static feature into a vector and utilize MLP to encode it. Specifically, we used a 2-layer MLP with a hidden dimension of 128 and an output dimension of 96 to encode the static feature. The obtained output is concatenated with the representation learned by the vision transformer to make the prediction. We also presented the performance of our approach without using static features (only a Swin to model time series image data). The corresponding results for these experiments are detailed below.\n\n| P19 | AUROC | AUPRC | \n| --- | --- | --- |\n| Swin | 89.4$\\pm$1.8 | 50.2$\\pm$3.0 |\n| Swin-MLP | 88.6$\\pm$1.3 | 51.4 $\\pm$3.7 |\n| Swin-Roberta | 89.4$\\pm$1.9 | 52.8$\\pm$3.8 |\n\n| P12 | AUROC | AUPRC | \n| --- | --- | --- |\n| Swin | 84.3$\\pm$0.6 | 49.3$\\pm$3.7 |\n| Swin-MLP | 84.6$\\pm$0.9 | 48.7 $\\pm$3.2 |\n| Swin-Roberta | 85.6$\\pm$1.1 | 49.8$\\pm$2.5 |\n\nAs can be seen, when both our method and the baselines use the vector converted from static feature for modeling, our approach still outperforms the baselines. Utilizing a language model to encode patient information, which preserves its inherent meaning, appears more effective than simply employing an MLP to process vectorized features. Furthermore, leveraging a language model may offer broader applicability, especially in contexts with textual data like clinical notes.\n\n**Response to W5: did all the baseline coverage in 20 epochs**\nAll baseline models were trained for a maximum of 20 epochs and converged within this period. This setup is consistent with previous work [1].\n\n**Response to  W6/7: missing experiment on MIMIC-III and comparison with recent ODE-based methods**\nWe will endeavor to incorporate experiments on MIMIC-III and comparisons with recent ODE-based methods in our revised version.\n\n**Response to W8: Does the size of image change if the sequence length changes?**\n\nNo, the sequence length will not directly influence the size of images. Regardless of the length, any time series can be fitted into a line graph in a grid cell. The dimensions of this grid, which are user-defined, will determine the overall image size. To illustrate, let's consider the EW dataset detailed in Section 4.4. It has a sequence length of 17984 with 6 variables. This can be represented in a 2x3 grid. We choose the grid size of 128x128, resulting in an image size of 256x384. However, one can also choose any other grid size and subsequently change the corresponding image size.\n\n[1] Zhang, Xiang, et al. \"Graph-guided network for irregularly sampled multivariate time series.\" arXiv preprint arXiv:2110.05357 (2021).\n"}, {"review_id": "Vy9R2JC8kg", "reviewer": "Reviewer_kdpF", "summary": "The paper investigates the use of pre-trained Vision Transformers (ViT) for both regularly and irregularly sampled time series classification. Two primary aspects are discussed: Transforming time series into images and performing time series classification using pre-trained ViT. Through a series of experiments, the authors illustrate the feasibility of this research direction.", "questions": "+ What distinguishes the use of ViT for time series modeling from other foundational models like language or acoustic models (e.g., Voice2Series)? Are there unique advantages, beyond performance, to using ViT?\n+ What theoretical factors contribute to such a successful cross-domain transfer? According to the experiments, some factors like line colors may significantly affect the model performance. A follow-up question is how to claim the robustness of the proposed framework in general time series classification when changing line shapes, colors, or even time series datasets to other domains other than healthcare.\n+ What types of interpolation are used in the experiments?\n+ What is the motivation behind Section 4.5? Is it for a second pre-training based on pre-trained ViT or is it for pre-training from scratch?\n+ In the transformation of time series to images, why do the authors opt to combine different univariate time series into a single image, as opposed to potential alternative implementations like introducing an additional layer to fuse different input images before feeding them into ViT?", "limitations": "The authors have not adequately addressed the limitations and potential negative societal impacts of their work.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is well-organized and presents an engaging perspective on time series classification, particularly focusing on the potential of ViT in relevant tasks. The authors conducted extensive empirical evaluations to showcase the effectiveness of the proposed framework, which is a strong aspect of the work.", "weaknesses": "+ The paper falls short in providing an in-depth analysis of several critical aspects, such as the advantages of using ViT over other foundational models for time series, factors contributing to successful cross-domain transfer when employing ViT for time series classification, and how key design factors impact performance. See the listed questions for details.\n+ In addition, some crucial technical details are missing. For example, it is unclear which layers in ViT are frozen during downstream fine-tuning. Justification for the use of line graphs over other visualization strategies (e.g., frequency maps) is not adequately discussed in Section 3.1. The model's efficiency compared to specialized models in time series modeling is also not clear.\n+ There are some writing issues. For example, the title should be “…for Irregularly Sampled Time Series Classification” unless the effectiveness of ViTST is evident on other time series tasks such as forecasting and imputation. Besides, it is unknown whether the backbone ViT model are frozen in Fig. 2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your valuable feedback. Here is our response to address your concerns:\n\n**Response to weakness**:\n - **Layer frozen**: In our implementation, we did not freeze any of the layers. All the parameters are tunable during fine-tuning on the time series dataset. \n - **Visualization strategy**: Our primary is to showcase the potential of a simple and intuitive approach that uses vision transformers for time series modeling. Line graphs were selected as they are straightforward for humans to interpret. We show that such a strategy has already achieved competitive results. Our current work serves as a pivotal first step to exploring the adaptation of pre-trained vision transformers for time series modeling, and we demonstrate its feasibility. We value the potential of exploring other visualization techniques to further enhance the approach in future work.\n - **Model Efficiency**: we present the performance comparison of our method with several representative specialized methods on the irregularly sampled time series dataset in Table 2 and Figure 3. We also tested on ten more diverse time series classification datasets as described in Section 4.4. The results from these experiments indicate the efficacy and versatility of our approach in addressing a diverse range of time series shapes and types.\n - **Writting issues**: Thank you for pointing it out. We will change the title as suggested. As for Figure 2, none of the layers or parameters are frozen during fine-tuning on the time series dataset. We will add a notification for clarity.\n\n**Response to questions**:\n - **Comparison with other fundamental methods for time series modeling**: Our work aims to showcase that vision transformers can be adapted for time series modeling. We do not claim it is the optimal or only foundation model that could work for this purpose. However, compared with language models and voice models (Voice2Series) for time series modeling, our vision-based approach might offer the following advantages beyond performance: (1) **Simplicity and Compression**: Translating time series data into images provides a more compressed and straightforward representation than converting them into language. In our preliminary experiments, we attempted to transform time series data into text. We observed challenges, especially with multivariate time series that have a significant number of variables or long sequences. This is because most of affordable language models have a context window ranging less than 4096 tokens. Even if each observation is considered a single token, accommodating time series that surpasses this context window size is problematic. In contrast, time series data of any dimension can seamlessly fit into a single image, unconstrained by length or magnitude. (2) **Generality**: While Voice2Series showcases potential in handling univariate time series, its applicability for multivariate and irregularly sampled time series remains unclear. (3) **Maturity of Domain**: using our vision-based method provides the opportunity to leverage well-studied techniques in the established computer vision field. \n - **Theoretical factors contribute to successful transfer**: The success is based on the hypothesis that the model pre-trained on large image datasets can capture generic features within the images, which can then be fine-tuned for specific tasks like time series classification. Our experiments indicate that certain factors within the images, such as line colors/styles/width and grid orders (more details can be found [here](https://openreview.net/forum?id=ZmeAoWQqe0&noteId=13G2qHdC3K)), and also the pre-training of vision transformers do influence performance. We recognize the need for a more comprehensive theoretical analysis in future studies. This work represents the initial exploration of utilizing vision transformers for time series modeling, and our results demonstrate its feasibility and promise.\n- **Robustness to general time series classification tasks**: we conducted experiments on two healthcare datasets (P19/P12), a human activity dataset PAM, and also ten general time series classification datasets as outlined in Section 4.4. In total, we assessed our method on 13 distinct datasets, employing distinguished image rendering settings for each dataset. Our results show that as long as the images are rendered with the same setting in the training set and evaluation set, the performance of our approach is consistently competitive.\n - **Interpolation**: Taking the minimalist approach, our approach employs linear interpolation, i.e., using straight lines to connect consecutive data points on the line graph. The main focus of our work is to show that adapting vision transformers for modeling time series line graphs work, and our current results have proven that. However, we value the potential of exploring other interpolation methods to further enhance the performance.\n - **Motivation behind Section 4.5**: As mentioned above, our approach bridges the time series and computer vision domain, opening up the possibility of leveraging well-studied computer vision techniques into the time series domain. Therefore, in Section 4.5, we explored the prevalent masked image modeling method for time series modeling. Likewise, we started the self-supervised learning with pre-trained checkpoints.\n - **Motivation of image combination**: In our approach, we plot the time series of each variable, which can be seen as a sub-image. These sub-images are assembled into a single \"super image.\". In our preliminary experiment, we tested obtaining separate image representations for each sub-image and concatenating and feeding them into a prediction layer. However, this approach consistently underperformed the unified image processing technique. We hypothesize that integrating them into a single image enables the model to discern more intricate correlations among different variables directly from raw features."}, {"review_id": "mjtkJlzfnG", "reviewer": "Reviewer_BBUf", "summary": "This paper explores a very interesting direction to represent time series as plotted images and then stack vision transformers to perform representation learning. Following this idea, this paper has conducted empirical studies on some classification benchmarks of both irregularly sampled time series and regularly sampled ones. By leveraging this new data representation, the proposed method has obtained the state-of-the-art performance on three irregular time-series benchmarks and also produced competitive results on multiple regular time-series dataests.", "questions": "See the weakness part.", "limitations": "The major limitation is that we are not clear many other critical factors ensuring the success of ViTST and whether it is robust to different plotting setups.", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "In my view, the biggest strength of this paper is the first successful demonstration of applying image-driven methods on representation learning of irregularly sampled time series. While after reading the related work section, I have also learned that this paper is not the first to represent time series as real images in general. Neverthess, I think the successful demonstration of image-based time-series representation learning on time-series classification tasks is interesting and has sufficient novelty.", "weaknesses": "However, I do have several concerns on the experimental results at the current stage.\n\n- Given the relatively high performance variations when employing different plotting configurations and specific early stopping epochs on different datasets, I would like to know the detailed hyper-parameter tuning procedure when you develop ViTST. Are you tuning these options directly based on the feedback from the test set? If not, can you elaborate on how to divide the validation part and please report the details of performance on both validation and test sets if possible.\n- This paper includes some ablation tests, but I think the existing results are far from enough to convince me about the robustness of ViTST. For example, as table 3 shows, this method is sensitive to the existences of interpolation, markers, colors. I want to know that whether ViTST is robust to 1) different line styles and widths given the existence of interpolation, 2) different marker sizes and types given the existence of markers, and 3) different color permutations assigned to multivariate time series. My central confusions lie in which aspects play indispenable roles in making ViTST work and whether ViTST is robust to those irrelevant options. Moreover, I would like to see more ablation results about grid layouts and image sizes. Tables 2, 3, 4 only cover a few, and there seem to large performance variations, too.\n- Another question bothering me is that why ViTST can obtain the best performance for irregularly sampled time series but is only roughly comparable to existing solutions for regular time series.\n- I have tried the code but cannot reproduce the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our sincere gratitude for the valuable feedback and constructive comments. Please find our response addressing the concerns below.\n\n**Response to W1: Detailed hyper-parameter tuning procedure and Robustness test**\n\nThe training/validation/test sets are randomly split. Initially, our selections for line style, line width, marker size, and colors were based on heuristic methods, with the goal of clearly illustrating patterns in human observation (on the training set). Following this, we adjusted the hyper-parameters based on the performance observed on the validation set. \nThe performance of the validation and test set along training process on the P12 dataset is as follows:\n| Steps | Training loss | Validation AUROC | Validation AUPRC | Test AUROC | Test AUPRC | \n| --- | --- | --- | --- | --- | --- |\n| 40 | 0.35 | 0.6988 | 0.269 | 0.6502 | 0.2098 |\n| 80 | 0.5951 | 0.7843 | 0.373 | 0.7953 | 0.3698 |\n| 120 | 0.5066 | 0.8124 | 0.3915 | 0.8286 | 0.4113 |\n| 160 | 0.4837 | 0.8252 | 0.4091 | 0.8409 | 0.4686 | \n| 200 | 0.4847 | 0.8304 | 0.4471 | 0.8531| 0.4857 | \n| 240 | 0.4395 | 0.8336 | 0.4569 | 0.8574 | 0.4781 |\n| 280 | 0.4151 | **0.8357** | 0.4502 | 0.8578 | 0.4881 |\n| 320 | 0.4017 | 0.8344 | 0.4443 | 0.8558 | 0.4882 |\n...\n\nWe choose the checkpoint that achieves the best AUROC score on the validation set for testing.\n\nThe robustness study on choices of image creations are shown below:\n\n**Color:**\nWe manually assigned colors, with the principle of making adjacent line graphs have distinguishable colors. For comparison, we randomly selected colors over 3 times. The results are provided below. \nOur manual choices weren't optimal for the P19 dataset; random color selections yielded superior results, boosting the AUPRC score to 54.5. The PAM dataset, however, remained less affected by changes in line color.\n\n| P19 | AUROC | AUPRC |\n| --- | --- | --- |\n| Default | 89.4$\\pm$1.9 | 52.8$\\pm$3.8 |\n| Random color 1 | 89.3$\\pm$1.4 | 53.6$\\pm$1.9 |\n| Random color 2 | 89.1$\\pm$1.9 | 54.5$\\pm$3.5 |\n| Random color 3 | 88.9$\\pm$2.1 | 52.9$\\pm$2.7 |\n\n| P12 | AUROC | AUPRC |\n| --- | --- | --- |\n| Default | 85.6$\\pm$1.1 | 49.8$\\pm$2.5 |\n| Random color 1 | 85.0$\\pm$1.6 | 49.3$\\pm$3.7 |\n| Random color 2 | 83.9$\\pm$1.5 | 48.6$\\pm$2.7 |\n| Random color 3 | 85.2$\\pm$1.9 | 49.2$\\pm$3.2 |\n\n| PAM | Accuracy | Precision | Recall | F1 Score |\n| --- | --- | --- | --- | --- |\n| Default | 96.1$\\pm$0.7 | 96.8$\\pm$1.1 | 96.5$\\pm$0.7 | 96.6$\\pm$0.9 |\n| Random color 1 | 94.5$\\pm$1.3 | 95.8$\\pm$0.8 | 95.0$\\pm$1.2 | 95.3$\\pm$1.0 | \n| Random color 2 | 95.2$\\pm$1.5 | 96.5$\\pm$1.1 | 95.2$\\pm$1.4 | 95.8$\\pm$1.3 |\n| Random color 3 | 95.4$\\pm$1.7 | 96.6$\\pm$0.9 | 95.9$\\pm$1.5 | 95.9$\\pm$1.1 |\n| Random order 1 | 95.5$\\pm$1.0 | 96.8$\\pm$0.8 | 95.7$\\pm$1.0 | 96.2$\\pm$0.9 |\n| Random order 2 | 95.2$\\pm$0.7 | 96.6$\\pm$0.6 | 95.6$\\pm$0.8 | 96.0$\\pm$0.7 |\n| Random order 3 | 95.4$\\pm$0.8 | 96.6$\\pm$0.7 | 95.8$\\pm$1.1 | 96.1$\\pm$0.7 |\n\n**Line width/style and marker size/style:**\nWe configured line style to solid, and marker as '*'. For the P12 and P19 datasets, the line width and the marker size are set to 1 and 2, while for the PAM dataset, they are set to 0.5 and 0.5. The guiding principle is to optimize the line graphs for human visualization, ensuring that the lines effectively show trends and markers clearly indicate observed data without obscuring the lines. We also tested the other configuration. Due to limited space, we only show the results on P19 here.\n\n| Line style | Line width | Marker | Marker size | AUROC | AUPRC |\n| --- | --- | --- | --- | --- | --- |\n| Solid | 1 | * | 2 | 89.4$\\pm$1.9 | 52.8$\\pm$3.8 |\n| Dotted | 1 | * | 2 | 88.7$\\pm$2.0 | 53.3$\\pm$3.4 |\n| Dashed | 1 | * | 2 | 88.5$\\pm$2.8 | 53.3$\\pm$4.2 |\n| Solid | 2 | * | 2 | 88.5$\\pm$2.3 | 53.6$\\pm$3.4 |\n| Solid | 3 | * | 2 | 88.9$\\pm$1.9 | 52.8$\\pm$3.2 |\n| Solid | 1 | o | 2 | 88.8$\\pm$2.6 | 53.2$\\pm$3.8 |\n| Solid | 1 | ^ | 2 | 88.8$\\pm$2.2 | 53.1$\\pm$4.0 |\n| Solid | 1 | * | 1 | 88.6$\\pm$2.4 | 52.6$\\pm$3.6 |\n| Solid | 1 | * | 3 | 89.2$\\pm$2.1 | 53.1$\\pm$4.2 |\n\nFrom the result, we can see that the line style has a significant impact on the performance, whereas markers appear to have minimal impact. This observation is consistent with our initial heuristic that a solid line would likely be more effective, and markers might not make big influence. As for other factors, an optimal combination of line and marker could potentially enhance visualization, thereby improving performance. We did not conduct comprehensive hyperparameter tuning for these variables. As such, it seems that the peak performance on P19 has not yet been fully explored.\n\n**Response to Q: why ViTST can obtain the best performance for irregularly sampled time series but is only roughly comparable to existing solutions for regular time series?**\nAs shown in Table 4, our ViTST's average accuracy is only behind TST, which is a transformer that directly operates on numerical time series. Given the shared architecture between ViTST and TST, we assume their performance gap might be attributed to the fact that extracting patterns from complete fixed-sized numerical time series data is more direct and introduces less noise than operating on the image data transformed from time series.  However, when there is missing data and imputation is required to convert the data into fixed-sized input can introduce much noise, leading to suboptimal performance of standard numerical-based specialized methods. However, our method might not be such sensitive to the missing part in the line graph. By simple linear interpolation, the model might be able to approximate the pattern and trend from the line graph, ensuring that ViTST remains effective even with irregularly sampled data."}]

Author rebuttal:

Meta-review: The idea behind this paper is simple. When making predictions with time series data, instead of using a time series model, use matplotlib to plot the time series on an image and then use a ViT to predict outcomes. The paper studies variations on this problem including the effect of pretraining the vision transformers on Imagenet, comparisons against several time series models and datasets (in healthcare and outside of it) showcase that the model can capture signal from time series data. Most reviewers liked the work, in particular its simplicity and broad applicability.

During the discussion phase, the reviewers asked for robustness results obtained by varying plotting artefacts which the rebuttal provided, reviewers also requested predictive models on MIMIC which was not provided in the rebuttal.
The claims in the paper were requested to be toned down given the paper does not present a rigorous study to quantify the kinds of patterns it works well/less well on.
Given the simplicity of the proposal, two reviewers independently ran the code to check if it works and it reproduces the reported results. After some back and forth, the reviewers verified that the work does run (using the authors' instructions) and produce useful results but highlighted that the codebase is not well organized and the results are not SOTA but rather in the same ballpark as existing prior work.
More importantly, the reviewers found that the model was setup to work using statistics obtained on the train and test set which violates the typical practice of normalization using training set statistics. This is an oversight and should be fixed in all the experiments (the rebuttal focused on P19) since it could lead to artificial inflation of results and unfairly disadvantages other baselines. This is also a practical concern about the utility of the model. There are scenarios where test patients' biomarkers are significantly above or below what happens to patients at training time. When this happens, without using test statistics for normalization (which is incorrect from a modeling perspective), there will invariably time series data at test time that goes outside the image (out of bounds) and remain invisible to the model.
At the end of the discussion period, this remains a truly borderline paper. the reviewers remained split along whether the paper should be accepted or not where one can argue for acceptance based on the simplicity of ideas or ask for a revamp based on cleaning up the code further.

On the whole I believe this is a very strong and simple idea which will have an impact even if it is imperfect.

The manuscript should be updated (in terms of toning down the claims made) and incorporated with changes made to the experimental setup to correct for the normalization issue (which seems like it might affect all the experimental results). Attention should be paid to the code so that it is accessible enough to reliably reproduce the reported results.
Decision: Accept (poster)