SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Comparison to state-of-the-art. We compare our approach with several state-of-the-art methods that are specifically designed for irregularly sampled time series, including Transformer [38], Trans-mean (Transformer with an imputation method that replaces the missing value with the average observed value of the variable), GRU-D [3], SeFT [16], mTAND [34], IP-Net [33], and Raindrop [49]. In addition, we also compared our method with two approaches initially designed for forecasting tasks, namely DGM $^2$ -O [42] and MTGNN [43]. The implementation and hyperparameter settings of these baselines were kept consistent with those used in Raindrop [49]. Specifically, a batch size of 128 was employed, and all compared models were trained for 20 epochs. To ensure fairness in our evaluation, we averaged the performance of each method over 5 data splits that were kept consistent across all compared approaches.
As shown in Table 2, our proposed approach demonstrates strong performance against the specialized state-of-the-art algorithms on all three datasets. Specifically, on the P19 and P12 datasets, ViTST achieves improvements of  $2.2\%$  and  $0.7\%$  in absolute AUROC points, and  $1.3\%$  and  $2.9\%$  in absolute AUPRC points over the state-of-the-art results, respectively. For the PAM dataset, the improvement is even more significant, with an increase of  $7.3\%$  in Accuracy,  $6.3\%$  in Precision,  $6.2\%$  in Recall, and  $6.7\%$  in absolute F1 score points.
Leaving-sensors-out settings. We conducted additional evaluations to assess the performance of our model under more challenging scenarios, where the observations of a subset of sensors (variables) were masked during testing. This setting simulates real-world scenarios when some sensors fail or become unreachable. Following the approach adopted in [49], we experimented with two setups using the PAM dataset: (1) leave-fixed-sensors-out, which drops a fixed set of sensors across all
$^{2}$ https://huggingface.co/microsoft/swin-base-patch4-window7-224-in22k
(a) Leave-fixed-sensors-out
Figure 3: Performance in leave-fixed-sensors-out and leave-random-sensors-out settings on PAM dataset. The x-axis is the "missing ratio" which denotes the ratio of masked variables.
(b) Leave-random-sensors-out
samples and compared methods, and (2) leave-random-sensors-out, which drops sensors randomly. It is important to note that only the observations in the validation and test sets were dropped, while the training set was kept unchanged. To ensure a fair comparison, we dropped the same set of sensors in the leave-fixed-sensors-out setting as in [49].
The results are presented in Fig. 3, from which we observe that our approach consistently achieves the best performance and outperforms all the specialized baselines by a large margin. With the missing ratio increasing from  $10\%$  to  $50\%$ , our approach maintains strong performance, staying above  $80\%$ . In contrast, the comparison baseline shows a marked drop. The advantage of ViTST over the comparison baselines becomes increasingly significant. Even when half of the variables were dropped, our approach was still able to achieve acceptable performance over 80, surpassing the best-performing baseline Raindrop by  $33.1\%$  in Accuracy,  $40.9\%$  in Precision,  $39.4\%$  in Recall, and  $42.8\%$  in F1 score in the leave-fixed-sensors-out setting (all in absolute points). We also notice that the variances in our results are notably lower compared to the baselines. These results suggest that our approach is highly robust to varying degrees of missing observations in time series.
4.3 Additional Analysis
(a) P19
(b) P12
Figure 4: Performance of different backbone vision models on P19, P12, and PAM datasets. We do not use static features for our approach here to exclude their influence.
(c) PAM
Where does the performance come from? Our approach transforms time series into line graph images, allowing the use of vision transformers for time series modeling. We hypothesize that vision transformers can leverage their general-purpose image recognition ability acquired from large-scale pre-training on natural images (such as ImageNet [30]) to capture informative patterns in the line graph images. To validate that, we compared the performance of a pre-trained Swin
Table 3: Ablation studies on different strategies of time series-to-image transformation.
Methods	P19	P12	PAM
AUROC	AUPRC	AUROC	AUPRC	Accuracy	Precision	Recall	F1 score
Default	89.2 ± 2.0	53.1 ± 3.4	85.1 ± 0.8	51.1 ± 4.1	95.8 ± 1.3	96.2 ± 1.1	96.2 ± 1.3	96.5 ± 1.2
w/o interpolation	89.6 ± 2.1	52.9 ± 3.4	85.7 ± 1.0	51.9 ± 3.4	95.6 ± 1.1	96.6 ± 0.9	95.9 ± 1.0	96.2 ± 1.0
w/o markers	89.0 ± 2.1	51.7 ± 2.5	85.3 ± 0.9	50.3 ± 3.2	95.8 ± 1.1	96.9 ± 0.7	96.0 ± 1.0	96.4 ± 0.9
w/o colors	88.8 ± 1.8	51.4 ± 4.1	84.4 ± 0.7	47.0 ± 2.9	95.0 ± 1.0	96.2 ± 0.7	95.3 ± 1.0	95.7 ± 0.9
w/o order	89.3 ± 2.3	52.7 ± 4.5	84.0 ± 1.8	47.8 ± 4.6	-	-	-	-
Transformer with a Swin Transformer trained from scratch, as shown in Fig. 4. The significant drop in performance without pre-training provides evidence that Swin transformer could transfer the knowledge obtained from pre-training on natural images to our synthetic time series line graph images, achieving impressive performance. Nevertheless, the underlying mechanism needs further exploring and probing in future studies.
How do different vision models perform? We benchmarked several backbone vision models within our framework. Specifically, we tried another popular pre-trained vision transformer ViT<sup>3</sup> and a pre-trained CNN-based model, ResNet<sup>4</sup>. The results are presented in Fig. 4. The pre-trained transformer-based ViT and Swin Transformer demonstrate comparable performance, both outperforming the previous state-of-the-art method, Raindrop. In contrast, the pre-trained CNN-based ResNet lagged considerably behind the vision transformer models. This performance gap is consistent with observations in image classification tasks on datasets like ImageNet, where vision transformers have been shown to excel in preserving spatial information compared to conventional CNN models. This advantage enables vision transformers to effectively capture the positions of patches within each line graph sub-image and the entire image and facilitates the modeling of complex dynamics and relationships between variables.
How to create time series line graph images? Using line graphs to visualize time series offers us an intuitive way to interpret the data and adjust the visualization strategy for enhanced clarity and potentially augment the performance. To offer insights on effective time series-to-image transformation, we analyze the effects of several key elements in practice: (1) the default linear interpolation to connect partially observed data points on the line graphs; (2) markers to indicate observed data points; (3) variable-specific colors to differentiate between line graphs representing different variables; (4) the order determined by missing ratios when organizing multiple line graphs in a single image.
The results are presented in Table 3. Given the balanced missing ratios in the PAM dataset, we excluded results without the sorting order. Interestingly, plotting only observed data points without linear interpolation led to better results on P19 and P12 datasets. This could be attributed to the potential inaccuracies introduced by interpolation, blurring distinctions between observed and interpolated points. Additionally, omitting markers complicates the model's task of discerning observed data points from interpolated ones, degrading its performance. The absence of distinctive colors led to the most significant performance drop, underlining the necessity of using varied hues for individual line graphs to help the model to distinguish them. While a specific sorting order may not ensure optimal outcomes across all datasets, it does provide relatively stable results over multiple datasets and evaluation metrics. For the PAM dataset, these nuances seem to have minimal impact, indicating the robustness of our approach against these variations in some scenarios.
Effects of grid layouts and image sizes. We explored the influence of grid layouts and image dimensions on our approach's efficacy. For a fair comparison across grid layouts, we fixed the size of each grid cell as  $64 \times 64$  and altered the grid layouts. As shown in Figure 5, we observed our approach's robustness to variations in grid layouts, with square layout consistently yielding good results across different datasets and metrics, which was particularly evident for the P12 dataset. Regarding image size, when we maintained the grid layout but reduced the overall image dimensions, a noticeable performance decline was observed on the P12 and PAM datasets, which complies with our intuition.
Robustness against varied plotting parameters. To gauge the robustness of our approach against different plotting parameters, we assessed aspects including line style/width and marker style/size,
<sup>3</sup>https://huggingface.co/google/vit-base-patch16-224-21k
4https://huggingface.co/microsoft/resnet-50
(b) P12
Figure 5: Ablation study of the influence of grid layouts and image sizes. For instance,  $4 \times 9$  (256x576) denotes a grid layout of  $4 \times 9$  with an image size of  $256 \times 576$  pixels.
primarily using the P19 dataset. As shown in Table 4, our approach demonstrates robustness against changes in these parameters, maintaining strong performance across different plotting configurations.
What does ViTST capture? To gain insights into the patterns captured by ViTST in the time series line graph images, we analyzed the averaged attention map of a ViTST model with ViT as the backbone, as depicted in Fig. 6. The attention map reveals that the model consistently attends to the informative part, i.e., line graph contours within the image. Furthermore, we observed that the model appropriately focuses on observed data points and areas where the slopes of the lines change. Conversely, flat line graphs that lack dynamic patterns seem to receive less attention. This demonstrates that ViTST might be able to discern between informative and uninformative features in the line graph images, enabling it to extract meaningful patterns.
Table 4: Robustness regarding the style and size of lines and markers. In the brackets, the first element denotes style, and the second represents size.
Line	Marker	AUROC	AUPRC
(solid,1)	(*, 2)	89.2 ± 2.0	53.1 ± 3.4
(dashed,1)	(*, 2)	89.2 ± 2.1	53.7 ± 4.1
(dotted,1)	(*, 2)	89.2 ± 2.1	52.8 ± 4.0
(solid,0.5)	(*, 2)	88.6 ± 1.7	53.0 ± 3.6
(solid,1)	(*, 2)	89.2 ± 2.0	53.1 ± 3.4
(solid,2)	(*, 2)	88.5 ± 2.3	53.6 ± 3.1
(solid,1)	(*, 2)	89.2 ± 2.0	53.1 ± 3.4
(solid,1)	(∧, 2)	89.3 ± 1.9	52.6 ± 4.0
(solid,1)	(o, 2)	89.1 ± 1.9	51.3 ± 4.2
(solid,1)	(*, 1)	88.2 ± 1.4	52.1 ± 4.5
(solid,1)	(*, 2)	89.2 ± 2.0	53.1 ± 3.4
(solid,1)	(*, 3)	88.9 ± 1.9	52.8 ± 3.2
Figure 6: Illustration of the averaged attention map of ViTST.
4.4 Regular Time Series Classification
An advantage of our approach is its ability to model time series of diverse shapes and scales, whether they are regular or irregular. To evaluate the performance of our approach on regular time series data,
Table 5: Performance comparison on regular multivariate time series datasets. Bold indicates the best performer, while underline represents the second best.
Datasets	EC	UW	SCP1	SCP2	JV	SAD	HB	FD	PS	EW	Average
Dataset statistics
#Variables	3	3	6	7	12	13	61	144	963	6	-
Length	1,751	315	896	1,152	29	93	405	62	144	17984	-
Model performances
DTWD	0.323	0.903	0.775	0.539	0.949	0.963	0.717	0.529	0.711	0.618	0.717
LSTM	0.323	0.412	0.689	0.466	0.797	0.319	0.722	0.577	0.399	-	0.523
XGBoost	0.437	0.759	0.846	0.489	0.865	0.696	0.732	0.633	0.983	-	0.727
Rocket	0.452	0.944	0.908	0.533	0.962	0.712	0.756	0.647	0.751	-	0.741
TST	0.326	0.913	0.922	0.604	0.997	0.998	0.776	0.689	0.896	0.748	0.791
ViTST	0.456	0.862	0.898	0.561	0.946	0.985	0.766	0.632	0.913	0.878	0.780
we conducted experiments on ten representative multivariate time series datasets from the UEA Time Series Classification Archive [2]. These datasets exhibit diverse characteristics, as summarized in Table 5. It is worth noting that the PS dataset in our evaluation contains an exceptionally high number of variables (963), while the EW dataset has extremely long time series (17984). We specifically selected these two datasets to assess the effectiveness of our approach in handling large numbers of variables and long time series. We follow [47] to use these baselines for comparison:  $\mathrm{DTW}_D$  which stands for dimension-Dependent DTW combined with dilation-CNN [11], LSTM [13], XGBoost [5], Rocket [7], and a transformer-based TST [47] which operates on fully observed numerical time series.
The performance of our approach on regular time series datasets is consistently strong, as demonstrated in Table 5. With an average accuracy that is second-best and closely aligned with the top-performing baseline method TST, our approach showcases its competitive capabilities. Notably, it excels on challenging datasets PS and EW with massive variables and observation length. These results were achieved using the same image resolution  $(384 \times 384)$  as the other datasets, indicating the effectiveness and efficiency of our approach. The ability of our approach to handle both irregular and regular time series data further emphasizes its versatility and broad applicability.
In this paper, we introduce a novel perspective on modeling irregularly sampled time series. By transforming time series data into line graph images, we could effectively leverage the strengths of pre-trained vision transformers. This approach is straightforward yet effective and versatile, enabling the modeling of time series of diverse characteristics, regardless of irregularity, different structures, and scales. Through extensive experiments, we demonstrate that our approach surpasses state-of-the-art methods designed for irregular time series and maintains strong robustness to varying degrees of missing observations. Additionally, our approach achieves promising results on regular time series data. We envision its potential as a general-purpose framework for various time series tasks. Our results underscore the potential of adapting rapidly advancing computer vision techniques to time series modeling.
In this work, we utilized a straightforward method to image multivariate time series by converting them into line graph images using matplotlib and then saving them as RGB images. While our results are promising and exhibit robustness against variations in the time series-to-image transformation process, there may be alternative ways to visualize the data. This includes potentially more controllable and accurate plotting methods or different image representations beyond line graphs. Our findings also highlight the efficacy of pre-trained vision transformers for time series classification, suggesting that these models might leverage knowledge acquired from pre-training on natural images. Yet, the underlying reasons for their remarkable success still need deeper exploration and investigation. This research serves as a promising starting point in this domain, suggesting various potential directions. We leave these further explorations and investigations for future work.
A More Details on Time Series Line Graph Image Creation
Implementation The time-series-to-image transformation can be implemented using the Matplotlib package<sup>5</sup> with the following few lines of code.
1 def TS2Image(t, v, D, colors, image_height, image_width, grid_height, grid_width):
2 import matplotlib.pyplot as plt
3 plt.figure(figsize=(image_height/100, image_width/100), dpi=100)
4 for d in range(D): # enumerate the multiple variables
5 plt.subplot(grid_height, grid_width, d+1) # position in the grid
6 # plot line graph of variable d
7 plt.plot(t[d], v[d], color=colors[d], linestyle="--", marker="*")
Axis Limits of Line Graphs The axis limits determine the plot area of the line graphs and the range of displayed timestamps and values. By default, we set the limits of the x-axis and y-axis as the ranges of all the observed timestamps and values. However, we found that some extreme observed values for some variables can largely expand the range of the y-axis, causing most plotted points of observations to cluster in a small area and resulting in flat line graphs. Common normalization and standardization methods will not solve this issue, as the relative magnitudes remain unchanged in the created images. We thus tried the following strategies to remove extreme values and narrow the range of the y-axis in our preliminary experiments:
- Interquartile Range (IQR): IQR is one of the most extensively used methods for outlier detection and removal. The interquartile range is calculated based on the first and third quartiles of all the observed values of each variable in the dataset and then used to calculate the upper and lower limits.
- Standard Deviation (SD): The upper and lower boundaries are calculated by taking 3 standard deviations from the mean of observed values for each variable across the dataset. This method usually assumes the data is normally distributed.
- Modified Z-score (MZ): A z-score measures how many standard deviations away a value is from the mean and is similar to the standard deviation method to detect outliers. However, z-scores can be influenced by extreme values, which modified z-scores can better handle. We set the upper and lower limits as the values whose modified z-scores are 3.5 and -3.5.
We show examples of the created images with these strategies in Figure 7.
Table 6: Preliminary experiments on different strategies to decide the line graph limit. The default strategy is to directly set the axis limit as the range of all observed values on the dataset. "IQR", "SD", and "MZS" denote three strategies to remove extreme value, i.e., Interquartile Range, Standard Deviation, and Modified Z-score. The reported numbers are averaged on 5 data splits.
Strategies	P19	P12	PAM
AUROC	AUPRC	AUROC	AUPRC	Accuracy	Precision	Recall	F1 score
Default	89.4 ± 1.9	52.8 ± 3.8	85.6 ± 1.1	49.8 ± 2.5	96.1 ± 0.7	96.8 ± 1.1	96.5 ± 0.7	96.6 ± 0.9
IQR	88.2 ± 0.8	49.6 ± 1.7	84.5 ± 1.1	48.9 ± 2.6	95.9 ± 0.7	96.8 ± 0.7	96.1 ± 0.7	96.4 ± 0.7
SD	87.4 ± 1.6	51.2 ± 3.6	84.6 ± 1.7	47.1 ± 2.9	96.6 ± 0.9	97.1 ± 0.8	97.0 ± 0.6	97.0 ± 0.7
MZS	87.3 ± 1.0	50.8 ± 3.7	84.3 ± 1.4	47.1 ± 2.1	96.0 ± 1.1	96.8 ± 0.99	96.4 ± 0.9	96.6 ± 0.9
The performance comparison of models trained on images created with different strategies is shown in Table 6. We observe that the methods that remove extreme values hurt the performance, except for SD on the PAM dataset. Although these methods narrow the value range and highlight the dynamic patterns of line graphs, they discard the extreme values that might be informative themselves. Therefore, we stick with the default way that sets the axis limits as the range of all observed values.
**Computation cost** We list the inference time (in seconds) of different methods on the test sets of three datasets in Table 7. All the inferences are made in a single Nvidia A6000 GPU. It is observed that our vision-based method consumes more inference time than the non-vision baselines. However, we believe this cost remains within an acceptable range in the context of today's ML practice and medical applications, considering the inference on each sample only costs around 0.01 seconds.
<sup>5</sup>https://matplotlib.org/
(b) P12
(c) PAM
Figure 7: The images created with different strategies for three samples from P19, P12, and PAM dataset, respectively (sample "p000019" for P19, "132548" for P12, and "0" for PAM).
Table 7: The inference time (in seconds) of different methods on the test sets.
Datasets	Trasformers	mTAND	SeFT	Raindrop	MTGNN	DGM2-O	GRU-D	ViTST
P19	0.21	0.52	2.72	3.05	3.62	2.47	31.04	44.51
P12	0.12	0.44	0.97	1.27	1.46	2.80	10.13	12.14
PAM	0.06	0.23	0.89	0.67	1.16	2.98	4.55	5.30
We used the datasets processed by [49], whose details are given below.
P19: PhysioNet Sepsis Early Prediction Challenge 2019. The P19 dataset [29] consists of clinical data for 38,803 patients, and aims to predict whether sepsis will occur within the next 6 hours. The dataset includes 34 irregularly sampled sensors with 8 vital signs and 26 laboratory values for each patient, as well as 6 demographic features. To process the static features, we use templates outlined in Table 8, and utilize a pre-trained Roberta-base model to extract textual features. These textual features are then combined with visual features obtained from the vision transformer to perform binary classification. The dataset is highly imbalanced with only  $4\%$  of samples being positive, and has a missing ratio of  $94.9\%$ .
P12: PhysioNet Mortality Prediction Challenge 2012.  $^{7}$  P12 dataset [12] includes clinical data from 11,988 ICU patients, with 36 irregularly sampled sensor observations and 6 static demographic features provided for each patient. The goal is to predict patient mortality, which is a binary classification task. The dataset is highly imbalanced, with around  $86\%$  of samples being negative. The missing ratio of the dataset is  $88.4\%$ .
PAM: PAMAP2 Physical Activity Monitoring. The PAM dataset originally contains data of 18 physical activities with 9 subjects wearing 3 inertial measurement units. However, to make it suitable for irregular time series classification, [49] excluded the ninth subject due to its short length of sensor readouts, and 10 out of the 18 activities that had less than 500 samples were also excluded. As a result, the task is an 8-way classification with 5,333 samples, each with 600 continuous observations. To simulate the irregular time series setting,  $60\%$  of the observations are randomly removed. No static features are provided, and the 8 categories are approximately balanced. The missing ratio is  $60.0\%$ .
Table 8: Templates for transforming static features to natural language sentences.
Dataset	Static features	Template	Example
P19	Age, Gender, Unit1 (medical ICU), Unit2 (surgery ICU), HospAdmTime; ICULOS (ICU length-of-stay)	A patient is {Age} years old, {Gender}, went to {Unit1&Unit2} {HospAdmTime} hours after hospital admit, had stayed there for {ICU-LOS} hours.	A patient is 65 years old, female, went to the medical ICU 10 hours after hospital admit, had stayed there for 20 hours.
P12	RecordID, Age, Gender, Height (cm), ICUType, Weight (kg)	A patient is {Age} years old, {Gender}, {Height} cm, {Weight} kg, stayed in {ICUType}.	A patient is 48 years old, male, 171 cm, 78 kg, stayed in surgical ICU.
Table 9: Ablation studies on different methods to encode static features.
Methods	P19	P12
AUROC	AUPRC	AUROC	AUPRC
Raindrop	87.0 ± 2.3	51.8 ± 5.5	82.8 ± 1.7	44.0 ± 3.0
Swin	89.4 ± 1.8	50.2 ± 3.0	84.3 ± 0.6	49.3 ± 3.7
Swin-MLP	88.6 ± 1.3	51.4 ± 3.7	84.6 ± 0.9	48.7 ± 3.2
Swin-Roberta	89.4 ± 1.9	52.8 ± 3.8	85.6 ± 1.1	49.8 ± 2.5
B.2 Experiments on Static Features
Time series data is often associated with information from other modalities, such as the textual clinical notes in electronic health records (EHRs) in the healthcare domain. Our approach is naturally suitable for incorporating such information since we convert time series data to images, and thus various vision-language and multi-modal techniques can be utilized to incorporate the visual (time series) information and information from other modalities. For example, the CLIP [27] learns a shared
$^{6}$ https://physionet.org/content/challenge-2019/1.0.0/
<sup>7</sup>https://physionet.org/content/challenge-2012/1.0.0/
<sup>8</sup>https://archive.ics.uci.edu/ml/datasets/pamap2+physical+activity+monitoring