SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zekun Li, Shiyang Li, Xifeng Yan
{zekunli, shiyangli, xyan}@cs.ucsb.edu
Irregularly sampled time series are increasingly prevalent, particularly in medical domains. While various specialized methods have been developed to handle these irregularities, effectively modeling their complex dynamics and pronounced sparsity remains a challenge. This paper introduces a novel perspective by converting irregularly sampled time series into line graph images, then utilizing powerful pretrained vision transformers for time series classification in the same way as image classification. This method not only largely simplifies specialized algorithm designs but also presents the potential to serve as a universal framework for time series modeling. Remarkably, despite its simplicity, our approach outperforms state-of-the-art specialized algorithms on several popular healthcare and human activity datasets. Especially in the rigorous leave-sensors-out setting where a portion of variables is omitted during testing, our method exhibits strong robustness against varying degrees of missing observations, achieving an impressive improvement of  $42.8\%$  in absolute F1 score points over leading specialized baselines even with half the variables masked. Code and data are available at https://github.com/Leezekun/ViTST.
Time series data are ubiquitous in a wide range of domains, including healthcare, finance, traffic, and climate science. With the advances in deep learning architectures such as LSTM [13], Temporal Convolutional Network (TCN) [18], and Transformer [38], numerous algorithms have been developed for time series modeling. However, these methods typically assume fully observed data at regular intervals and fixed-size numerical inputs. Consequently, these methods encounter difficulties when faced with irregularly sampled time series, which consist of a sequence of samples with irregular intervals between their observation times. To address this challenge, highly specialized models have been developed, which require substantial prior knowledge and efforts in model architecture selection and algorithm design [24, 20, 3, 16, 49, 34, 48].
In parallel, pre-trained transformer-based vision models, most notably vision transformers, $^{1}$  have emerged and demonstrated strong abilities in various vision tasks such as image classification and object detection, nearly approaching human-level performance. Motivated by the flexible and effective manner in which humans analyze complex numerical time series data through visualization, we raise the question: Can these powerful pre-trained vision transformers capture temporal patterns in visualized time series data, similar to how humans do?
To investigate this question, we propose a minimalist approach called ViTST (Vision Time Series Transformer), which involves transforming irregularly sampled multivariate time series into line graphs, organizing them into a standard RGB image format, and finetuning a pre-trained vision
<sup>1</sup>In this paper, we use the term "vision transformers" to denote a category of pre-trained transformer-based vision models, such as ViT [9], Swin Transformer [22], DeiT [36], etc.
Figure 1: An illustration of our approach ViTST. The example is from a healthcare dataset P12 [12], which provides the irregularly sampled observations of 36 variables for patients (we only show 4 variables here for simplicity). Each column in the table is an observation of a variable, with the observed time and value. We plot separate line graphs for each variable and arrange them into a single image, which is then fed into the vision transformer for classification.
transformer for classification using the resulting image as input. An illustration of our method can be found in Figure 1.
Line graphs serve as an effective and efficient visualization technique for time series data, regardless of irregularity, structures, and scales. They can capture crucial patterns, such as the temporal dynamics represented within individual line graphs and interrelations between variables throughout separate graphs. Such a visualization technique benefits our approach because it is both simple and intuitively comprehensible to humans, enabling straightforward decisions for time series-to-image transformations. Leveraging vision models for time series modeling in this manner mirrors the concept of prompt engineering, wherein individuals can intuitively comprehend and craft prompts to potentially enhance the processing efficiency of language models.
We conduct a comprehensive investigation and validation of the proposed approach, ViTST, which has demonstrated its superior performance over state-of-the-art (SoTA) methods specifically designed for irregularly sampled time series. Specifically, ViTST exceeded prior SoTA by  $2.2\%$  and  $0.7\%$  in absolute AUROC points, and  $1.3\%$  and  $2.9\%$  in absolute AUPRC points for healthcare datasets P19 [29] and P12 [12], respectively. For the human activity dataset, PAM [28], we observed improvements of  $7.3\%$  in accuracy,  $6.3\%$  in precision,  $6.2\%$  in recall, and  $6.7\%$  in F1 score (absolute points) over existing SoTA methods. Our approach also exhibits strong robustness to missing observations, surpassing previous leading solutions by a notable  $42.8\%$  in absolute F1 score points under the leave-sensors-out scenario, where half of the variables in the test set are masked during testing. Furthermore, when tested on regular time series data including those with varying numbers of variables and extended sequence lengths, ViTST still achieves excellent results comparable to the specialized algorithms designed for regular time series modeling. This underscores the versatility of our approach, as traditional methods designed for regularly sampled time series often struggle with irregularly sampled data, and vice versa.
In summary, the contributions of this work are three-fold: (1) We propose a simple yet highly effective approach for multivariate irregularly sampled time series classification. Despite its simplicity, our approach achieves strong results against the highly specialized SoTA methods. (2) Our approach demonstrates excellent results on both irregular and regular time series data, showcasing its versatility and potential as a general-purpose framework for time series modeling. It offers a robust solution capable of handling diverse time series datasets with varying characteristics. (3) Our work demonstrates the successful transfer of knowledge from vision transformers pre-trained on natural images to synthetic visualized time series line graph images. We anticipate that this will facilitate the utilization of fast-evolving and well-studied computer vision techniques in the time series domains, such as better model architecture [23], data augmentation [32], interpretability [4], and self-supervised pre-training [15].
Irregularly sampled time series. An irregularly sampled time series is a sequence of observations with varying time intervals between them. In a multivariate setting, different variables within the
same time series may not align. These characteristics present a significant challenge to standard time series modeling methods, which usually assume fully observed and regularly sampled data points. A common approach to handling irregular sampling is to convert continuous-time observations into fixed time intervals [24, 20]. To account for the dynamics between observations, several models have been proposed, such as GRU-D [3], which decays the hidden states based on gated recurrent units (GRU) [6]. Similarly, [46] proposed an approach based on multi-directional RNN, which can capture the inter- and intra-steam patterns. Besides the recurrent and differential equation-based model architectures, recent work has explored attention-based models. Transformer [38] is naturally able to handle arbitrary sequences of observations. ATTAIN [50] incorporates attention mechanism with LSTM to model time irregularity. SeFT [16] maps the irregular time series into a set of observations based on differentiable set functions and utilizes an attention mechanism for classification. mTAND [34] learns continuous-time embeddings coupled with a multi-time attention mechanism to deal with continuous-time inputs. UTDE [48] integrates embeddings from mTAND and classical imputed time series with learnable gates to tackle complex temporal patterns. Raindrop [49] models irregularly sampled time series as graphs and utilizes graph neural networks to model relationships between variables. While these methods are specialized for irregular time series, our work explores a simple and general vision transformer-based approach for irregularly sampled time series modeling.
Numerical time series modeling with transformers. Transformers have gained significant attention in time series modeling due to their exceptional ability to capture long-range dependencies in sequential data. A surge of transformer-based methods have been proposed and successfully applied to various time series modeling tasks, such as forecasting [19, 51, 41, 52], classification [47], and anomaly detection [45]. However, these methods are usually designed for regular time series settings, where multivariate numerical values at the same timestamp are viewed as a unit, and temporal interactions across different units are explicitly modeled. A recent work [25] suggests dividing each univariate time series into a sequence of sub-series and modeling their interactions independently. These methods all operate on numerical values and assume fully observed input, while our proposed method deals with time series data in the visual modality. By transforming the time series into visualized line graphs, we can effectively handle irregularly sampled time series and harness the strong visual representation learning abilities of pre-trained vision transformers.
Imaging time series. Previous studies have explored transforming time series data into different types of images, such as Gramian fields [39], recurring plots [14, 37], and Markov transition fields [40]. These approaches typically employ convolutional neural networks (CNNs) for classification tasks. However, they often require domain expertise to design specialized imaging techniques and are not universally applicable across domains. Another approach [35] involves utilizing convolutional autoencoders to complete images transformed from time series, specifically for forecasting purposes. Similarly, [31] utilized CNNs to encode images converted from time series and use a regressor for numerical forecasting. These approaches, however, still need plenty of specialized designs and modifications to adapt to time series modeling. Furthermore, they still lag behind the current leading numerical techniques. In contrast, our proposed method leverages the strong abilities of pre-trained vision transformer to achieve superior results by transforming the time series into line graph images, sidestepping the need of prior knowledge and specific modifications and designs.
As illustrated in Fig. 1, ViTST consists of two main steps: (1) transforming multivariate time series into a concatenated line graph image, and (2) utilizing a pre-trained vision transformer as an image classifier for the classification task. To begin with, we present some basic notations and problem formulation.
Notation. Let  $\mathcal{D} = \{(S_i, y_i) | i = 1, \dots, N\}$  denote a time series dataset containing  $N$  samples. Every data sample is associated with a label  $y_i \in \{1, \dots, C\}$ , where  $C$  is the number of classes. Each multivariate time series  $S_i$  consists of observations of  $D$  variables at most (some might have no observations). The observations for each variable  $d$  are given by a sequence of tuples with observed time and value  $[(t_1^d, v_1^d), (t_2^d, v_2^d), \dots, (t_{n_d}^d, v_{n_d}^d)]$ , where  $n_d$  is the number of observations for variable  $d$ . If the intervals between observation times  $[t_1^d, t_2^d, \dots, t_{n_d}^d]$  are different across variables or samples,  $S_i$  is an irregularly sampled time series. Otherwise, it is a regular time series.
Problem formulation. Given the dataset  $\mathcal{D} = \{(S_i, y_i) | i = 1, \dots, N\}$  containing  $N$  multivariate time series, we aim to predict the label  $\hat{y}_i \in \{1, \dots, C\}$  for each time series  $S_i$ . There are mainly two components in our framework: (1) a function that transforms the time series  $S_i$  into an image  $x_i$ ; (2) an image classifier that takes the line graph image  $x_i$  as input and predicts the label  $\hat{y}_i$ .
3.1 Time Series to Image Transformation
Time series line graph. The line graph is a prevalent method for visualizing temporal data points. In this representation, each point signifies an observation marked by its time and value: the horizontal axis captures timestamps, and the vertical axis denotes values. Observations are connected with straight lines in chronological order, with any missing values interpolated seamlessly. This graphing approach allows for flexibility for users in plotting time series as images, intuitively suited for the processing efficiency of vision transformers. The practice mirrors prompt engineering when using language models, where users can understand and adjust natural language prompts to enhance the model performance.
In our practice, we use marker symbols “*” to indicate the observed data points in the line graph. Since the scales of different variables may vary significantly, we plot the observations of each variable in an individual line graph, as shown in Fig. 1. The scales of each line graph  $\mathbf{g}_{i,d}$  are kept the same across different time series  $S_{i}$ . We employ distinct colors for each line graph for differentiation. Our initial experiments indicated that tick labels and other graphical components are superfluous, as an observation's position inherently signals its relative time and value magnitude. We investigated the influences of different choices of time series-to-image transformation in Section 4.3.
Image Creation. Given a set of time series line graphs  $\mathcal{G}_i = \mathbf{g}_1, \mathbf{g}_2, \dots, \mathbf{g}_D$  for a time series  $S_i$ , we arrange them in a single image  $\mathbf{x}_i$  using a pre-defined grid layout. We adopt a square grid by default, following [10]. Specifically, we arrange the  $D$  time series line graphs in a grid of size  $l \times l$  if  $l \times (l - 1) < D \leq l \times l$ , and a grid of size  $l \times (l + 1)$  if  $l \times l < D \leq l \times (l + 1)$ . For example, the P19, P12, and PAM datasets contain 34, 36, and 17 variables, respectively, and the corresponding default grid layouts are  $6 \times 6, 6 \times 6$ , and  $4 \times 5$ . Any grid spaces not occupied by a line graph remain empty. Figure 6 showcases examples of the resulting images. As for the order of variables, we sort them according to the missing ratios for irregularly sampled time series. We explored the effects of different grid layouts and variable orders in Section 4.3.
3.2 Vision Transformers for Time Series Modeling
Given the image  $\mathbf{x}_i$  transformed from time series  $S_i$ , we leverage an image classifier to perceive the image and perform the classification task. The time series patterns in a line graph image involve both local (i.e., the temporal dynamics of a single variable in a line graph) and global (the correlation among variables across different line graphs) contexts. To better capture these patterns, we choose the recently developed vision transformers. Unlike the predominant CNNs, vision transformers are proven to excel at maintaining spatial information and have stronger abilities to capture local and global dependencies [9, 22].
Preliminary. Vision Transformer (ViT) [9] is originally adapted from NLP. The input image is split into fixed-sized patches, and each patch is linearly embedded and augmented with position embeddings. The resulting sequence of vectors is then fed into a standard Transformer encoder consisting of a stack of multi-head attention modules (MSA) and MLP to obtain patch representations. An extra classification token is added to the sequence to perform classification or other tasks. ViT models global inter-unit interactions between all pairs of patches, which can be computationally expensive for high-resolution images. Swin Transformer, on the other hand, adopts a hierarchical architecture with multi-level feature maps and performs self-attention locally within non-overlapping windows, reducing computational complexity while improving performance. We use Swin Transformer as the default backbone vision model unless otherwise specified, but any other vision model can also be used within this framework.
Swin Transformer captures the local and global information by constructing a hierarchical representation starting from small-sized patches in earlier layers and gradually merging neighboring patches in deeper layers. Specifically, in the W-MSA block, self-attention is calculated within each non-overlapping window, allowing for the capture of local intra-variable interactions and temporal dynamics of a single line graph for a variable  $d$ . The shifted window block SW-MSA then enables
connections between different windows, which span across different line graphs, to capture global interactions. Fig. 2 illustrates this process. Mathematically, the consecutive Swin Transformer blocks are calculated as:
$$
\hat {\mathbf {z}} ^ {l} = \operatorname {W - M S A} \left(\operatorname {L N} \left(\mathbf {z} ^ {l - 1}\right)\right) + \mathbf {z} ^ {l - 1},
$$
$$
\mathbf {z} ^ {l} = \operatorname {M L P} \left(\operatorname {L N} \left(\hat {\mathbf {z}} ^ {l}\right)\right) + \hat {\mathbf {z}} ^ {l},
$$
$$
\hat {\mathbf {z}} ^ {l + 1} = \operatorname {S W - M S A} \left(\operatorname {L N} \left(\mathbf {z} ^ {l}\right)\right) + \mathbf {z} ^ {l},
$$
$$
\mathbf {z} ^ {l + 1} = \operatorname {M L P} \left(\ln \left(\hat {\mathbf {z}} ^ {l + 1}\right)\right) + \hat {\mathbf {z}} ^ {l + 1}, \tag {1}
$$
where  $\hat{\mathbf{z}}^l$  and  $\mathbf{z}^l$  denote the output features of the (S)W-MSA module and the MLP module for block  $l$ , respectively; LN stands for the layer normalization [1]. After multiple stages of blocks, the global interactions among patches from all line graphs can be captured, enabling the modeling of correlations between different variables. We have also explored the use of additional positional embeddings, including local positional embeddings to indicate the position of each patch within its corresponding line graph, and global positional embeddings to represent the index of the associated line graph in the entire image. However, we didn't observe consistent improvement over the already highly competitive performance, which might suggest that the original pre-trained positional embeddings have already been able to capture the information regarding local and global patch positions.
Figure 2: Illustration of the shifted window approach in Swin Transformer. Self-attention is calculated within each window (grey box). When the window is contained within a single line graph, it captures local interactions. After shifting, the window includes patches from different line graphs, allowing for the modeling of global cross-variable interactions.
Inference. We use vision transformers to predict the labels of time series in the same way as image classification. The outputs of Swin Transformer blocks at the final stage are used as the patch representations, upon which a flattened layer with a linear head is applied to obtain the prediction  $\hat{y}_i$ .
Table 1: Statistics of the irregularly sampled time series datasets used in our experiments.
Datasets	#Samples	#Variables	#Avg. obs.	#Classes	Demographic info	Imbalanced	Missing ratio
P19	38,803	34	401	2	True	True	94.9%
P12	11,988	36	233	2	True	True	88.4%
PAM	5,333	17	4,048	8	False	False	60.0%
Datasets and metrics. We conducted experiments using three widely used healthcare and human activity datasets, as outlined in Table 1. The P19 dataset [29] contains information from 38,803 patients, with 34 sensor variables and a binary label indicating sepsis. The P12 dataset [12] comprises data from 11,988 patients, including 36 sensor variables and a binary label indicating survival during hospitalization. Lastly, the PAM dataset [28] includes 5,333 samples from 8 distinct human activities, with 17 sensor variables provided for each sample. We used the processed data provided by Raindrop [49]. To ensure consistency, we employed identical data splits across all comparison baselines, and evaluated the performance using standard metrics such as Area Under a ROC Curve (AUROC) and Area Under Precision-Recall Curve (AUPRC) for the imbalanced P12 and P19 datasets. For the more balanced PAM dataset, we reported Accuracy, Precision, Recall, and F1 score.
Implementation. We utilized the Matplotlib package to plot line graphs and save them as standard RGB images. For the P19, P12, and PAM datasets, we implemented grid layouts of  $6 \times 6$ ,  $6 \times 6$ , and  $4 \times 5$ , respectively. For a fair comparison, we assigned a fixed size of  $64 \times 64$  to each grid cell (line graph), resulting in image sizes of  $384 \times 384$ ,  $384 \times 384$ , and  $256 \times 320$ , respectively. It is important to note that image sizes can also be directly set to any size, irrespective of grid cell dimensions. We plot the images according to the value scales on training sets. We employed the checkpoint of Swin
Table 2: Comparison with the baseline methods on irregularly sampled time series classification task. Bold indicates the best performer, while underline represents the second best.
Methods	P19	P12	PAM
AUROC	AUPRC	AUROC	AUPRC	Accuracy	Precision	Recall	F1 score
Transformer	80.7 ± 3.8	42.7 ± 7.7	83.3 ± 0.7	47.9 ± 3.6	83.5 ± 1.5	84.8 ± 1.5	86.0 ± 1.2	85.0 ± 1.3
Trans-mean	83.7 ± 1.8	45.8 ± 3.2	82.6 ± 2.0	46.3 ± 4.0	83.7 ± 2.3	84.9 ± 2.6	86.4 ± 2.1	85.1 ± 2.4
GRU-D	83.9 ± 1.7	46.9 ± 2.1	81.9 ± 2.1	46.1 ± 4.7	83.3 ± 1.6	84.6 ± 1.2	85.2 ± 1.6	84.8 ± 1.2
SeFT	81.2 ± 2.3	41.9 ± 3.1	73.9 ± 2.5	31.1 ± 4.1	67.1 ± 2.2	70.0 ± 2.4	68.2 ± 1.5	68.5 ± 1.8
mTAND	84.4 ± 1.3	50.6 ± 2.0	84.2 ± 0.8	48.2 ± 3.4	74.6 ± 4.3	74.3 ± 4.0	79.5 ± 2.8	76.8 ± 3.4
IP-Net	84.6 ± 1.3	38.1 ± 3.7	82.6 ± 1.4	47.6 ± 3.1	74.3 ± 3.8	75.6 ± 2.1	77.9 ± 2.2	76.6 ± 2.8
\( DGM^2-O \)	86.7 ± 3.4	44.7 ± 11.7	84.4 ± 1.6	47.3 ± 3.6	82.4 ± 2.3	85.2 ± 1.2	83.9 ± 2.3	84.3 ± 1.8
MTGNN	81.9 ± 6.2	39.9 ± 8.9	74.4 ± 6.7	35.5 ± 6.0	83.4 ± 1.9	85.2 ± 1.7	86.1 ± 1.9	85.9 ± 2.4
Raindrop	87.0 ± 2.3	51.8 ± 5.5	82.8 ± 1.7	44.0 ± 3.0	88.5 ± 1.5	89.9 ± 1.5	89.9 ± 0.6	89.8 ± 1.0
ViTST	89.2 ± 2.0	53.1 ± 3.4	85.1 ± 0.8	51.1 ± 4.1	95.8 ± 1.3	96.2 ± 1.3	96.1 ± 1.1	96.5 ± 1.2
Transformer pre-trained on the ImageNet-21K dataset<sup>2</sup>. The default patch size and window size are 4 and 7, respectively.
Training. Given the highly imbalanced nature of the P12 and P19 datasets, we employed upsampling of the minority class to match the size of the majority class. We finetuned the Swin Transformer for 2 and 4 epochs on the upsampled P19 and P12 datasets, respectively, and for 20 epochs on the PAM dataset. The batch sizes used for training were 48 for P19 and P12, and 72 for PAM, while the learning rate was set to 2e-5. The models were trained using A6000 GPUs with 48G memory.
Incorporating static features. In real-world applications, especially in the healthcare domain, irregular time series data often accompanies additional information such as categorical or textual features. In the P12 and P19 datasets, each patient's demographic information, including weight, height, and ICU type, is provided. This static information remains constant over time and can be expressed using natural language. To incorporate this information into our framework, we employed a template to convert it into natural language sentences and subsequently encoded the resulting text using a RoBERTa-base [21] text encoder. The resulting text embeddings were concatenated with the image embeddings obtained from the vision transformer to perform classification. The static features were also applied to all compared baselines.