SlowGuess/NeurIPS_2023_OCR huggingface dataset:
hidden feature space where the paired image and text stay close. Under our framework, such a shared space can also be learned for the paired visual time series images and textual clinical notes, which is our future direction. It also paves the way for the application of multi-modal models such as GPT-4 [26] to handle the visualized time series data and the clinical notes simultaneously. In our current experiments, we used a text encoder, Roberta-base, to encode textual demographic information in the P19 and P12 datasets. We also experimented with normalizing the original categorical features and encoding them using an MLP as in previous work, and compare with the strong baseline, Raindrop. The results are shown in Table 9. We observe that even without using static features, our method has already outperformed Raindrop. In addition, utilizing Roberta to encode and incorporate the textual feature is more effective than applying MLP over categorical features.
Table 10: Preliminary experiments on two settings to fuse the line graph images for different variables. The default setting is to first arrange all line graph images into a single image and then learn the representation for classification. ViT-subimage stands for that we first learn the representation for each line graph subimage separately and then concatenate their representation for classification.
Strategies	P19	P12	PAM
AUROC	AUPRC	AUROC	AUPRC	Accuracy	Precision	Recall	F1 score
ViT	87.9 ± 2.5	51.6 ± 3.7	84.8 ± 1.3	48.1 ± 3.8	93.4 ± 0.7	94.7 ± 0.9	94.1 ± 0.7	94.3 ± 0.7
ViT-subimage	85.1 ± 1.5	47.9 ± 3.4	77.6 ± 3.2	35.5 ± 6.2	90.4 ± 1.3	92.9 ± 1.0	91.1 ± 0.9	91.9 ± 1.0
B.3 Experiments on Image Fusion
In our preliminary experiments, we examined two distinct approaches to fusing the line graph sub-images in each multivariate time series data. First, we processed each sub-image independently to learn the patch/image representations and then concatenated their respective patch representations to input into the final prediction layer. In contrast, our default method aggregates all sub-images into a single image to learn the patch embeddings. The key distinction between these strategies is whether a patch can attend to those from other sub-images and how position embedding factors in during the representation learning phase. Aside from this, other parameters were consistent across both methods. We tested on ViT and the performance comparisons with these two settings are shown in Table 10. It is evidenced that attending to patches from other sub-images offers advantages. This likely allows for capturing cross-variable correlations at a granular level within the self-attention layers, as opposed to only at the final linear prediction layer.
Table 11: Statistics and hyperparameter settings of evaluated regular multivariate time series datasets.
Datasets	Variables	Classes	Length	Train size	Grid layout	Image size	Learning rate	Epochs
EC	3	4	1,751	261	2 × 2	256 × 256	1e-4	20
UW	3	8	315	120	2 × 2	256 × 256	1e-4	100
SCP1	6	2	896	268	2 × 3	256 × 384	1e-4	100
SCP2	7	2	1,152	200	3 × 3	384 × 384	5e-5	100
JV	12	9	29	270	4 × 4	384 × 384	1e-4	100
SAD	13	10	93	6599	4 × 4	384 × 384	1e-5	20
HB	61	2	405	204	4 × 4	384 × 384	1e-4	100
FD	144	2	62	5890	12 × 12	384 × 384	5e-4	100
PS	963	7	144	267	32 × 32	384 × 384	5e-4	100
EW	6	5	17984	128	2 × 3	256 × 384	2e-5	100
B.4 Experiment on Regular Time Series
We selected ten representative multivariate time series datasets from the UEA Time Series Classification Archive [2] with diverse characteristics, including the number of classes, variables, and time series length. The datasets we chose are EthanolConcentration (EC), Handwriting (HW), UWaveGestureLibrary (UW), SelfRegulationSCP1 (SCP1), SelfRegulationSCP2 (SCP2), JapaneseVowels (JV), SpokenArabicDigits (SAD), Heartbeat (HB), FaceDetection (FD), PEMS-SF (PS), and EigenWorms (EW). Notably, the PS dataset has an exceptionally high number of variables (963), while the EW dataset has extremely long time series (17984). These two datasets allow us to assess the effectiveness of our approach when dealing with large numbers of variables and long time series. We applied different image sizes according to the grid layouts for these datasets. The hyperparameter settings are
provided in Table 11, and we applied cutout [8] data augmentation methods to SCP1, SCP2, and JV datasets due to the small size of their training sets.
B.5 Self-supervised Learning
We preliminary explored masked image modeling self-supervised pre-training on the time series line graph images. We randomly mask columns of patches with a width of 32 on each line graph within a grid cell. The masking ratio is set as  $50\%$ . We finetuned the Swin Transformer model for 10 epochs with batch size 48. The learning rate is 2e-5. Following [44], we use a linear layer to reconstruct the pixel values and employ an  $\ell_1$  loss on the masked pixels:
$$
\mathcal {L} = \frac {1}{\Omega \left(\mathbf {p} _ {\mathrm {M}}\right)} \left\| \mathbf {p} _ {\mathrm {M}} - \mathbf {p} _ {\mathrm {M}} \right\| _ {1}, \tag {2}
$$
where  $\mathbf{p}_{\mathrm{M}}$  and  $\hat{\mathbf{p}}_{\mathrm{M}}$  are the masked and reconstructed pixels, respectively;  $\Omega(\cdot)$  denotes the number of elements. With self-supervised masked image modeling, the performance improves by 1.0 in AUPRC points (percentage) from  $52.8 (\pm 3.8)$  to  $53.8 (\pm 3.2)$ . The AUROC points (percentage) slightly dropped from  $89.4 (\pm 1.9)$  to  $88.9 (\pm 2.1)$ .
B.6 Full Experimental Results
We presented the full experimental results in the leave-sensors-out settings in Table 12.