IMTS is Worth Time $\times$ Channel Patches: Visual Masked Autoencoders for Irregular Multivariate Time Series Prediction
Authors: Zhangyi Hu
Jiemin Wu
Hua XU
Mingqian Liao
Ninghui Feng
Bo Gao
Songning Lai
Yutao Yue
Venue: ICML 2025 poster (2025)
Keywords: Irregular Multivariate Time Series Prediction; Visual Mask Autoencoder; Self-Supervised Learning
Primary area: deep_learning->sequential_models_time_series

Abstract: Irregular Multivariate Time Series (IMTS) forecasting is challenging due to the unaligned nature of multi-channel signals and the prevalence of extensive missing data. Existing methods struggle to capture reliable temporal patterns from such data due to significant missing values. While pre-trained foundation models show potential for addressing these challenges, they are typically designed for Regularly Sampled Time Series (RTS). Motivated by the visual Mask AutoEncoder's (MAE) powerful capability for modeling sparse multi-channel information and its success in RTS forecasting, we propose **VIMTS**, a framework adapting **V**isual MAE for **IMTS** forecasting. To mitigate the effect of missing values, VIMTS first processes IMTS along the timeline into feature patches at equal intervals. These patches are then complemented using learned cross-channel dependencies. Then it leverages visual MAE's capability in handling sparse multichannel data for patch reconstruction, followed by a coarse-to-fine technique to generate precise predictions from focused contexts. In addition, we integrate self-supervised learning for improved IMTS modeling by adapting the visual MAE to IMTS data. Extensive experiments demonstrate VIMTS's superior performance and few-shot capability, advancing the application of visual foundation models in more general time series tasks. Our code is available at https://github.com/WHU-HZY/VIMTS.

IMTS is Worth Time x Channel Patches: Visual Masked Autoencoders for
Irregular Multivariate Time Series Prediction
Zhangyi Hu"
Jiemin Wu
Hua Xu *
Mingqian Liao Yutao Yue ?
Ninghui Feng
Bo Gao
Songning Lai
Abstract
Irregular Multivariate Time Series (IMTS) fore- casting is challenging due to the unaligned nature of multi-channel signals and the prevalence of ex- tensive missing data. Existing methods struggle to capture reliable temporal patterns from such data due to significant missing values. While pre- trained foundation models show potential for ad- dressing these challenges, they are typically de- signed for Regularly Sampled Time Series (RTS). Motivated by the visual Mask AutoEncoder's (MAE) powerful capability for modeling sparse multi-channel information and its success in RTS forecasting, we propose VIMTS, framework adapting Visual MAE for IMTS forecasting. To mitigate the effect of missing values, VIMTS first processes IMTS along the timeline into feature patches equal intervals. These patches are then complemented using learned cross-channel depen- dencies. Then it leverages visual MAE's capabil- ity in handling sparse multichannel data for patch reconstruction. followed by a coarse-to-fine tech- nique to generate precise predictions from focused contexts. addition, we integrate self-supervised learning for improved IMTS modeling by adapt- ing the visual MAE IMTS data. Extensive experiments demonstrate VIMTS's superior per- formance and few-shot capability, advancing the application of visual foundation models in more general time series tasks. Our code available at https: / gi www.hubcooo.. com/ WWUU-WUUUUA HHZY H//////IHHHT/. /VIMTS.
GONNN-DDE haved
(b) Foundation model based
IMTS
N-OIF
OCN/
X Sppursity Challenge IMTS Suppontd
RTS
Model
trimedd o Pro-mmin Benelia
xXSTS Unsupported
Ours
stade
So Superchee Learning
Embed
Recorstract
Patchity
Poodic
Obsrriation
Furecasting Fine-taning
MAE
Obsrrvotion Profiction
LN
Obsurvation Prodiction
Figure I. Illustration of our idea: (a) Current IMTS-spectiee meth- ods struggle to capture reliable temporal patterns from such data due to significant missing values. (b) Pre-trained models show po- tential modeling sparse data, are limited to as illustrated in (c) and (d), VIMTS segments data into time-aligned patches and imputes missing values at the representation level us- ing time x channel patchify. It then leverages self-supervised learning to adapt the visual MAE's pre-trained capability for han- dling semantically sparse multi-channel data to IMTS data, leading to powerful performance and few-shot capability.
contrast,
2017), transportation (Gong et al., 2021), and metcorology (Das & Ghosh, 2017). However, unlike structured data such as images or text, the semantic information in IMTS is embedded in complex dynamics across multiple channels over time, which are disrupted by irregular sampling and missing values. These challenges arise from various factors, including the randomness of monitored subjects, the relia- bility issues of data collection devices, and privacy concerns (Wang et al., 2024), thus complicating downstream tasks such as traffic flow forecasting and weather forecasting.
1. Introduction
Irregular Multivariate Time Series (Weerkkdy et al., 2021) forecasting plays erucial role in various domains, in- cluding finance (Bai & Ng, 2008), healtheare (Esteban et al.,
Early methods involve statistical imputation methods for synchronizing timestamps (Hamzaçebi, 2008; Van Buuren & Grootthis-Oudshoor 2011), 1). but they require deep understanding of system dynamics and inadvertently dis- card information contained in missing points (Horn et al., 2020). Although recent GCN-based methods (Zhang et al., 2024a) and Neural-ODE-baal based methods (Chen et al., 2018; De Brouwer et al., 2019; Rubanova et al., 2019; Schirmer et al., 2022) show progress modeling cross-channt de- pendency and temporal dependencies of irregular samples,
Eqqull contribution 'The Hong Kong University of Sci- ence and Technology (Guangzhou). Guangzhou 51 5100 China Beijing University of Posts and Telecommunications, Beijing, China 'Institute of Deep Perception Technology. JITRI, Wuxi 214000, China. Correspondence to: Yutao Yue <yutaoyue hkust- gz.edu.cn>..
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

IMTS is Worth Time x Channel Patches: Visual Masked Autoencoders for Irregular Multivariate Time Series Prediction
Human Activity (MSE) 2
treats IMTS as time x channel image-like structure. It divides the data into sections along the timeline at equal intervals and employs a Transformable Time-aware Con- volutional Network (TTCN) to extract intra-section feature patches. This addresses unstructured inputs and tempo- ral misalignment. These patches that suffer from missing values are then complemented at the feature-level using cross-channel dependencies learned by Graph Convolutional Networks (GCNs) (Kipf & Welling, 2016). These comple- mented patches are then fed into pre-trained visual MAE for understanding and reconstruction. This process mod- els temporal dependencies for patches within each channel. Finally, a coarse-to torsee-to-finee fonein technique generates precise predic- tions by querying patch-level time period representations using their corresponding timestamps, thereby focusing on relevant temporal-channel context. To fully leverage the po- tential of the visual MAE and fully utilize historical data, we develop two-stage training strategy. First, self-supervised learning is employed to improve IMTS modeling through adapting visual MAE to IMTS data. Second, supervised fine- tuning are utilized for more precise prediction. This strategy leads to significant improvement forecasting accuracy and robust few-shot capability. Our main contributions include: We introduce VIMTS, a pioneering framework that lever- ages the powerful capability of visual MAE in modeling semantically sparse multichannel data for IMTS forecast- ing. As shown in Fig. 2, extensive experiments on four real-world datasets demonstrate its superior performance compared to existing baselines and its robust few-shot capability. paving the way for applying visual foundation models to more general time series forecasting tasks.
Human Activity (MAE),
PhysioNet (MAE)
2.70
3:34
2.75
3:19
3,02
3.25
USHCN
(MSE)
4.66
4.96
3.00
5.10
5.12
3.02
1.91
11
PhysioNet (MSE)
3,17
6:88
3
6:32
3,04
1.45
5.56
USHCN
MIMIC (MAE)
(MAE)
SOTA/100% camal VIMTS(20%
VIMTS130% datal
VINTS1100% dats)
Figure 2. The illustration highlights VIMTS's superior Mean Abso- lute Error (MAE) and Mean Squared Error (MSE) relative state- of-the-art methods on the PhysioNet, Human Activity, USHCN, and MIMIC datasets. Moreover, VIMTS maintains competitive performance in few-shot scenarios.
GCN-based methods alternately model temporal and chan- nel information, causing severe cumulative error due to the sparsity, while N-ODE-based methods struggle to construct accurate models from some individual channels with signifi- cant missing values and simultaneously require substantial computational resources. These challenges lead to unreli- able pattern capturing and poor few-shot capability.
In parallel, foundation models have revolutionized various areas (Jin et al., 2024; Das et al., 2024; Brown et al., 2020; He al., 2022) by leveraging capability benefiting from large-scale pre-training to capture important features for downstream tasks with limited fine-tuning data. VisionTS (Chen et al., 2025) demonstrates that visual Mask AutoEn- coders (MAFs) pre-trained on large-scale RGB images are naturally adaptable to time series data, as they share pat- tern similarities with natural images in terms of information density, and multichannel patterns. This suggests signifi- cant potential for applying visual foundation models time series forecasting, Nevertheless, most existing pre-trained model based methods are designed for RTS data, limiting their application in more general and practical scenarios.
We propose new -ecddinng strategy. For encod- ing, IMTS is processed into time- amee-lligeed feature patches along the timeline at equal intervals, which is then com- pensated with cross-channel information to mitigate the ef- feet of missing values. For decoding, a coarse-to -oerse--t--ineee strat- egy progressively generates predictions from patches to specific time points, focusing on related temporal-channel contexts for enhanced accuracy.
We develop two-stage training strategy. First, VIMTS employs self-supervised learning to improve IMTS model- ing by adapting the capabilities of visual MAES to IMTS data. Second, supervised fine-tuning is proposed for task- specific adaptation. This strategy leads to significant per- formance improvement and robust few-shot capability.
As illustrated in Fig. 1, motivated by the capabilities of vi- sual MAEs in modeling semantically sparse multichannel information and their adaptability to the time series do- main, we introduce a pioneering framework that leverages Visual pre-trained MAE for IMTS forecasting (VIMTS). The core idea is to adapt the powerful capabilities of pre- trained visual MAES (He et al., 2022) to IMTS data via self- supervised latent space mask reconstruction for enhanced performance and few-shot capability. Specifically, VIMTS
2. Methodology
2.1. Overview
The overall methodology is illustrated in Fig. 3. The architecture of VIMTS consists of three main compo- nents: timexchannel patchify, time-wise reconstruction, and patch2point prediction. We employ a two-stage train- ing strategy that encompasses self-supervised learning and supervised fine-tunin fine-tuning. In the following sections, we will
2

---

IMTS is Worth Time x Channel Patches: Visual Masked Autoencoders for Irregular Multivariate Time Series Prediction
Stage Sellf-Speeriied Learning
Time Chanaei Fatchity
Time Wise Reconstructioo
Patch2Poist Pretlietion
TTCN
Mask
GCUN
Imsert
Decodel
Observation
Observation
Stage Supervised Fine-tuning
Time x Chanael Putchity
Reconstruction
Patch2Point Prediction
TTCN
GCN
Emboo
Appand
Decode I
Osservation Pred
Frod
Chonnnl Mark
Feutsrre Patch
Learnuble
Reconstrucced Putch
Mutchest
Embed
Mutching
Figure 3.
The overall architecture of VIMTS. The irregularly sampled data in each channel is divided into sections with equal-intervals along the timeline. Each section undergoes intra-section feature extraction using Time-aware Convolutional Network (TTCN) and cross-channel information compensation via Graph Convolutional Networks (GCNs). These compensated patches are then fed into a pre-trained MAE for patch reconstruction, thereby modeling temporal dependencies among patches within each channel. Finally, coarse-to to fo-fine technique gradually generates precise predictions from patch-level to point-level. The training encompasses two stages. First, supervised learning aims to improve IMTS modeling by adapting the capabilities Second, the supervised fine-tuning is employed to enhance forecasting performance.
visual pre-trained MAE to IMTS data.
introduce each part of our pipeline in detail.
$ p S P, spans from to tattrt to sstort + s, where Extart 4 + 1)s, and l the initial time. This approach ensures section-level temporal alignment and preserves the multichannel structure (Zhang et al., 2024a).
2.2. Task Definition
IMTS Observation. IMTS data with N variables is rep- resented by the triplet O (T, MM). Here, T Liff=1 E R contains L unique timestamps, while matrix X' = Merlw E f...N records observed values x at the I-th timestamp ty for the n-th variable or 'NA' if unob- served. The mask matrix M = [[m]]] 1 E {0, 1) indicates the availability of observation at the 1-th timestamp for the n-th variable with m. 1, otherwise m² 0.
A After dividing, we utilize learnable time embeddings to capture temporal patterns and encode continuous time infor- mation (Shukla Marlin, 2021a). For given timestamp t, its embedding
<(t) is defined as:
(()) = wwo + cao
if sin sin(wa t ++aa), if < d Die
(1)
IMTS Forecasting. The task is to develop model θ that, given historical observations O and future query timestamps @ {[ J== where qf denotes that the j-th query timestamp of the 2n queries for the n-th variable, to forecast the corresponding target values X = [ 2., 1 where 1 denotes the ground truth value at the query timestamp of. This process is represented O Q) + X.
where wy and ag are learnable parameters and Dte is the embedding dimension. These embeddings combine linear and periodic terms capture non-periodic and periodic temporal patterns.
2.3.2. TEMPORAL FEATURE EXTRACTION
After time-wise dividing and embedding, we employ a Transformable Time-aware Convolutional Network (TTCN) to process variable-length sequences within time intervals (Zhang et al., 2024b) into patches with aligned shapes and semantics.
2.3. Time x Channel Patchify
This section aims to transform unstructured IMTS data into patches while mitigating the effect of missing values. To achieve this, the module first processes IMTS into feature patches along the timeline at equal time intervals. Each time interval contains patches from all channels. These patches that suffer from missing values are then complemented by information from related channels according to the learned cross-channel dependencies. This process enables more reliable inter-patch temporal dependency modeling in MAE. 2.3.1. TIME-WISE DIVIDING AND EMBEDDING In our method, an IMTS dataset is divided into P patches using uniform time windows of size s. Each patch p. for
In detail, for the n-th channel, we concatenate the time embeddings ((tT)) and the observation xy within the p-th time section:
x = [o(E")=)TTT + E 1 &
Start + s),
(2)
where denotes the 'concatenate' operation, I, and rp de- note the start and end index the observation respectively within the p-th time section in the n-th channel.
3

---

IMTS is Worth Time x Channel Patches: Visual Masked Autoencoders for Irregular Multivariate Time Series Prediction
TTCN then captures intra-section information within each
R-xDD denotes the hy concatenation across N channels,
channel by employing adaptive convolution filters:
W E Dx and w E (DD+D.))xl are learnable weights. These hybrid embeddings Ep.1. E RNx P
&
exp(Fa(x(1i)
f = exp(F((xX 1=1
(3) are then used to calculate the adaptive adjaceney matrix Ap E E for the p-th time section, which dynamically captures directional dependencies among channels:
where, for the n-th chamnel, Lp = 1p Tp + 1 is the number of points within the p-th time section, f E xR repre- sents the filter for the d-th feature map, Din is the number of filters, and F denotes the d-th meta-filter (mlp).
Ap Softmax (ReLU,,,, E222))
(8)
Then graph convolution operations with skip connections are applied to exchange information among channels ac- cording to Api
With Din filters derived based on Eq. 3. we attain the p- th feature patch in the n-th channel hp E Ri by the following temporal convolution:
M E(A,)
Hom ReLU
H Wgen
+
Hpp
(9)
DNn
hm
TH
(4)
where M is the number of GCN layers.
Finally, to ensure comprehensive representation while pre- serving original information, we concatenate the original Hp with Hgen after cross-channel interaction inputs H for MAE, represented as:
d=1
To handle sparse IMTS data, we enhance representations by concatenating a binary mask indicating the availability:
h. = !/6 [|mp] E R
(5)
H H. 1H [ E RRNx22
(10)
where D Dm + 1, m = 1 indicates the presence of observations while ma = 0 indicates an empty patch, hi hp. denotes the feature patch concatenated with the mask indi- cator.
2.4. Time- Wise Reconstruction
After cross-channel complementation, we leverage the ca- pability of visual MAE for modeling semantically sparse multichannel data obtained from pretraining to model tem- poral dependencies among patches within each channel.
We further incorporate channel-specific embeddings to cap- ture channel-specific traits (e.g., units, stats, missing pat- terns), thereby distinguishing heterogeneous channels to en- hance the following cross-channe compensation and inter- patch temporal modeling within channels. In detail, for the n-th channel, we define learnable embeddings e, E IR which is added to the feature patches to create the patches hy for cross- cooss-channel dependency modeling:
2.4.1. INPUT EMBEDDING AND TEMPORAL POSITION EMBEDDINGS
For a concise representation, if not emphasized in the fol- lowing, we use [h ] E Px2D to denote the sequence of feature patches within each single channel. Before MAE encoding, we compress the cross-channel information com- plementation and original information using linear pro- jection W enc E R2DxDe, to adapt it to the MAE input dimension D.:
hp k m En.
(6)
2.3.3. PROSS-CHANNEL INFORMATION INTERACTION
Due to extensive missing values in IMTS, patches of indi- vidual channels contain insufficient information for reliable temporal dependency modeling. To mitigate this, we em- ploy GCN to model bidirectional channel dependencies and enrich each channel's 's representation with complementary information from correlated channels.
Cp =
1 W p
(11)
Next, to enable temporal reconstruction, we employ learnable temporal period embeddings, similar to positional embeddings. Specifically, for a sequence of patches of length P. the temporal period embedding for the p-th patch is initialized using 2D sine-cosine encoding, represented as:
Inspired by (Zhang et al., 2024a), to learn bidirectional channel dependency graphs, we fuse static channel char- acteristics with dynamic patch features to create graph vertex embeddings. In the beginming, maintain two learn- able embedding dictionaries Ej, E- E R...... which en- code static characteristics (e.g., representing inflow/outflow nodes). They are then updated with dynamic patch informa- tion by a gated mechanism to obtain hybrid embeddings:
TPE [2x] = sin (in(p/000002///
(12)
TPE [2A: + 1] cos cos(p/10000 c)0(0//0000)
(13)
TPE = sin (1/0000044)))
(14)
TPE 22k + 1] = cos cos(01000000
(15)
where & E [0, d/4 - 1]. and d is half of the encoder em- bedding dimension D or decoder embedding dimension Da/2. The complete time period embeddings for the p-th patch for encoder and decoder are then represented as:
Ep. E + Ap. @ H. We. k
E (1,2),
(7)
where 9p.k = (tah H E W)) controls the fu- sion of static and dynamic information, H, = M E
TPE >
= [TPE):0: D2/2)/TTPE((D./' Del], (16)
4

---

IMTS is Worth Time x Channel Patches: Visual Masked Autoencoders for Irregular Multivariate Time Series Prediction
TPE [O Pa/2][TPE Dull- (17)
Fgg = F(b(t)))
(23)
This strategy treats embeddings as a T x patch sequence, allowing the model to adapt MAE's pretrained position understanding capabilities to temporal representations and capture periodic features during optimization. We then add this embedding to Ep to get the inputs of the MAE encoder:
This strategy offers three key advantages: (1) it enables flexible and accurate predictions at arbitrary continuous timestamps within target temporal periods; (2) it compre- hensively utilizes patch-level temporal patterns and cross- channel complementary information; (3) it effectively filters out irrelevant information from other temporal-channe con- texts, ensuring focused and precise predictions.
Cp enc Cp TPE,"
(18)
2.4.2. ENCODE AND RECONSTRUCTION
With the input embeddings. MAE aims to learn the temporal dependencies among patches within each channel along- side the cross-chammn information complementation, and reconstruct them at target time segments. The embedded sequence is encoded by the MAE encoder E:
2.6. Training Strategy
We employ two-stage strategy for training: self-supervised learning and supervised fine-tuning. The rationale is de- tailed in Appendix A.3.
{ E((c," (p)))
(19)
Self-supervised learning for IMTS modeling. Given a mask ratio r, we randomly mask a portion of patches be- fore encoding. Specifically, from the embedded sequence
For future patch reconstruction, we append Nrec learn- able mask tokens {[M]]]-T to the linearly projected to- kens {= Wiee- In addition, we concatenate the corre- sponding temporal positional embeddings (TPE$$)) Nrre of the target time periods with those of the encoded tokens (TPEde)) 1--1- The MAE decoder D takes this augmented sequence as input:
{Ep"C)p Jp=1+ we randomly select [Mi = [r PI patches to
mask. The remaining patches {egne) pPEv are encoded:
{zplpev (((eg""))
(24)
where V denotes the set of unmasked indices, M denotes the set of masked indices. The projected tokens {sphpev Weec are then added with (TPE PEV and concatenated with
{2p+ Neec- D(Z + TPE"),
i=
(20)
Zt = [ppp)p=11 W deci [MM])))) N
(21)
learnable mask tokens {[M) IMI added with (TPp)e)) "p JPEM JPE
i=1
TPE* H[PPE p fo= (TPE (TPEP4thH11) (ars)
(22)
to reconstruct 2
{ JIEM D(Z + TPE
(25)
where 1 denotes sequence concatenation, (2p4 h=1 represents reconstructed representations of target time peri- ods, and Wdee E R. x Da projects the encoded representa- tion into the input dimension of the decoder R2
2 = 122pppv Wdeci {MM))
(26)
TPE [[PE)ee)) jevv; (TPE: )i,
(27)
The sule-super training loss is formulated as follows:
This approach leverages the visual pre-trained capabilities of MAE, enabling historical and future patch reconstruction during self-supervised training and supervised fine-tuning, respectively.
N
Hn
1
Lest =
[F((()), - -0 2 2m
(28)
N
Hn
==
where {[t)) 1 represents the history query times- tamps set across N channels with t'tart S 2% < thart s. For the n-th channel, 5Ta.") denotes the reconstructed 1t patch, and æb is the ground truth value at 6g.
2.5. Patch2Point Prediction
We employ a coarse-to-fine technique to generate predic- tions for specific timestamps. In the coarse phase, period- level patches are reconstructed via MAE. Then, in the fine- grained phase, these patches are queried with timestamp embeddings for point-level predictions.
Supervised fine-tuning for task adaptation. We follow the same reconstruction and forecasting process detailed in Sec. 2.4.2 and See. 2.5. Here, {TPE/ Nare repre- sents future TPFs for Nree target periods. Given future query timestamps set {(())em J across N channels with ty S 19 < thy ttarrt + 8, we minimize the prediction loss:
In detail, given query timestamp ty. first generate query embedding o(t,)) and select its corresponding patch index iy matching Etart S ty S th + s. where is the patch size and stride length.
Q
Cx: = N
1 Σ
Σ
1F(6))
=m x (29)
Q q=1
We calculate the TPE for the target patch and utilize ' the method introduced in Sec. 2.4.2 to reconstruct the ig-th patch -
where for the n-th channel, 2 denotes the reconstructed iy-th patch, and =i is the ground truth value at 1n Dur- ing this stage, we selectively optimize some components to adapt to the forecasting task while preserving basic capabil- ities, which is detailed in Sec. 3.I.
The prediction is generated through a 2-layer MLP network F that takes the query and the reconstructed patch as input:
5

---

IMTS is Worth Time x Channel Patches: Visual Masked Autoencoders for Irregular Multivariate Time Series Prediction
Table 1. Overall performance evaluated by MAE and MSE (mean = std). The best perffornrgn results are highlighted in bold, the second-bes best results are highlighted in blue bold, and the third-hest results are highlighted in underline. 'Zero' and 'Linear' are different imputation methods adapting IMTS to VisionTS. **' denotes that the performance are reproduced following the original paper.
PlysioNet
Human Activity
USHCN
MIMIC
Algorithms
10
MAEx 10
MSEX10
1A
MSEx10 10
MAEX10
MSEXT0
MAEx 10
DLinear
41.86 + 0.05
15.52 + 0.03
4.03 + 0.01
4.21 101+000 0.01
6.21 + 0.00
3.88 + 0.02
4.90 1.90+00 0.00
16.29 + 0.05
TimesNet
16.18 + 0.11
6.14 0.03
3.12 0.01
0.02
5.58 4 0.05
3.60 0.04
5.88 - 0.08
13.62 = 0.07
PatchTST
12.00 0.23
6.02
4.29 0 0.14
4.80 0.09
5.75 +: 0.01
3.57 0.02
3.78 + 0.03
12.43
Crossformer
6.66 + 0.11
4.81
4.29 +: 0.20
4.89 1
5.25 + 0.04
3.27 1:009
2.65 0.10
9.56 +: 0.29
Graph Wavenet
6.04 0.22
4.41
2.89 +: 0.03
3.40 0.05
5.29 +: 0.04
3.16
2.93 0.09
10.50
MIGNN
6.26 0.18
4.46
3.03 0.03
3.53 1:03::0.3
5.39 4: 0.05
0.02
2.71 0.23
9.53 + 0.65
StemGNN
6,86 0.28
0.19
8.81
6.90
5.75 0.09
3.00
1.73 0.02
7.71
CrossGINN
0.36
4.96
0.10
3.48
5.66
0.05
2.95 0.16
10.82 0.21
FourierGNN
6,81 0.35
0.12
2.99
0.02
5,82 0.06
0.07
2,55 0.00
10.22 0.08
GRU-D
5.59 0.09
4,08
2.94
3.53 0.06
5.54 + 0.38
3.40
1.76 0.00
7,50+
SeFT
9.22 0.18
5.40 0.08
12.20 +: 0.17
8.43 + 0.07
5.80 + 0.19
3.70 + 0.11
+.01
7.84 0.08
RainDrop
9.82 0.08
5.57
0.066
14.92 0.14
9.45+ 0.05
5.78 + 0.22
3.67 + 0.17
1,99 +.00
8.27
Warpformer
5.94 +: 0.35
4,21 + 0.12
2.79 + 0.04
3.39 +: 0.03
5.25 +: 0.05
3.23 +: 0.05
1.00
7,58 +: 0.13
aTAND
6,23 0.24
0.11
3.22 + 0.07
3.81 0.07
5.33 + 0.05
3.26 + 0.10
1.85 0.06
7.73 0.13
Latent-ODE
6,05 0.57
0.26
3.31 -
3.91 0.12
5.62 + 0.03
3.60 0.12
1.89
8.11 0.52
CRU
8.56 + 0.26
5.16+0000
6.97 + 0.78
6.30 +
6.09 + 0.17
3.54+
1.89
8.11 0.52
Neural Flow
7,20 0.07
4.67 0.04 0.04
1.05 + 0.13
4.46 0.09
5.35 + 0.05
3.25 = 0.05
0.00
8,03 = 0.19
T-PATCHGNN
4.98 + 0.08
3.72 0.03
2.66 + 0.03
3.15 + 0.02
5.00 1: 0.04
3.08 + 0.04
1.36 + 0.02
6.56 + 0.11"
VSsionTS (zero)
42.41 +: 0.02
13.13 +: 0.02
8.45 +: 0.01
6.55 0.04
7,89 + 0.05
5.13 0.05
8.61 +.00
18.76 + 0.01
VisionIS (linear)
40.50 0.05
12.85 0.01 0.01
7.77 0.01
6.27 0.04
6.77 +: 0.04
4.32 0.03
0.02
17.06 0.05
VIMTS(20% data)
1.93 +: 0.09
3.63 0.06
2.72 0.01
3.11 0.00
5.01 + 0.01
3.06 +: 0.04
1.47
6.71 0.06
VIMTS(50% data)
4.86 1: 0.09
3.57 1 0.05
2.69 0.01
3.11 + 0.01
4.86 1 0.01
2.97 + 0.08
1.41
6.47 + 0.12
VIMTS(100% data)
4.81 + 0.07
3.54 + 0.04
2.65 + 0.01
3.08 + 0.01
4.88 + 0.02
2.98 + 0.05
1.36 + 0.02
6.40 = 0.17
Table 2. Datasets Technical Specifications
its fine-tuning stage. We use visual MAE-base (He et al., 2022) as the backbone, the Adam optimizer with learning rate of x 10 for training, and apply early stopping if the validation loss does not decrease for 15 consecutive epochs. To ensure robustness, each experiment is repeated with five different random seeds, and the mean and standard deviation of the results are reported. Further hyperparameter details are elaborated in Appendix A.5.
Description
PhysioNet
Human Activity
USHCN
MIMIC
Samples
12,000
5,400
1,114
23,457
Channels
41
12
5
96
Missing rutio
85.7%
75.0%
77.9%
96.7%
Observation
24 h
3,000 ms
24 months
h4
Prediction
Next 24 h
Next 1,000 ms
Next month
Next 24 h
3. Experiments
3.1. Experimental Setup
Datasets and Evaluation Metrics. To evaluate the perfor- mance of models on the IMTS forecasting task, we utilize four datasets from diverse domains: healthcare (PhysioNet (Silva et al., 2012), MIMIC((ohnson et al., 2016)), biome- chanics (Human Activity), and climate science (USHCN (Menne et al., 2015)). Technical specifications for these datasets are summarized in Table 2. Each dataset is split into training, validation, and test sets at ratios of 60%, 20%, and 20%, respectively. Performance is measured using Mean Squared Error (MSE) and Mean Absolute Error (MAE), where MAE-
Parameter Optimization Details. We optimize all pa- rameters during the self-supervised learning stage. In the fine-tuning stage, for USHCN, PhysioNet, and Human Ac- tivity, we freeze the GCN and MAE (except for normaliza- tion layers); for MIMIC, we only freeze the MAE (except for normalization layers, position embedding, and patch projection layer).
Baselines. To establish a comprehensive benchmark for the IMTS forecasting task, we select baselines from four methodological domains. Specifically, we include: (1) MTS Forecasting: .innear (Zeng et al., 2023), TimesNet (Wu et al., 2022), PatchTST (Nie et al., 2022), Crossformer (Zhang et al., 2022), Graph GraphWaveNet (Wu et al., 2019), MT- GNN (Wu et al., 2020), StemGNN (Cao et al., 2020), Cross- GNN (Huang et al., 2023), FourierGNN (Yi et al., 2024) and VisionTS (Chen et al., 2025); (2) IMTS Classifica- tion: GRU-D (Che et al., 2018), SeFT (Horn et al., 2020), RainDrop (Zhang et al., 2022), Warpformer (Zhang et al., 2023); (3) IMTS Interpolation: mTAND (Shukla & Marlin, 2021a); (4) IMTS Forecasting: Latent ODEs (Rubanova et al., 2019), CRU (Schirmer et al., 2022), Neural Flows (Biloš et al., 2021), and t-PatchGNN (Zhang et al., 2024a).
[ i.] and L E (I; - &;)², with Ne, S. and |Q] repre-
b
MSE = O senting the ground truth, predicted value, and the number of queries, respectively.
Implementation Details. Experiments are performed on individual NVIDIA RTX 4090 GPUs For VIMTS setups, set the hidden dimensions to 32 for USHCN and Phy- sioNet, 40 for MIMIC, and 64 for Human Activity. The batch size is 32 for the two training stages of PhysioNet and Human Activity's 's pre-training stage, 64 for the fine- tuning stage of Human Activity and the two training stages of USHCN, 12 for MIMIC's ' pre-training stage, and 16 for
6

---

IMTS is Worth Time x Channel Patches: Visual Masked Autoencoders for Irregular Multivariate Time Series Prediction
Table 3. Ablation results of VIMTS on four datasets evaluated by MAE and MSE (mean + std). The best best-perforning results are highlighted
in bold
PhysioNet
Human Activity
USHCN
MIMIC
Ablation
MSEx10 10
MAEX10
MSEx10
MAEx 10
MSEx 10
MAEx10
MSEx10
MAEX 10
Complete
4.81 + 0.07
3.54 0.04
2.65 + 0.01
3.08 + 0.01
4.86 + 0.02
2.98 + 0.05
1.36 + 0.02
6.40 + 0.17
w/o Pre
6.03
3.75 0.04
0.02
0.06
4.5 + 0.01
3.05 0.08
1.39 0.02
w/o SSL
5.46 0.30
3.93 0.28
2.76 0.08
3.26
5,05 + 0.06
3.14
141
6.67 + 0.15
w/o Pre & SSL
5.70
4.24 0.33
2.84
3.32
5.01
3.06 + 0.08
1.45 0.05
6.99
0.33
w/o GCN
0.04
3.55 = 0.03
2.06
= 0.01
0.001
4.93 + 0.01
2.97 + 0.07
2.25 0.02
8,82 0.15
Tp Transformer
5.37
3.99
0.24
2.84 + 0.07
3.12
5.09 1: 0.06
3.14
1.40 0.04
6.66
Table 4. Patch2Point vs. Direct Projection of VIMTS on four datasets evaluated by MAE and MSE (mean + std). In detail, training stages
marked with v applied with Patch2Point, the others represent stages with direct projection heads. The best/worst-erffrmmng performing results are
highlighted in bold/red bold.
Patch2Point
PhysioNet
Human Activity
USHCN
MIMIC
SSL
Finetune
MSRx 10
MABX10-2
MSEx10-3
MAEx10
MSEX10
MAEX10-
MAEx10-2 MSEx10
4.81 0.07
3.54 0.04
2.65 + 0.01
3.08 + 0.01
4.86 = 0.02
2.98 + 0.05
+.30
6.10
4.97 + 0.07
3.62 + 0.05
2.72 + 0.01
3.17 + 0.02
5.03 = 0.02
3.09 0.13
1.34 + 0.01
6.37 + 0.08
4.98 +: 0.07 3.62 0.03
2.74 1: 0.01
3.20 1 0.01
5.01 + 0.04
3.10 0.10
1.38 +: 0.02
6.50 + 0.15
This selection ensures cross-methoddoogiial comparisons across regular/irregular time series forecasting, classifica- tion, and interpolation tasks, providing robust evaluation of generalization capabilities.
w/o Pre removes the visual pre-training of MAE; (3) w/o
SSL skips IMTS-specific self-supervised training; (4) w/o Pre & SSL trains the model entirely from scratch without using visual pre-training or self-supervised learning; (5) w/o GCN removes cross-channel graph convolutions; (6) rp Transformer replaces MAE with a vanilla Transformer encoder. In addition, we also explore the effectiveness of the Patch2Point prediction by replacing the coarse-to fine predictor with a Natten-projection layer in self-supervised learning stage and fine-tuning stage.
3.2. Main results
We evaluated VIMTS against 20 baseline models across different domains: clinical (PhysioNet, MIMIC), biome- chanics (Human Activity), and climate (USHCN), using MSE and MAE as performance metries, as shown in Ta- ble 1. Conventional methods like DLinear, TimesNet, and PatchTST, while effective for regular time series, struggle with IMTS due to their inability to handle irregular sam- pling and cross-channel dependencies, leading to significant errors. VSsionTS, whether using Zero' orLinear' interpo- lation for data adaptation, fails to perform well on IMTS tasks. This highlights the inadequacy of existing vision foundation model-based methods in dealing with the miss- ing values, varying data structures, and complex temporal and cross-channel dependencies of IMTS data.
As shown in Table 3 and Table 4, replacing MAE with a standard Transformer leads to a significant performance drop, demonstrating MAE's architectural advantage for han- dling semantically sparse data across multiple channels. Similarly, removing either visual pre-training (w/o Pre) or self-supervised learning (w/o SSL) results in notable perfor- mance declines, highlighting that visual priors provide valu- able initialization while SSL helps adapt to IMTS-specific characteristics. Moreover, the ablation of GCN and coarse- to-fine decoding show their effectiveness: while individual channels may suffer from missing values, GCN compensates through cross-channel dependencies. Additionally, ablation studies the Patch2Point predictor's two-stage training demonstrate its coarse-to-fin fine strategy enhances forecasting precision. It achieves this by focusing on relevant tempo- ral context and reducing interference from unrelated time periods, thereby outperforming direct timestamp prediction which susceptible to irrelevant information.
In contrast, VIMTS consistently outperforms other meth- ods, including the currently best-performing baseline, t- PatchGNN. When only 20% or 50% of training data are available, VIMTS matches the performance of t-PatchGNNN with complete data and exceeds the performance of all other methods. Increasing the utilization of training data further to 100%, VIMTS demonstrates even better performance, achieving the lowest MSE and MAE across all four real- world datasets. These results validate VIMTS's superior adaptability and effectiveness in handling IMTS forecasting tasks.
3.4. Few-Shot Learning
To evaluate the few-shot capability of our method and the contributions of visual pre-training (Pre) and supervupd learning (SSL), we conduct experiments different few- shot scenarios (10% 20%, and 50% of the entire training data) and ablation settings (complete, w/o Pre, w/o SSL and w/o Pre & SSL.). T-PatchGNN serves as a baseline model.
3.3. Ablation Study
To validate the necessity of core components in VIMTS, we conducted an ablation study comparing multiple variants.
(1) Complete represents the model without any ablation; (2)
7

---

IMTS is Worth Time x Channel Patches: Visual Masked Autoencoders for Irregular Multivariate Time Series Prediction
Table 5. Few-shot results of VIMTS on four datasets evaluated by MAF and MSE (mean # std). highlighted bold, the second-hest results are highlighted in blue bold.
The best-performing results are
Model
VIMTS
i-PaccGNN
VIMTS Pre & SSL)
VIMTS SSL)
ViMT Pre)
Dataset
Ratio|
MSE
MAEx10
MSEx10
MA 2
MSEx10->
MAEx10-2
MSEx100
MAEx10-2
MSEx
MAEx10
2.87
0.02
3.24 0.03
3.21 0.07
3.62
7:28 5.72
5.81
3:1 + 0.14
0:10
13.15 5.07
8.81 5:335
Activity
0.2
2.72 +: 0.01
3.14 0.00
3.01 0.06
3.42 + 0.08
4.24 0.22
3.45 0.14
3.00 + 0.11
8.43 0.13
4
4.14 0.08
0.5
2.69
0.01
3.11 + 0.01
2.90 +:00:000 0.10
3.35 0.06
2.91 = 0.10
3,36 2 0.14
2.91 = 0.06
3.35 0.12
2.92 0.06
3.31 + 0.06
Datasel
Ratio
MSEX10-"
MAEx10"
MSEX10"
MAEX10-2
MSE
MAEX10"
MSEX 10
MARX10"?
1SE
MABx10"2
5.16 0.26
3.73
0.17
4.1
6.03
4.25
6.35 0.71
5.93
0.06
4.17 0.06
PhysioNet
0.2
4.92 +: 0.08
3.63 0.06
7.23 1 0.22
1.21 4: 0.22
5.87 2: 0.47
4,21 4: 0.23
6.04 2 0.67
1.26 1. 0.37
5.45
0.08
3.89 0.09
0.5
4.86 à: 0.09
3.57
0.05
6.97 0.13
4.03 + 0.13
5.60 # 0.45
4.09 + 0.31
5,50 + 0.40
4.07 +: 0.26
5,29 d: 0.07
3.8 +: 0.02
Dataset
Ratio
MSEx10"1
MARx10
MSEx 10
MARX10"1
MSFx10
MAEx101
MSFx10
MARx10"
MSEx10
MARx
5.09 0.07
3.12
0.04
5.23
5.32 0.08
5.12
3.13 0.05
USHCN
0.2
5.01 + 0.02
3.06 0.04
5.11 + 0.05
3.20 + 0.07
5.26 0.13
3.18 + 0.07
5.24 = 0.04
3.16 + 0.04
5.05 +: 0.02
3.07 + 0.05
0.5
4.86 +: 0.01
2.97 0.08
5.04 à 0.03
3.07 d: 0.07
5,10 =: 0.02
3,16 d: 0.14
5,10 = 0.04
3.15 à: 0.08
4.99 +: 0.02
3.13 1 0.09
Dataset
Ratio
MSEx10
MSEx10 14Ex10
MSFx100 MARX10"2
MAEx10"
MSHX110
MARX10"2
MSEX10-2
MARx10
1.53 + 0.01
7.06
0.10
0.16
7-66 0.55
1.71
7.85
1.77 0.13
0.45
1.61 0.01
1.18 0.20
MIMIC
0.2
1.47 + 0.01
6.71 E 0.0G
1.46 1: 0.05
7.07 +: 0.32
1.64 = 0.10
7.62 +: 0.35
1.57 = 0.07
7.30 d 0.29
1.53 4 0.03
6.90 0.05
D.5
1.41 + 0.01
6.47 0.12
1.39 + 0.02
0.13
1.46 0.03
6.93 0.19
1.45 + 0.03
6.32 0.14
1.43 + 0.00
6.64 0.08
Autivity MSE
Physinhet- MSE
USHEN- MSE
MMCC- MSE
2.4
00
ISE 1100100-1
125
24
Percestage
12
16
122
Artuliyy MAE
Physolett MAE
USHCN- MMEE
MHIC MMEE
2.4
3.5
25
24
100
18
Daca Porconage
Data Percettage
Percertage
Data Porcontage
VIMTSiiee SGLI
Best before PanthGGN
W/MS/A0 Pre & 55L)
VMMTSicd
L-PaecnChN (tstlo-1
Figure 4. Performance comparison VIMTS model variants under low-resource conditions (10%. 20% 50%. and 100% data) on MSE and MAE metrics across Activity, PhysioNet, USHCN, and MIMIC datasets. Variants include models with and without visual pre-trained initialization and self-supervised learning. T-PatchGNN is used as baseline, Lower values indicate better performance.
As shown in Figure 4, VIMTS achieves superior perfor- mance across few-shot settings, with lower prediction errors and more stable response curves under varying training data availability. While t-PatchGNN exhibits sensitivity to train- ing data scarcity, VIMTS demonstrates more substantial few-shot generalization, which may benefit from integrating visual pre-training and self-supervised learning. To confirm this, we conducted ablation studies and found that removing either component may degrade performance stability. Al- though trends differ slightly across the four datasets, these results collectively highlight that the synergistic integra- tion of cross-domain visual pre- pre-training and self-supervised learning enables effective key pattern extraction from lim- ited data, thus advancing sample-efficient IMTS modeling.
4. Analysis of Computational Cost
4.1. Time and Space Complexity Comparison
Parameter Numbers. As shown in Table 6, VIMTS utilizes visual MAE-base as its backbone. This backbone can be fine-tuned on a single NVIDIA RTX 4090. Furthermore, the GCN layers in VIMTS are lightweight, contributing only around 32.4k parameters. The number of trainable pa- rameters is acceptable in real-world applications and lower than other vision foundation model- based methods, such as VITST.
Time Complexity. As shown in Table 6, VIMTS has ac- ceptable training efficiency (87.8 s/epoch) while its infer- ence speed (4.815 ms/instance) outperforms VITST (visual
8

---

IMTS is Worth Time x Channel Patches: Visual Masked Autoencoders for Irregular Multivariate Time Series Prediction
Table 6. The Temporal and Spacial Complexity of Different Methods.
Methods
VIMTS
VIMTS w/o GCN
VITST
tPatchGNN
CRU
WrapFormer
Training Time (s/epoch)
87,80
84.96
140.2
17.14
172.45
22.67
Inference Time (ms//ample)
4.82
4.34
7.09
1.60
7.66
5.48
Trainable Param
111.0IM
110.98M
212.31M
943.99k
28.65k
178.06K
foundation model, 7.094 ms/instance). CRU (Neural-ODE, 7.655 ms/instance), and WrapFormer (Transformer, 5.475 ms/instance), making it practical for real-world deployment. Though its inference is slightly slower than lightweight mod- els like t-PatchGNN (1.604 ms/instance), it's still efficient and practical in most accuracy-first application scenarios, with sub-5 ms latency.
IMTS forecasting. To mitigate the effect of missing val- ues, VIMTS processes sparse IMTS along the timeline into image-liko like patches with equal-interaals, then complements these patches with information from related channels using learned cross-channel dependencies. Then it leverages the capability of visual MAE for handling sparse multichannel data for patch reconstruction, followed by a coarse-to tonfee- technique that progressively generates precise predictions from focused context. The framework is trained with a two-stage strategy. First, eelf-supervised learning is em- ployed to enhance IMTS data modeling by adapting visual MAE's strengths to IMTS data, while supervised fine-tuning is applied as follows for task-specific adaptation. Exten- sive experiments on four real-world datasets demonstrate VIMTS' superior performance and robust few-shot capabili- ties, achieving competitive accuracy even with limited data compared to baselines trained on full datasets, paving the way for applying visual foundation models more general time series forecasting tasks.
4.2. The Trade-off between Self-Supervised Learning and Computational Cost
Self- Seff-Supervised Learning (SSL) effectively offers favor- able trade-off between computational cost and performance, which is evident in the following three aspects.
High Data Efficiency and Few-Shot Capability. With SSL, VIMTS achieves competitive results against state-of- the-art models on four IMTS datasets while utilizing only 20% of the training data. This significant reduction of re- quired data volume accelerates model development and de- ployment, especially in data-scarce environments.
Manageable Model Complexity. Our analysis demon- strates excellent performance without requiring exces- sive scaling. VIMTS typically uses approximately three lightweight GCN layers and several simple MLP predictors, aside from the MAE backbone. Previous trials show that scaling MAE beyond its base level doesn't 't improve results and often leads to memory issues, consistent with observa- tions in VS (Chen et al., 2025), another model that applies visual MAE to RTS forecasting.
6. Limitations and Future Work
While VIMTS advances IMTS forecasting, limitations per- sist in scalability and structural flexibility. For scalability, the design of a larger IMTS foundation model to achieve more powerful performance and better generalization across downstream datasets remains problem, which may be re- solved by constructing larger-scale pre-training datasets and developing well-designed fine-tuning strategies. For struc- tural flexibility, current models are limited to fixed patch sizes and the number of channels. struggling with dynamic data structures, thereby hindering true zero-shot capabil- ities without parameter tuning. Future directions should prioritize 'time-contextual scaling' mechanisms that dynam- ically adjust semantic hierarchies using timestamp metadata and general cross-channel dependency graph foundation model that flexibly handles information exchange among any number of channels.
Efficient Inference and Improved Performance. While Self Self-Superviee Learning (SSL) increases overall training time by approximately 40%, the training time per epoch is still faster compared to CRU and VITST, which remains acceptable. Importantly, SSL doesn't 't increase inference cost, allowing VIMTS to maintain faster inference speeds than most competitors while remaining competitive with t-PatchGNN. Given that real-world applications often face data limitations and prioritize inference efficiency over train- ing expenses, this trade-off, which involves accepting a rea- sonable increase in training time for improved accuracy and efficient inference, offers significant practical advantages.
5. Conclusion
This paper introduced VIMTS, pioneering framework that leverages the capability of visual pre-trained MAE for modeling semantically sparse multichannel data for

---

IMTS is Worth Time x Channel Patches: Visual Masked Autoencoders for Irregular Multivariate Time Series Prediction
Impact Statement
Esteban, C., Hyland, S. L., and Rätsch, G. Real-valued (medical) time series generation with recurrent condi- tional gans. arXiv, 2017.
This paper aims to advance research in Irregular Multivariate Time Series (IMTS) prediction. There exist many potential societal consequences of our work, but none that we feel require specifie highlighting here.
Gong, Y., Li, Z.,, Zhang, J., Liu, W., Yin, Y., and Zheng, Y. Missing value imputation for multi-view urban statistical data via spatial correlation learning. TKDE, 2021.
References
Goswami, M., Szafer, K., Choudhry, A., Cai, Y., Li, S., and Dubrawski. A. Moment: A family of open time-series foundation models. In ICML, 2024.
Bai, J. and Ng, S. Forecasting economic time series using targeted predictors. Journal of Econometrics, 2008.
Biloš, M., Sommer, J., Rangapuram, S. S., Januschowski, T., and Günnnemann, S. Neural flows: Efficient alternative to neural odes. NewrIPS, 2021.
Hamzaçebi, C. Improving artificial neural networks' perfor- mance in scasonal time series forecasting. Information Sciences, 2008.
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosse- lut, A., Brunskill, E., Brynjolfsson, E., Buch, S., Card, D., Castellon, R., Chatterji, N., Chen, A., Creel, K., Davis, J. Q., Demszky, D., Donahue, C., Doumbouya, M., and et. al. On the opportunities and risks of foundation models. arXiv, 2022.
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. Masked autoencoders are scalable vision learners. In CVPR, 2022.
Horn, M., Moor, M., Bock, C., Rieck, B., and Borgwardt, K. Set functions for time series. In ICML, 2020.
Huang, Q., Shen, L.,, Zhang, R., Ding, S., Wang, B., Zhou, Z., and Wang, Y. Crossgnn: Confronting noisy multivari- ate time series via cross interaction refinement. NeurIPS, 2023.
Brown, T,, Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., al. Language models are few-shot learners. NeurlPS, 2020.
Jin, M., Wang, S., Ma. L., Chu, Z., Zhang, J. Y., Shi, X., Chen, P.-Y,, Liang, Y., Li, Y.-F,, Pan, S., and Wen, Q. Time-llm: Time series forecasting by reprogramming large language models. arXiv, 2024.
Cao, D., Wang. Y., Duan, J., Zhang, C., Zhu, X., Huang, C., Tong, Y., Xu, B., Bai, J., Tong, J., et al. Spectral tem- poral graph neural network for multivariate time-series forecasting. NeurIPS, 2020.
Johnson, A. E., Pollard, T. J., Shen, L., Lehman, L.-w. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., An- thony Celi, L., and Mark, R. G. Mimic-iii, freely acces- sible critical care database. Scientific Data, 2016.
Che, Z., Purushotham, S., Cho, K., Sontag, D., and Liu, Y. Recurrent neural networks for multivariate time series with missing values. Scientific Reports, 2018.
Chen, M., Shen, L., Li, Z., Wang. X. J., Sun, J., and Liu, C. Visionts: Visual masked autoencoders are free-lunch zero-shot time series forecasters. In /CML,, 2025.
Jungo, J., Xiang, Y., Gashi, S., and Holz, C. Representation learning for wearable-based applications in the case of missing data, arXiv, 2024.
Chen, R. T. Q., Rubanova, Y., Bettencourt, J., and Duvenaud, D. K. Neural ordinary differential equations. In NewrlPS, 2018.
Kidger, P., Morrill, J., Foster, J., and Lyons, T. Neural controlled differential equations for irregular time series. In NuerlPS, 2020.
Das, A., Kong, W., Sen, R., and Zhou, decoder- only foundation model for time-series forecasting, arXiv, 2024.
Kipf, N. and Welling, M. Semi-supervised classification with graph convolutional networks. arXiv, 2016.
Das, M. and Ghosh, S. K. sembnet: semantic bayesian network for multivariate prediction of meteorological time series data. PRL, 2017.
Li, Z., Li, S., and Yan, X. Time series as images: Vi- sion transformer for irregularly sampled time series. In NeurlPS, 2023.
De Brouwer, E., Simm, J., Arany, A., and Moreau, Y. bayee: Continuous modeling of sporadically- observed time series. NewrlPS, 2019.
Lipton, Z. C., Kale, D., and Wetzel, R. Directly modeling missing data in sequences with rnns: Improved classifica- tion of clinical time series. In MLHC, 2016.
10

---

IMTS is Worth Time x Channel Patches: Visual Masked Autoencoders for Irregular Multivariate Time Series Prediction
Marlin, B. M., Kale, D. C., Khemani, R. G., and Wetzel, R. C. Unsupervised pattern discovery in electronic health care data using probabilistic clustering models. In ACM SIGHIT, 2012.
Yi, K., Zhang, Q., Fan, W., He, H., Hu, L., Wang, P., An, N., Cao, L.,, and Niu, Z. Fouriergnn: Rethinking multivariate time series forecasting from pure graph perspective. NeurIPS, 2024.
Menne, M., Williams Jr, C., Vose, R., and Files, D. Long- term daily climate records from stations across the con- tiguous united states, 2015.
Yoon, J., Zame, W. R., and van der Schaar, M. Estimat- ing missing data in temporal data streams using multi- directional recurrent neural networks. IEEE Transactions on Biomedical Engineering. 2018
Nie, Y., Nguyen, N. H., Sinthong, P., and Kalagnanam, J. A time series is worth 64 words: Long-term forecasting with transformers. arXiv, 2022.
Zeng, A., Chen, M., Zhang, L., and Xu, Q. Are transformers effective for time series forecasting? In AAAI, 2023.
Zhang. J., Zheng, S., Cao, W., Bian, J., and Li, J. Warp- former: multi-scale modeling approach for irregular clinical time series. ACM SIGKDD, 2023.
Rubanova, Y., Chen, R. T., and Duvenaud, D. K. Latent ordinary differential equations for trreguarly-ssmpple time series. NeurIPS, 2019.
Zhang, W., Yin, C., Liu, H., Zhou, X., and Xiong, H. Irreg- ular multivariate time series forecasting: A transformable patching graph neural networks approach. ICML, 2024a.
Schirmer, M., Eltayeb, M., Lessmann, S., and Rudolph, M. Modeling irregular time series with continuous recurrent units. In ICML, 2022.
for irregularly sampled time series. In ICLR, 2021a. Shukla, S. N. and Marlin, B. Multi-time attention networks
Zhang, W., Zhang, L.,, Han, J., Liu, H., Fu, Y., Zhou, J., Mei, Y., and Xiong, H. Irregular traffic time series forecasting based on asynchronous spatio-temporal graph convolu- tional networks. In ACM SIGKDD, 2024b.
Shukla, S. N. and Marlin, B. M. A survey on principles, models and methods for learning from irregularly sam- pled time series. arXiv, 2021b.
Zhang, X., Zeman, M., Tsiligkaridis, T., and Zitnik, M. Graph-guided network for irregularly sampled multivari- ate time series. arXiv, 2022.
Silva, I., Moody, G., Scott, D. J., Celi, L. A., and Mark, R. G. Predicting in hospittll mortality of icu patients: The physionet/compuiing in cardiology challenge 2012. In 2012 Computing in Cardiology, 2012.
Zhang, Y. and Yan, J. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In ICLR, 2023.
Van Buuren, S. and Ouddhoos, K. mice: Multi- variate imputation by chained equations in Journal of Statistical Software, 2011.
Wang, J., Du, W., Cao, W., Zhang, K., Wang. W., Liang, Y., and Wen, Q. Deep learning for multivariate time series imputation: A survey, arXiv, 2024.
Weerakody, P. B., Wong, K. W., Wang, G., and Ela, W. A review of irregular time series data handling with gated recurrent neural networks. Neurocomputing, 2021.
Wu, H., Hu, T. Liu, Y., Zhou, H., Wang. J., and Long, M. Timesnet: Temporal 2drviiittion modeling for general time series analysis. arXiv, 2022.
Wu, Z., Pan, S., Long, G., Jiang, J., and Zhang, C. Graph wavenet for deep 2019.
graph modeling. arXiv,
Wu, Z., Pan, S., Long, G., Jiang, J., Chang, X., and Zhang, C. Connecting the dots: Multivariate time series forecasting with graph neural networks. In ACM SIGKDD, 2020.

---

IMTS is Worth Time x Channel Patches: Visual Masked Autoencoders for Irregular Multivariate Time Series Prediction
A. Appendix
A.I. Related Work
A. IRREGULAR MULTIVARIATE TIME SERIES FORECASTING
Ireeularly sampled time series forecasting poses unique challenges due to non-uniform observation intervals across multiple variables (Shukla Marlin, b021b; Horn et al., 2020). Basic approaches include fixed discretization, which simplifies processing but introduces missing data issues (Marlin et al., 2012; Lipton et al., 2016), and interpolation methods that improve robustness by leveraging past and future data (Yoon al., 2018; Horn al., 2020). Recent advancements have been made with Neural ODEs (Chen et al., 2018) for continuous dynamics modeling, extended by frameworks like ODE-RNN (Rubanova et al., 2019) and Neural CDE (Kidger et al., 2020), enhancing efficiency and adaptabilityy CRU (Schirmer et al., 2022) integrates probabilistic models for better interval management, while GNN-based approaches, such as RAINDROP (Zhang et al., 2022), model data point interactions effectively for global understanding.
A.1.2. FOUNDATION MODEL FOR TIME SERIES FORECASTING
Foundation models (Bommasani et al., 2022) have transformed time series forecasting, notably through the adaptation of masked autoencoders (MAEs) originally designed for visual tasks (He et al., 2022). VisionTS (Chen al., 2025) innovates by treating time series forecasting as image reconstruction, achieving zero-shot performance through MAE's patch-level reconstruction paradigm. This method captures temporal patterns without modality-specif adaptations, outperforming language-modell-asse approaches like Time-LLM (Jin et al., 2024).
A.2. Methodology Comparison and Clarification
1.2.1. CLARIFICATION OF ORIGINALITY AND DIFFERENTIALS FROM (JUNGO ET AL., 2024).
While both works explore patchification, our core innovation lies in the exploration of visual MAE's architecture and the capability benefiting from visual initialization and self-supervised learning for unstructured IMTS data reconstruction. Beyond this, VIMTS distinguishes itself through several key aspects not addressed by (Jungo et al., 2024).
Enhanced Channel Dependency Modeling with GCN. Unlike Jungo's projection based forecasting, VIMTS leverages Graph Convolutional Networks (GCNs) to capture both static and dynamic information. This allows to model the bidirectional information flow among channels, which is more consistent with real-world information dependencies and enables explicit cross-channel compensation to effectively impute missing values. Jungo et al.'s work (Jungo et al., 2024) solely relies on projection, lacking this sophisticated cross-channel dependency modeling. Our ablation study confirms the significant performance boost from our GCN module.
Leveraging Vision Pretraining. A crucial aspect of VIMTS is our explicit utilization and exploration of vision pretraining's foundational capabilities and their importance for the model's overall performance. This vital component, which provides a strong initialization for sparse pattern learning and few-shot learning, not mentioned or explored in (Jungo et al., 2024).
Fine-Grained Time Channel Prediction. VIMTS employs a coarse-to torfiee prediction strategy, allowing for more precise predictions at specific time segments and channels. In contrast, Jungo's projection-basee approach operates at coarser level. We have even conducted experiments with an encoder architecture using direct projection at different training stages, similar to (Jungo et al., 2024), which yielded inferior results, further highlighting the advantage of the proposed VIMTS.
A.2.2. COMPARISON WITH OTHER MASK-RECON RER--REEOOOTTTTOOOO METHODS
Our experiments include of-t-of-the-ard art masking-based transformer variants such as PatchTST and VisionTS, and demon- strate VIMTS' superior performance, which benefits from its ability to effectively handle IMTS irregularities and missingness. While the transformer-based models with mask reconstruction, PatchTST (Nie et al., 2022) have shown strong performance on regularly sampled Multivariate Time Series (MTS) data, they lack explicit mechanisms to handle irregular sampling prevalent in IMTS data. Their design also does not incorporate cross-channel modeling, which is crucial for imputation in the presence of missingness, leading to significant performance drops. Moreover, VisionTS (Chen et al., 2025), a vision-based foundation model with masked reconstruction, performs well on MTS but fails to generalize to IMTS tasks due to its reliance on rigid grid-like patch resizing and normalization, which may cause information loss. Similarly, MOMENT (Goswami al., 2024), another transformer-base foundation model with mask-reconstruutton underperforms even PatchTST MTS
12

---

IMTS is Worth Time x Channel Patches: Visual Masked Autoencoders for Irregular Multivariate Time Series Prediction
tasks and struggles to address irregular sampling or missing value issues.
A.2.3. COMPARISON WITH VITST (LI ET AL., 2023)
Although both ViTST and VIMTS employ image-like representations for time series, there are key distinctions in Innovation Points: To extract information from IMTS data, VITST deals with missingness with interpolation and by transforming IMTS to images, which causes information loss with additional computational overhead, makes inputs less precise for understanding patterns in data, and fails to perform well in forecasting. In contrast, our model divides data by time intervals, patchifies it into time x channel patches, and extracts block features without interpolation, which preserves data integrity and creates more precise inputs for MAE to model internal data structures without computational overhead from image construction. Further, it explicitly models cross-channel interaction with GCN, thereby compensating for missingness across channels. Our analysis is further supported by empirical evidence, which evaluates the model performance on PhysioNet.
Method
MSE(x10 3)
MAE(x 10
VIMTS (Ours)
4.81 # 0.07
3.54 0.04
VITST
66.37 = 0.08
20.16 2 0.05
Vision (Zero Interp.)
42.41 # 0.02
13.13 # 0.02
TS (Linear Interp.)
40.50 2: 0.05
12.85 0.01
Table 7. Comparison of different methods.
Note that this experiment also involves VisionTS (Chen al., 2025) with similar image-based methods, confirming that despite its powerful visual MAE framework and strong performance on regularly sampled time series, it similarly deteriorates when processing irregular data.
As for computational cost, VIMTS employs a visual MAE-base with around 3 GCN layers as its backbone. Our hyperparam- eter experiments show that in general settings, a relatively lightweight configuration is optimal, with no significant benefits from additional complexity. In comparison, ViTST uses a Swin Transformer as the visual backbone and a RoBERTa as the text backbone, and requires additional computational cost from image construction, leading to considerable overall costs. The quantified experimental results shown in Table 6 further validate our claims.
A.3. Discussion about the Effectiveness of Self-Supervised Learning
Our self-supervised learning (SSL) strategy effectively adapts the capability of vision pre-training to series data, leading to robust few-shot capability. In detail, with the GCN module, which learns cross-channel dependencies and complements missing values with information from other channels, VIMTS effectively transforms IMTS into regularly sampled multivariate time series (MTS) at the feature level. Consequently, given that mask-reconstruutionn-be based SSL is effective in learning temporal dependencies in MTS, applying it to enhance IMTS modeling after cross-channel complemen- tation is justified. Furthermore, as the vision pre-training is initially performed on 3-channel data (R-G-B), our strategy is crucial for adapting the model to more diverse application scenarios with greater number of channels. Our ablation study and fho- experiments clearly demonstrate a trend: by learning from -ppeiii time-channel contexts through SSL, our model can effectively generalize from the 3-channel pre-trained state to handle more varied, complex, and limited data.
time
A.4. Analysis of Fine-Tuning Strategies
Table 8. Comparison of Different Finetune Strategies.
denotes variant of the existing strategy.
PhysioNet
Human Activity
USHCN
MIMIC
Finetune
MSFx
MARx 10 2
MSEx
MARx 10 2
MSFx 10
MARx 10 2
MSEx 10
MAEx 10
ALL
4.95 + 0.06
3.53 = 0.06
2.69 0.01
3.31 = 0.01
4.91 0.04
2.98 + 0.15
1.39 0.02
6.59
Attn
4.91
3.55 0.06
2.67 0.01
3.12 0.01
4.00
2.91 0.03
1.01
6.09
Bias
0.05
0.08
0.00
3.08 0.01
4.85
2.96 - 0.06
1.00
6.508
Frezze
4.98 0
3.04
2.01
4.00
2.05
1.43
6.65 6.11
MLP
4.91 0.03
3.51 0.04
2.68 0.01
0.01
4.04
2.99
1.39 0.01
6.44
Norm
4.81 0.07
3.54 0.04
2.65 0.01
3.08 0.01
4.86
2.98 1 0.05
1.36
+ 0.02"
6.40
To identify the optimal fine-tuning strategy for maximizing the potential of our architecture, we systematically evaluate three distinct approaches, with the results demonstrated in Table 8. (1) ALL: Updates all model parameters; (2) Freeze:
13

---

IMTS is Worth Time x Channel Patches: Visual Masked Autoencoders for Irregular Multivariate Time Series Prediction
Retains the pre-trained MAE and the GCN module, and optimizes the remaining parts; (3) Partial Tuning: Selectively updates specific components within the retained parts of Freeze, including Attn (attention layers), Bias (bias terms), MLP (MLP blocks), and Norm (normalization layers), A variant of Norm, marked with **, includes GCN, normalization layers, position embeddings, and patch projection layers, and is adapted for datasets with greater number of channels.
Experiments on three IMTS datasets (PhysioNet, Human Activity, USHCN) reveal that fine-tuning solely the normalization layers (Norm) achieves the best overall results, offering an optimal balance in performance across datasets and metrics, Therefore, we have chosen the Norm strategy as our default fine-tuning method, which allows for efficient adaptation and maintains the semantic sparse modeling capabilities aequired from visual pre-training. For the MIMIC dataset, which has a larger number of channels, we utilize the variant of Norm marked with **' This adjustment enables the capture of more intricate cross-channel dependencies while retaining the advantages of the standard Norm strategy. Comparisons of various tuning strategies on the MIMIC dataset confirm that this approach delivers the best overall performance.
A.5. Hyperparameter Sensitivity
We analyze the sensitivity of critical hyperparameters: hidden dimension, patch size, mask ratio, GCN layer depth, and time embeddings dimenson in TTCN (TE) and graph vertex embeddings dimenson in GCN (VB), on all four datasets (PhysioNet, Human Activity, USHCN, MIMIC). We vary each parameter's value while fixing others to their optimal settings derived from preliminary experiments.
We test different hidden dimension to balance model performance and computational efficiency. Observations across all three datasets (PhysioNet, Human Activity, USHCN, MIMIC) in Fig. 5 indicate that hidden dimension size of 32 yields the optimal results for Physionet and USHCN, 64 for Human Activity, while 40 for MIMIC
Hidden Dimension.
x25
110
xte-
2,700
5.15
1.37
4.50
1.36-
5.35
3.50
4.35
4.55
1.34
1002
125
Pilzenn Dineniee
Hidden Dimersion
H42den D'mensioo
Hilzon D'ieersten
(a) Human Activity
(b) PhysioNet
(c) USHCN
(d) MIMIC
Figure 5. Sensitivity of Hidden Dimension
Patch Size. As shown in 6ig.6, we evaluate patch sizes to find the optimal temporal gramularity for each dataset. Too small sizes lack sufficient information due to data sparsity and may cause memory issues, while too large sizes miss fine-grained temporal changes. Based on the results, we select a patch size of 300 (time steps) for Human Activity, 8 for PhysioNet and MIMIC, and for USHCN
X12-2
236.
5,075
5.2
234
5. 450
1.38-
232
5.025
5.002
270
4.973
4.0
4.60
134
103
300
750 Paich Size
1500
1
6
12
12
Pacth She
Patch Siee
Phanis Sit
(a) Human Activity
(b) PhysioNet
(c) USHCN
(d) MIMIC
Figure Sensitivity of Patch Size
Mask Ratio. During self-supervised learning, we vary mask ratios from 0.1 0.9 in Fig. 7. For the Human Activity dataset, ratio of 0.7 provides the best performance. The PhysioNet dataset, which is larger and faces out-of-memory issues, benefits most from ratio of 0.6. For USHCN and MIMIC, which is larger than PhysioNet, the ratio of 0.4 is optimal.
14

---

IMTS is Worth Time x Channel Patches: Visual Masked Autoencoders for Irregular Multivariate Time Series Prediction
1324
K10-
2.71
3A-
5.4
1.29
52-
2.48
30
4.8
1.34
0.4 Mosk tato
98
0.7
0.1
02
Mosk
Mast Rado
Miac ho00
(a) Human Activity
(b) PhysioNet
(c) USHCN
(d) MIMIC
Figure 7. Sensitivity of Mask Ratio
GCN Layer Depth. Testing GCN layers from to find that the optimal depths are 2 for the Human Activity dataset, 3 for PhysioNet, USHCN and MIMIC, in Fig. 8. These configurations provide the best balance between model complexity and performance, ensuring effective learning without unnecessary computational overhead and risking overfitting.
x10
2.73
W
X100%
1.475-
x13-1
277
4.800
1.453-
4.850
1.373
4.125
1.300
(a) Human Activity
(b) PhysioNet
USHCN
(d) MIMIC
Figure 8. Sensitivity of GCN Layer Depth
TE and VE Dimension. For effective time x channel feature extraction, as shown in Fig. 9, we test different time TE and VE dimension. For Human Activity and PhysioNet, 5 is optimal. For USHCN, 10 the best. And for MIMIC, 40 the best.
2.74
X12
4.300 2.10%
1.44
4.925
1.42
4.300
1.40
2.70
3.32
4.975
-02
20
50
51013
80
Humaa Activity
(b) PhysioNet
USHCN
(d) MIMIC
Figure 9, Sensitivity of TE and VE Dimension
15

---

IMTS is Worth Time x Channel Patches: Visual Masked Autoencoders for Irregular Multivariate Time Series Prediction
B. Baseline
The performances of the models marked with 'f" are reported from (Zhang et al., 2024a), as they share the same task setting, evaluation protocols, and datasets split ratios.
B.1. Explanation of Results on MIMIC Dataset
We identified issues in t-PatchGNN's preprocessing pipeline for MIMIC (refer to their GitHub issues), thus failing to reproduce their results. Utilizing the same configuration as the original paper, after re-evaluating with corrected preprocessing in normal and few-shot settings, VIMTS exhibits competitive MSE, superior MAE compared to I-PatchGNN, and robust few-shot capability. This validates that VIMTS is able to effectively scale to complex and real IMTS data.
B.2. MTS Forecasting
DLineart (Zeng al., 2023) decomposes the time series into trend and remainder components using a moving average kernel, then applies two single-layer linear networks model each component for forecasting. This approach enhances prediction performance on data with clear trends by explicitly handling the trend component.
TimesNett (Wu et al., 2022) introduces novel approach for time series analysis by transforming ID time series into 2D tensors to capture both intra-perion and inter-period variations, thereby enhancing representation capability. utilizes TimesBlock, a task-general backbone featuring a parameter-efficien inception block, which can adaptively discover multi- purti-yeridiictty and extract complex temporal dynamics from the transformed 2D tensors for improved forecasting accuracy.
PatchTST+ (Nie et al., 2022) is Transformer-based model that segments time series into subseries-level patches as input tokens and employs channel independence, allowing each univariate time series to share the same embeddings and Transformer weights. This design retains local semantic information, significantly reduces computational and memory demands, and enables the model to consider a longer history.
Crossformer+ (Zhang Yan, 2023) is a Transformer- based model designed to address multivariate time series (MTS) forecasting by effectively capturing both cross-time and cross-dimension dependencies. It utilizes a Simension-Segment Wise (DSW) embedding preserve time and dimension information, followed by a Two-Stage Attention (TSA) layer to model these dependencies efficiently. Through its Hierarchical Encoder-Decoder (HED) structure, Crossformer integrates information at various scales for enhanced forecasting performance.
Graph Wavenet (Wu et al., 2019) is a CNN-based method that utilizes a self-adaptive adjacency matrix, learned through end-to-end supervised training. to capture hidden spatial dependencies in graph data. It employs stacked dilated causal convolutions to efficiently model long-range temporal dependencies with an exponentially growing receptive field. This enables Graph WaveNet effectively handle spatial-temporal graph data for forecasting, combining cross-dimension and cross-time dependency modeling with a gated mechanism.
MTGNN+ (Wu et al., 2020) tackles spatial and temporal dependencies through a novel graph learning layer, graph convolution module, and a temporal convolution module. It extracts a sparse graph adjacency matrix adaptively based on data to address spatial dependencies, specifically designed for directed graphs to avoid over-smoothing. The temporal convolution module uses modified ID convolutions to discover temporal patterns with multiple frequencies and handle very long sequences, effectively capturing cross- cross-dimensional relationships and cross-temporal dependencies.
StemGNNt (Cao al., 2020) is designed model intra-series temporal patterns and inter-series correlations by transforming multivariate time-series data into the spectral domain using Graph Fourier Transform (GFT) and Discrete Fourier Transform (DFT). This approach enables clearer pattern recognition and more effective predictions by converting structural multivariate inputs into orthogonal time-series representations and then further into frequency domain representations.
CrossGNN+ (Huang et al., 2023) models MTS forecasting by constructing multi-scale time series with varying noise levels using an Adaptive Multi-Scale Identifier (AMSI). then applies a cross-scale GNN to capture dependencies between different scales and cross-variabl GNN to handle homogeneity and heterogeneity among variables, using positive and negative edge weights. By focusing on high-saliency edges, CrossGNN achieves linear complexity.
FourierGNNt (Yi et al., 2024) improves MTS forecasting by transforming features into Fourier space to handle large-scale graphs more efficiently. Using Fourier Graph Operators (FGO) instead of traditional graph operations, it performs matrix
16

---

IMTS is Worth Time x Channel Patches: Visual Masked Autoencoders for Irregular Multivariate Time Series Prediction
multiplications in Fourier space, achieving log-linear complexity and high expressiveness. Stacking FGO layers enables effective pattern capture with reduced computational load. Theoretical analysis confirms FGO's equivalence time-domain graph convolutions.
B.3. IMTS Classification
GRU-Df (Che et al., 2018) is GRU-based model designed to handle irregularly sampled time series by incorporating representations of missing data patterns through masking and time intervals. Masking indicates which inputs are observed or missing, while time intervals, enhanced with a decay term, capture the patterns of input observations.
S (Horn et al., 2020) reimagines time series classification by treating time series as a set of observations, bypassing the need for ordered sequence processing, which can be disadvantageous in scenarios with irregular sampling or unsynchronized measurements. SEFT leverages advanced set function learning classify unaligned and irregularly sampled time series, offering improved classification performance and scalability.
RainDropt (Zhang et al., 2022) is a graph neural network designed to model the temporal dynamics and evolving relationships of sensor dependencies in irregularly sampled multivariate time series. By leveraging neural message passing and temporal self-attention, RAINDROP adapts to cross-sample shared relationships between sensors and dynamically estimates unaligned observations, improving the accuracy and interpretability of predictions.
Warpformert (Zhang et al., 2023) a Transformer-based model that handles intra-series irregularities and inter-series discrepancies by using specialized input representation. This encoding captures signal values, sampling times, and intervals. A warping module then aligns all series unified scale through down-sampling up-sampling. Subsequently, a doubly self-attention module processes the synchronized data for enhanced representation learning, improving the handling of irregular time series and predictive performance.
B.4. IMTS Interpolation
TANDD (Shukla & Marlin, 202 handle multivariate, sparse, and irregular time series using a continuous-time approach with learned embeddings and a time attention mechanism. This model re-represents time series data at fixed reference points, using an encoder to convert irregular inputs into fixed-length latent representations and a decoder for reconstruction or forecasting. By replacing fixed similarity kernels, mTANS offer greater flexibility and can be adapted for forecasting by modifying the interpolation queries.
B.5. IMTS Forecasting
Latent-ODE# (Rubanova et al., 2019) enhances traditional RNNs by modeling continuous-timg dynamics with neural ODEs. serves as both a standalone autoregressive model and a recognition network the Latent ODE model, which evolves an initial latent state over time for generating time series data. This approach integrates continuous-tima dynamics into RNNs, offering better management of continuous-time data and more expressive temporal patterns.
CRU+ (Schirmer et al., 2022) is a probabilistic architecture for modeling irregularly sampled time series, mapping observations into a latent space governed by linear SDE. Using the Kalman filter's 's continuous distrete formulation. CRU propagates latent states and integrates new observations. This approach provides explicit uncertainty estimates, ensures optimal state updates in locally linear spaces, and enables analytical resolution of latent states, thus avoiding numerical integration.
Neural Flow* (Biloš al., 2021) is a neural network approach that directly models the solution curves of ordinary differential equations (ODEs), eliminating the need for expensive numerical solvers required in traditional methods. By designing flow architectures that meet specific conditions, the model significantly improves computational efficiency while retaining the modeling capabilities of neural ODEs.
t-PatchGNNt (Zhang et al., 2024a) transforms univariate irregular time series into transformable patches with a unified time horizon, bypassing pre-alignment and capturing richer local semantics. This approach aligns IMTS in a consistent temporal resolution, addressing asynchrony issues. It uses a time-aware convolution network to encode patches into latent embeddings, which are processed by a Transformer for intra-series dependencies. Time-adaptive graph neural networks then model inter-series correlations through dynamic graphs, with final MLP layer generating forecasts based on the comprehensive latent representation.
17

---

IMTS is Worth Time x Channel Patches: Visual Masked Autoencoders for Irregular Multivariate Time Series Prediction
Vision'S (Chen al., 2025) explores the of pre-trained visual models for time series forecasting (TSF) by interpreting pixel variations in images temporal sequences. This approach leverages the similarities between images and time series, such their continuous nature, real-world origins, information density, and shared features. Focusing visual masked autoencoder (MAE), a popular computer vision model, VisualTSS reformulates TSF as patch-level image reconstruction task. By transforming ID time series data into 2D matrices and rendering them as images, the method aligns the forecasting window with masked image patches, enabling zero-shot forecasting without further adaptation. This innovative approach bridges the gap between pre-training on images and downstream TSF tasks, offering promising direction for leveraging visual models in TSF. During training, it applies learning rate le 44 and and aligns the input sequences uniform temporal grid via linear or zero interpolation, with interpolation resolution of 30 points per patch for the output.
18


Reviews:
[{"review_id": "utwugxJOdE", "reviewer": "Reviewer_vWeW", "summary": "This paper proposes a new approach for Irregular Multivariate Time Series (IMTS), characterized by unaligned multichannel signals and massive missing values. Instead of modeling temporal and channel patterns in isolation, as in most of current research, this paper proposes VIMTS, a framework that adapts Visual Mask autoencoder (MAE) for IMTS prediction and jointly model temporal-channel patterns. The intuition is that formatting IMTS into image-like patches will utilize cross-modality learning. The core idea is to utilize a pre-trained vision MAE (He et al., 2022) to model time period patches of IMTS data with multichannel information.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate the reviewers' recognition of our method's **novelty**, **comprehensive experiments**, and **potential for broad application**. We address your concerns below:\n\n\n## Q1: Originality and differentials from Jungo et al.\n\nWhile both works explore patchification, our core innovation lies in：\n\n1.  **Enhanced Channel Dependency Modeling with GCN:** VIMTS employs GCNs to learn both static and dynamic information in a asymmetric manner, therefore model bidirectional inter-channel dependencies, enabling explicit cross-channel compensation for missing values—which is more consistent with the real information dependence situation than Jungo et al.'s projection-only method. Ablation studies verify Its effectiveness.\n2.  **Leveraging Vision Pretraining:** We utilize vision pretraining capabilities, providing strong initialization for learning sparse patterns from limited data. It's a vital component not explored in Jungo et al.\n\n3.  **Coarse-to-fine Strategy**: As confirmed in our ablations, VIMTS forecasting with a encoder-decoder architecture, enabling more precise predictions generated from related time-segment and channel contexts, outperforming direct projection the approach like Jungo et al., which generates prediction through global projection features\n\n## Q2: Comparison to other transformer based baselines with mask reconstruction\n\nOur experiments have included SOTA Mask-Reconstruction (MR) transformer variants like PatchTST and VisionTS, demonstrating VIMTS's superior performance:\n\n**Limitations of Existing Methods on IMTS:** While the transformer with MR like **PatchTST** [3] perform well on regular sampled multivariate time series (MTS) data, they lack explicit mechanisms to handle irregular sampling and inter-channel modeling for missingness complementation. This leading to significant performance drops in IMTS data. Moreover, **VisionTS** [2], a vision foundation model with MR, performs well on MTS but fails to generalize to IMTS tasks due to its reliance on gridding, resizing, and interplotation when processing IMTS. Similarly, **MOMENT** [1], a time series pre-trained foundation model with MR underperforms even PatchTST in MTS forecasting (MOMENT - Table [1]) and cannot address irregular sampling or missing value issues.\n\n**Ablation Study Reinforces VIMTS's Design:** As mentioned in the comparison to Jungo's model, encoder-only architectures struggle with IMTS data even when using self-supervised learning. Our coarse-to-fine prediction strategy proves essential for achieving precise time-segment and channel-specific forecasts.\n\n## Q3: Revisions of notations and definitions\n\nFor improved clarity and technical rigor, we will add a **notation table** in the appendix and better integrate figures/tables throughout the manuscript.\n\nRegarding the ambiguity you've mentioned:\n\n* $N$: total number of channels; $L$: count of unique timestamps.\n* $q_j^n$: $j$-th query timestamp in channel $n$.\n* $y_i$: ground-truth value at $i$-th query timestamp; $\\hat{y}_i$: model's prediction.\n\n## Q4: Model performance and efficiency trade-off\n\nOur self-supervised learning (SSL) effectively offers a **favorable trade-off** with:\n\n* **High Data Efficiency and Few-Shot Performance:** With SSL, VIMTS achieves almost **SOTA** on three IMTS datasets using only **20% of the training data**. This significantly reduces the required training data volume, enabling faster model development and deployment in data-scarce scenarios.\n\n* **Manageable Model Complexity:** Our analysis shows **excellent performance without excessive scaling**, requiring only 2-4 GCN layers and several simple mlp predictor except for the MAE backbone. In more previous trials, scaling MAE beyond the base level did not improve results and caused memory issues, aligning with the case in VisionTS [2], which applies visual MAE to MTS forecasting.\n\n* **Improved Performance with Controllable Computation Cost:** On PhysioNet, our ssl delivers over 10% performance improvement for ~40% additional training time—a worthwhile trade-off for three reasons:\n    1.Our model relies on less training data for the same results, mitigating the data constraints.\n    \n    2.Our training of each epoch remains faster than ODE-based and other vision based methods (**Reviewer vVGF Q3**) and is acceptable in real-world applications. \n    \n    3.SSL adds no inference cost. VIMTS maintains faster inference than most competitors while staying competitive with t-PatchGNN. Since real-world applications deployment scenarios often face data limitations and prioritize inference efficiency over training costs, and this cost is also acceptable, consequently this trade-off provides substantial practical benefits.\n\n## Reference\n\n[1] MOMENT: A family of open time-series foundation models, ICML24\n\n[2] Visionts: Visual masked autoencoders are free-lunch zero-shot time series forecasters, arXiv:2408.17253\n\n[3] A Time Series is Worth 64 Words: Long-term Forecasting with Transformers, ICLR23.", "claims_and_evidence": "The claims that the proposed approach addresses the aforementioned issues are generally well supported. See detailed comments and suggestions in sections below.", "methods_and_evaluation_criteria": "The architecture in this paper consists of three main components: time×channel patchify, time-wise reconstruction, and the patch2point Prediction. The authors claim the time*channel patchification is one of the innovations of this paper. However, this approach seems to be similar to the approaches used in other papers, such as \nJungo, J., Xiang, Y., Gashi, S., & Holz, C. (2024). Representation Learning for Wearable-Based Applications in the Case of Missing Data. *Human-Centric Representation Learning workshop at AAAI 2024*.\nIt is also a natural way to adopt the vision MAE to the time series. Therefore, please clarify the difference of your approach and innovation.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "Experiments are included and results show the proposed method perform well comparing to baselines. The paper also includes a comprehensive ablation study to check the role of each component of the model architecture. \nThe major comment is that while there are about 20 baselines selected, not all of them are state of the art. In particular, the paper would benefit from comparing to approaches that use the transformer architecture and incorporates the masking approach.", "relation_to_broader_scientific_literature": "Although several studies have explored applying Vision Masked Autoencoder (MAE) methods to time series data, the challenges posed by irregular sampling frequencies and significant missing data remain common and critical, particularly in healthcare applications. This paper addresses these gaps and has good potential for broad application in practical scenarios.", "essential_references_not_discussed": "The literature review of this paper is comprehensive.", "other_strengths_and_weaknesses": "The notation part could use more detailed definitions. For example, please clarify the definition of notation N and L. While their meaning can be interpreted from the context, it would be more rigorous to formally define them and explain their practical meaning in applications. A thorough check of the notation and the formulae is recommended.\n\nIn line 312, the general definitions of MAE and MSE are given. However, it is not clear the definition of $y$ in the context of the tasks of this paper. Please further specify the definition of such errors.", "other_comments_or_suggestions": "It would be helpful to discuss the trade-off between the model's performance and computational efficiency, particularly given the multi-step nature of the proposed approach.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "None"}}, {"review_id": "GHnqgZzhJV", "reviewer": "Reviewer_kMse", "summary": "This paper presents VIMTS, which exploits the ability of visually pre-trained MAEs to model semantically sparse multi-channel data for IMTS prediction. Specifically, IMTS data is treated as image-like patches across temporal and channel dimensions during the encoding process, which are divided into equally spaced patches where TTCN extracts in-channel features and GCN aggregates cross-channel information to complement the patch representation with missing values. For decoding, VIMTS employs a coarse-to-fine technique that first generates patch-level predictions and then refines them into precise time-point predictions. Extensive experiments on real datasets validate the effectiveness of VIMTS.\n## update after rebuttal\nThe author's response has solved some of my confusion and I keep my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your recognition of the **novelty** and our efforts to leverage the **multi-channel semantic sparse information modeling capability** and **time series domain adaption** of visual MAE. Regarding the ambiguity you pointed out and your other concerns, we address them below.\n\n## Q1: Revisions to notations in Figure 3\n\nWe appreciate your careful review about the ambiguity in the notations used in Figure 3. To clarify the specific notations you mentioned:\n- $L_{ssl}$: This represents the L1 loss value calculated during the self-supervised learning phase.\n- $L_{ft}$: This denotes the L1 loss value calculated during the fine-tuning phase.\n- $Φ(t)$: This represents the time embedding for a given timestamp t, with the explicit formula provided in Equation (1), we utilize it to generate precise prediction from the matching time period and channel representations.\nWe will incorporate these detailed explanations into the caption of Figure 3 to enhance its clarity. Furthermore, we will include a comprehensive notation summary table within the appendix to improve overall readability. \n\n## Q2: Suggestions about validating the effectiveness of self-supervised learning\n\nThank you for highlighting the importance to validate the **effectiveness of self-supervised learning**. Our framework inherently benefits from multi-channel datasets. Our self-supervised learning (SSL) strategy effectively **adapts the vision pre-trained model to multi-channel time series data**, enabling our model to more **efficiently extract information from different time intervals** and improve performance on forecasting tasks. We will address this issue from both theoretical and empirical perspectives.\n\n**Theoretical Analysis**:\nWith the **GCN module**, which captures inter-channel interactions and complements missing values, VIMTS effectively transforming the IMTS problem into a regular sampled multivariate time series (MTS) problem. Given that MAE-based SSL is effective in learning temporal structures under complete data conditions, thus our SSL framework can leverage these representation to enhance the IMTS modeling. This theoretical advantage leads to improved performance on multi-channel datasets like MIMIC.\nFurthermore, as the vision pre-training is initially performed on 3-channel data (**RGB space**), our SSL strategy is crucial for adapting the model to more diverse application scenarios with a **greater number of channels**. By learning from other channels and temporal contexts, our model can effectively generalize from the 3-channel pre-trained state to handle more varied and complex data.\n\n**Empirical Analysis**:\nWe sincerely appreciate the reviewer’s suggestion to validate our method on high-dimensional datasets. Following this guidance, we conducted new experiments on the MIMIC dataset (with $96$ channels) and compared against the state-of-the-art t-PatchGNN. Relevant data and a supplementary figure are available in this website:\n\n **[anonymous repo]**(https://anonymous.4open.science/r/VIMTS-Rebuttal-028D/). \n\nOur ablation study and few-shot experiments clearly demonstrate a trend: with the SSL strategy, our model exhibits significant **data efficiency and strong performance even with limited data**. This supports our argument that SSL is effective in multi-channel scenarios.\n\nMoreover, our results demonstrate that VIMTS achieves superior performance compared to the current state-of-the-art t-PatchGNN, as shown in the table below: (mean of the results from 5 seeds)\n\n| Metric | VIMTS      | t-PatchGNN (Reported) |\n|--------|----|-----------------------|\n| MSE    | $1.3654 \\times 10^{-2}$ | $1.69 \\times 10^{-2}$     |\n| MAE    | $6.4482 \\times 10^{-2}$ | $7.22 \\times 10^{-2}$     |\n\nNotably, we identified inconsistencies and bugs in t-PatchGNN’s preprocessing pipeline for MIMIC (refer to their GitHub issues). Upon re-evaluating their model with corrected preprocessing, VIMTS still exhibits competitive MSE (vs. t-PatchGNN’s $0.013608 \\pm 0.000179$) and superior MAE (vs. $0.0656 \\pm 0.001141$). This validates that VIMTS not only scales effectively to high-dimensional IMTS but also generalizes robustly across varying missingness patterns. These additional experiments further confirm our theoretical analysis: the GCN-enhanced SSL framework more effectively leverages inter-channel correlations, leading to stronger performance in real-world IMTS scenarios. We will include these results in the revised manuscript.", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "yes", "experimental_designs_or_analyses": "yes", "relation_to_broader_scientific_literature": "This paper is inspired by the multi-channel semantic sparse information modelling capability and time series domain adaptation of visual MAE, which effectively improves IMTS prediction performance.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths：\n1, VIMTS performs well. Compared to previous methods such as T-PATCHGNN, VIMTS shows better prediction performance on three datasets.\n\n2, The methodology is novel. Inspired by VAE, VIMTS combines supervised fine-tuning with self-supervised learning to enhance IMTS modelling.\n\nWeaknesses:\n\n1, Figure 3 is difficult to understand, what does Lssl, Lft, Φ mean? Suggest adding figure notes.\n2，The datasets listed in Table 1 all have a relatively small number of variables, which does not adequately validate the effectiveness of self-supervised training in irregular multivariate time series forecasting tasks. Some datasets with more number of variables are suggested to be added. For example, the MIMIC dataset.\n3，", "other_comments_or_suggestions": "No", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3ia2g30Ohz", "reviewer": "Reviewer_vVGF", "summary": "The paper introduces VIMTS, a framework adapting MAE for IMTS prediction, addressing challenges like unaligned signals and missing values. Unlike existing methods that separately model temporal and channel patterns, VIMTS enhances representation learning by transforming sparse signals into image-like patches, capturing cross-channel dependencies. The model leverages MAE’s pre-trained capability for patch reconstruction and refines predictions with a coarse-to-fine approach. Additionally, it integrates self-supervised learning with supervised fine-tuning. Extensive experiments demonstrate VIMTS’s performance and few-shot capability, expanding the application of visual foundation models to time series prediction.\n\n-------Reply after rebuttal:\nThank you for the response, it addresses most of my concerns. However, after considering multiple factors, I’ve decided to maintain my original score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your recognition of the practical significance and our innovation for improving IMTS forecasting performance, as well as the insightful feedbacks. Below, we address each concern raised by the reviewer in detail:\n\n## Q1: Incorrect claims regarding pre-trained models on MTS\n\nWe will clarify in the revision that our claim applies specifically to IMTS forecasting scenarios, while most existing pre-trained model based forecasting methods are for regular sampled time series data.\n\n## Q2. Comparison with ViTST\n\nWe appreciate the reviewer's suggestion. Although both ViTST [1] and VIMTS both employ image-like representations for time series, there are key distinctions:\n- **Differences in Innovation Points**: To extract information from IMTS data, **ViTST** deal with missingness with **interpolation and transformation from IMTS to images**, which causes **information loss with computational overhead**, makes inputs less precise for understanding patterns in data, failing to perform well in forecasting. \n\n  In contrast, **our model divides data by time intervals, patchifying to time x channel patches and extracts block features** without interpolation, which preserves **data integrity** and creates more precise inputs for MAE  to model internal data structures without computational overhead of image construction.  Further, it explicitly models inter-channel interaction with GCN, compensating for missingness across channels.\n  \n  Our analysis is further supported by empirical analysis, which evaluates the model performance onPhysioNet.\n\n|Method| MSE($10^{-3}$) | MAE($10^{-2}$) |\n| - | - | -|\n| VIMTS (Ours)| 4.81 ± 0.07| 3.54 ± 0.04|\n| ViTST| 66.37 ± 0.08| 20.16 ± 0.05|\n| VisionTS (Zero interpolation) | 42.41 ± 0.02 | 13.13 ± 0.02 |\n\n\n  Note that this experiment also involves VisionTS [2] with similar image-based methods, and confirms that despite its powerful visual MAE framework and strong performance on regular time series, it similarly deteriorates when processing irregular data.\n\n- **Computational Cost**: **VIMTS** employs a visual MAE backbone (base) with limited number (around 3) of GCN layers. Our hyperparameter experiments show that a lightweight configuration (node features - 5 ~ 10 dimension, and projectors - 32 * 32 ~ 64 * 64) is optimal, with no significant benefits from additional complexity. \n\n  In comparison, **ViTST** uses a Swin Transformer as the visual backbone and a Robert as the text backbone, resulting in considerable computational costs, the quantified experimental results shown in the table in Q3 further validate our claims.\n\n## Q3. Time and space complexity analysis\n\nWe acknowledge the importance of complexity analysis, and will include the table below comparing parameters and inference times in the revision. \n\n| | VIMTS(base)| VIMTS(base) w/o GCN  | ViTST   | tPatchGNN | CRU  | WrapFormer |\n| - | - | - | - | - | - | - |\n| **Trainable Param**| Pretrain: 111.01M Finetune: 332.61k | Pretrain: 110.98M Finetune: 332.61k | 212.31M | 943.99k  | 28.65k  | 178.06K   |\n| **Avg Training Time (s) / epoch**  | 87.80   | 84.96 | 140.20   | 17.14 | 172.45 | 22.67 |\n| **Avg Inference Time (ms) / instance** | 4.82  | 4.34  | 7.09   | 1.60 | 7.66   | 5.48  |\n\n**Parameter Numbers**: VIMTS utilizes visual MAE-base as backbone, which can be finetuned on a single RTX 4090, the GCN layers in VIMTS are lightweight, contributing only around 32.4k parameters. The number of trainable parameters is acceptable in real-world application and lower than other vision foundation model based methods like ViTST.\n\n**Time Complexity**: VIMTS achieves efficient training (87.8s/epoch) while its inference speed (4.815ms/instance) outperforms ViTST (visual foundation model - 7.094ms/instance), CRU (Neural-ODE - 7.655ms/instance), and WrapFormer (Transformer - 5.475ms/instance), making it practical for real-world deployment. Though it's slightly slower than lightweight models like tPatchGNN (1.604ms), this trade-off is justified by superior accuracy and improved data efficiency from pre-training—critical in medical applications where sub-5ms latency remains acceptable.\n\n## Q4. Revisions for better readability\n\nAs for Figure 2, we will revise it by selecting representative baseline methods, adjusting the scale and using a clearer color scheme to highlights the model's superior performance and few-shot ability.\n\nAs for Figure 3, we will add more intuitive and descriptive text to the legends in the figure to enhance the readability.\n\nAs for the Contribution Section, we will streamline it in revision to focus on our core insights and make it more concise.\n\n## Reference\n\n[1] Time series as images: Vision transformer for irregularly sampled time series, NIPS23\n\n[2] Visionts: Visual masked autoencoders are free-lunch zero-shot time series forecasters, arXiv:2408.17253.", "claims_and_evidence": "The claims in line 23 (left) and lines 29-30 (right) on page 1 stating that existing pre-training models are limited to UTS are inaccurate, as many existing studies have already considered multivariate time series.", "methods_and_evaluation_criteria": "The model designed in the paper is both reasonable and effective for addressing the IMTS modeling problem. The dataset used is a commonly employed real-world dataset in IMTS modeling, which also provides practical guidance for solving real-world problems.", "theoretical_claims": "There are no theoretical claims in this paper.", "experimental_designs_or_analyses": "The experiments in the paper are sufficient, with appropriate comparison methods and thorough analysis. The use of commonly employed real-world datasets enhances the credibility of the results.", "relation_to_broader_scientific_literature": "This paper details efforts to advance IMTS forecasting. The paper provides a new perspective on the forecasting problem of IMTS to some extent and effectively improves forecasting performance.", "essential_references_not_discussed": "Li Z, Li S, Yan X. Time series as images: Vision transformer for irregularly sampled time series[J]. Advances in Neural Information Processing Systems, 2023, 36: 49187-49204.", "other_strengths_and_weaknesses": "1. The authors should conduct a more in-depth comparison and analysis between this work and **ViTST [1]**. If the authors choose not to include this comparison, the difference in tasks should not be used as an excuse, as both models first learn representations and then use a projection head for downstream tasks.\n2. The contributions section is overly lengthy.\n3. Since the model incorporates multiple GCN layers, it would be helpful for the authors to compare the **time and space complexity** of the proposed method with other SOTA models.\n\n[1] Li Z, Li S, Yan X. Time series as images: Vision transformer for irregularly sampled time series[J]. Advances in Neural Information Processing Systems, 2023, 36: 49187-49204.", "other_comments_or_suggestions": "1. **Figure 2** contains too many elements, making the focus unclear. It would be beneficial to simplify the figure by selecting key components that highlight the advantages of the proposed method.\n2. In **Figure 3**, the legend is only represented by letters, which is not sufficiently clear. It is recommended to add descriptive text, such as \"Channel Mask\", to improve clarity.", "questions_for_authors": "See Other Strengths And Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ecuT4xc9Xe", "reviewer": "Reviewer_rRas", "summary": "This paper addresses forecasting on Irregular Multivariate Time Series (IMTS), where observation intervals are variable and there are missing values. The authors propose to use Masked AutoEncoder (MAE) to efficiently handle missing values in IMTS and scale pre-training/fine-tuning. They also propose to use a Graph Convolutional Neural Network (GCNN) for handling a variable number of channels in different time series, temporal period embedding for handling timestamps as a variable in the MAE framework, and the coarse-to-fine strategy for focusing on relevant temporal context in the forecasting of the specific timestamp. Experimental results on multiple datasets demonstrated that the proposed method performs better than the existing methods.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback and recognition of our work’s **novelty and practical impact**, addressing each point below for clarity.\n## Overview of Method\nVIMTS processes IMTS by: 1) segmenting into fixed intervals with variable-length points per channel; 2) extracting unified patch representations via dynamic convolution; 3) compensating missing values via GCN capturing inter-channel dependencies; 4) modeling temporal patterns across P×N patches using MAE; 5) pre-training with masked patch reconstruction to transfer MAE’s sparse processing, then fine-tuning; 6) coarse-to-fine decoding (patch reconstruction → timestamp queries via continuous embeddings). This vision-inspired pipeline uniquely adapts to irregularity-aware forecasting. Notations will be clarified in revision.\n## Methodological Clarifications\n### Q1: Justification of Channel Embeddings (Eq.6)\nLearnable channel embeddings $e_n$ model channel-specific traits (units, stats, missing patterns), inspired by positional and cross-modal retrieval’s modality embeddings [3]. They distinguish heterogeneous channels (e.g., physiological vs. environmental) during cross-channel compensation and temporal modeling.\n### Q2: GCN Usage in Cross-Channel Interaction (Sec 2.3.3)\nOur GCN addresses sparse/misaligned observations by modeling bidirectional channel dependencies. We fuse static embeddings (inflow/outflow nodes) with dynamic patch features via gated fusion: $g_{p,k} = \\text{ReLU}\\left(\\tanh\\left([H_p \\parallel E_k^s]W_k^g\\right)\\right),$to obtain hybrid embeddings $E_{p,k}$. The adaptive adjacency matrix $A_p = \\text{Softmax}(\\text{ReLU}(E_{p,1}E_{p,2}^\\top))$ dynamically captures directional relationships, enabling context-aware message passing. Ablations (Tab 3) confirm GCN’s necessity in compensating missing values through cross-channel interactions, aligning with evolving graph structures in spatio-temporal GNNs [1,2].\n### Q3: Time Period Embeddings (Eq.11-15)\nOur TPE adapts ViT’s 2D positional encoding [4] for temporal data: horizontal axis encodes patch order $p$, vertical axis fixes duration $s$. Initialized with sinusoidal functions, it preserves MAE’s positional awareness while adapting to IMTS patterns during fine-tuning. This dual-axis design jointly maintains sequential order and temporal scale information.\n### Q4: MAE Reconstruction Mechanism (Sec.2.4.2)\nMAE reconstruction masks $r%$ patches (e.g., 60% for PhysioNet) to simulate missing segments. The model infers missing regions via neighboring patches and cross-channel dependencies. The decoder fuses visible tokens $z_p$ and mask tokens $[M]$ (projected via $W_{dec}$) with TPE ensuring temporal coherence, extending MAE’s sparse processing [6] to irregular series through patch-level masking.\n### Q5: Patch-to-Point Prediction (Sec.2.5)\nOur coarse-to-fine strategy bridges patch-level semantics and timestamp specific queries: (1) MAE decoder reconstructs dense patch representation $ẑ_{i_q}^m$ aggregating temporal patterns and cross-channel contexts. (2) Continuous-time embedding $\\phi(t_q)$ injects fine-grained temporal context. (3) An MLP fuses $ẑ_{i_q}^m$ and $\\phi(t_q)$ for final prediction $\\hat{x}_q$. This hierarchical approach mirrors image super-resolution (low-resolution features guide high-detail reconstruction) adapted for temporal irregularity via timestamp embeddings.\n## Technical Corrections \n- Notation\n  - $L$: total timestamps; $N$: channels; $q_j^n$: channel-specific queries;  unified $\\mathcal{O} = (\\mathcal{T}, \\mathcal{X}, \\mathcal{M})$\n  -  $t_\\text{start}^p$: patch. start; $l_p/r_p$: first/last timestamp index\n  - \"$\\|$\": feature concatenation\n- Equation Fixes\n  - Eq.5 : $m_p → m_p^n$ (channel-specific mask)\n  - Eq.20: Dimension $\\mathbb{R}^{P \\times 2D}$\n  - Losses (Eqs.26-27): Normalized by ($|\\mathcal{Q}|$) \n- Figs/Tabs  \n  - Fig 1 : Added contrast with prior work\n  - Fig 2: Adjusted DLinear/VisionTS scales  \n  - Tab 1 & 2: Order corrected.\n## Reviewer-Specific Queries\n### Q6 (Eq.167)\nFixed as $\\mathbf{H}'_p = [\\mathbf{H}_p \\| \\mathbf{H}_p^\\text{gcn}] \\in \\mathbb{R}^{N \\times 2D}$.  \n### Q7 (Eq.3)\nMeta-filter = MLP generating adaptive convolution kernels [5] for variable-length inputs.  \n### Q8 ($k$ in Eqs.7-9)\n$k \\in \\{1,2\\}$ denotes inflow/outflow embeddings for bidirectional dependencies.  \n### Q9 (Loss)\nLoss averages over queries ($|\\mathcal{Q}|$), not channels, due to timestamp misalignment.  \n\n[1] Graph wavenet for deep spatial-temporal graph modeling. Arxiv.\n\n[2] Irregular multivariate time series forecasting: A transformable patching graph neural networks approach. ICML24.\n\n[3] Cross-modality transformer for visible-infrared person re-identification. ECCV22.\n\n[4] An image is worth 16x16 words: Transformers for image recognition at scale. Arxiv.\n\n[5] Irregular traffic time series forecasting based on asynchronous spatio-temporal graph convolutional networks. KDD24.\n\n[6] Masked autoencoders are scalable vision learners. CVPR22.", "claims_and_evidence": "* Eq.6 is not justified well. It should be discussed in literature or theoretically.\n* The usage of GCN in Section 2.3.3. is not justified well. It should be discussed in literature or theoretically.\n* Eqs.11-15 for the time period embeddings are not justified well. It should be discussed in literature or theoretically.\n* Similar comments to the above for Sections 2.4.2. and 2.5.", "methods_and_evaluation_criteria": "Make sense.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "I checked Section 3 and the appendices.", "relation_to_broader_scientific_literature": "The usage of MAE for time series forecasting with missing values is novel.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "The proposed method may have novel points and have some practical impact, but the clarity issues are too severe to understand the method.\n* We tend to use \"forecasting in this problem setting rather than \"prediction.\"\n* Figure 1 is not referred to in the main text.\n* Figure 2 is hard to understand. Where are the results of DLinear and VisionTS?\n* In Section 2.2, L, N, and q^n_j are not defined.\n* Paragraphs \"IMTS Representation\" and \"IMTS Prediction\" look to share some portion of notations, but their relationship and dependencies are not explicitly described, which makes it hard to follow Section 2.2.\n* The relationship between t_start^p, l_p, and r_p is unclear.\n* In Eqs.2, 5, and 7, \"||\" might be some operator, but it is not defined.\n* In Eq.3 and the description under Eq.3, L_p and D_in are not defined (L_p can be s, maybe?). The character d may be used in different meanings from Eq.1. And, what is the meta-filter? \n* In Eq.4, the superscript \"*\" is not defined.\n* In Eq.4, can the subscript l_p:r_p be just i and remove [i]?\n* In Eq.5, the variable m is used inconsistently with subscript and without subscript.\n* Eqs.7-9 are hard to understand. What is the superscript s? Why k is in {1,2}\n* The equation in l.167 of the right-hand side of p.4 is incorrect. \n* In the equation in l.180 of the right-hand side of p.4, R^2D should be R^Px2D. \n* Eqs.11-15 are hard to follow.\n* In Eq.20, TPE does not require the projection into the input dimension of the decoder?\n* In Eq.21, the variable e is used inconsistently from the previous formulation.\n* The losses in Eqs.26 and 27, operation (summation or averaging) over n lacks.\n* In l.312 of the left-hand side of p.6, it is better to use consistent notations with the main text (x should be used, not y).\n* Tables 1 and 2 appear in an inverted order.", "other_comments_or_suggestions": "NA", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces VIMTS, a novel framework that adapts visual Masked Autoencoders (MAE) for the prediction of Irregular Multivariate Time Series (IMTS). Extensive experiments highlight VIMTS’s superior performance and robust few-shot capabilities, showcasing its potential to extend visual foundation models to broader time series prediction tasks.

The paper received four reviews. Following the rebuttal and subsequent discussion, three reviewers reached a consensus to accept the paper, citing its innovative contributions, strong empirical performance, and potential broader impact. The fourth reviewer, who issued a weak reject, raised concerns primarily related to the presentation rather than the core methodology. After reviewing the authors' rebuttal, the AC finds that the concerns about presentation have been adequately addressed.

Overall, the AC believes the paper presents valuable contributions and recommends it for acceptance. To further enhance its quality, the AC encourages the authors to revise the manuscript by incorporating all reviewer suggestions—particularly by including the additional experimental results presented during the rebuttal phase and improving the overall presentation.
Decision: Accept (poster)