S4M: S4 for multivariate time series forecasting with Missing values
Authors: Peng Jing
Meiqi Yang
Qiong Zhang
Xiaoxiao Li
Venue: ICLR 2025 Poster (2025)
Keywords: S4 Models
Multivariate Time Series Forecasting
Missing Value
Prototype Bank
Primary area: learning on time series and dynamical systems

Abstract: Multivariate time series data play a pivotal role in a wide range of real-world applications, such as finance, healthcare, and meteorology, where accurate forecasting is critical for informed decision-making and proactive interventions. However, the presence of block missing data introduces significant challenges, often compromising the performance of predictive models. Traditional two-step approaches, which first impute missing values and then perform forecasting, are prone to error accumulation, particularly in complex multivariate settings characterized by high missing ratios and intricate dependency structures. In this work, we introduce S4M, an end-to-end time series forecasting framework that seamlessly integrates missing data handling into the Structured State Space Sequence (S4) model architecture. Unlike conventional methods that treat imputation as a separate preprocessing step, S4M leverages the latent space of S4 models to directly recognize and represent missing data patterns, thereby more effectively capturing the underlying temporal and multivariate dependencies. Our framework comprises two key components: the Adaptive Temporal Prototype Mapper (ATPM) and the Missing-Aware Dual Stream S4 (MDS-S4). The ATPM employs a prototype bank to derive robust and informative representations from historical data patterns, while the MDS-S4 processes these representations alongside missingness masks as dual input streams to enable accurate forecasting. Through extensive empirical evaluations on diverse real-world datasets, we demonstrate that S4M consistently achieves state-of-the-art performance. These results underscore the efficacy of our integrated approach in handling missing data, showcasing its robustness and superiority over traditional imputation-based methods. Our findings highlight the potential of S4M to advance reliable time series forecasting in practical applications, offering a promising direction for future research and deployment. Code is available at https://github.com/WINTERWEEL/S4M.git.

Published as a conference paper at ICLR 2025
S4M: S4 FOR MULTIVARIATE TIME SERIES FORECAST- ING WITH MISSING VALUES
Jing Peng'
Meiqi Yang²
Qiong Zhang!*
Xíaoxiao Li
RRemmin University of China Princeton University The University of British Columbia Vector Institute {ipeng, qiong. zhang] xiaoxiao. li@ece. ubc. ca
B ruu. edu. en,
meigiv@ppriccttnnedd
ABSTRACT
Multivariate time series data play a pivotal role in a wide range of real-world appli- cations, such as finance, healthcare, and meteorology, where accurate forecasting is critical informed decision-making and proactive interventions. However, the presence of block missing data introduces significant challenges, often com- promising the performance predictive models. Traditional two-step approaches, which first impute missing values and then perform forecasting, prone to er- ror accumulation, particularly in complex multivariate settings characterized by high missing ratios and intricate dependency structures. In this work, we in- troduce S4M, an end-to-end time series forecasting framework that seamlessly integrates missing data handling into the Structured State Space Sequence (S4) model architecture. Unlike conventional methods that treat imputation arate preprocessing step, S S4M leverages the latent space of S4 models to directly recognize and represent missing data patterns, thereby more effectively captur- ing the underlying temporal and multivariate dependencies. Our framework com- prises two key components: the Adaptive Temporal Prototype Mapper (ATPM) and the Missing- Aware Dual Stream S4 (MDS- S4). The ATPM employs a prototype bank to derive robust and informative representations from historical data patterns, while the MDS-S4 processes these representations alongside missingness masks dual input streams to enable accurate forecasting. Through extensive empirical evaluations on diverse real-world datasets, we demonstrate that S4M consistently achieves state-of-the-an performance. These results underscore the efficacy of our integrated approach in handling missing data, showcasing its robustness and superiority over traditional imputation-basee methods. Our findings highlight the potential of S4M to advance reliable time series forecasting in practical applica- tions, offering a promising direction for future research and deployment. Code is available at https: / /gtttub....u.ggg com/ //////INNTEEEELLLE /0..//4...0. git.
sep-
INTRODUCTION
Multivariate time series are common in real-world applications, including finance (Zhang et al., 2024), health care (Kaushik et al., 2020), and meteorology (Duchon & Hale, 2012). Time series forecasting (Box et al., 2015) predicts future values based historical data. Accurate forecasting enables informed decision making and helps anticipate trends and take proactive measures, from op- timizing financial investments to improving patient care and responding to environmental changes.
Time series forecasting has been a long-standing area of research, with numerous methods devel- oped over the years. Traditional statistical methods typically build on linear assumptions and autore- gressive models to capture temporal dependency, such as ARIMA (Box & Jenkins, 1968), failing forecast well in complex multivariate time series. Recent machine learning advancements have introduced promising solutions, including RNN-based methods (Salinas et al., 2017: Rangapuram al., 2018: Lim et al., 2020; Hewamalage al., 2021) that capture long-term dependencies and attention- didention-bbsed models (Qin et al., 2017; Shih et al., 2019; Wu et al., 2021; 1; Liu et al., 2022; Shabani
"Correspondence Qiong Zhang giong.zhangee www.uuu.......ccc.n.

---

Published as a conference paper at ICLR 2025
et al., 2023; Nie et al., 2022; Liu et al., 2023; Xue et al., 2023) that leverage temporal attention mechanisms. A more recent and influential technique is the Structured State Space Sequence (S4) model (Gu et al., 2021), which combines the strengths of State-sacce models with modern deep learning architectures to efficiently model long sequences. This study highlights the strong suitabil- ity of S4 models for time-series forecasting, driven by their ability to address the growing demand for efficiency in large-scale applications where computational resources and scalability are paramount.
A significant challenge in time series forecasting is the effective handling of missing data, which often arises due to sensor failures, data collection issues, or external disruptions. Missing values can severely degrade model performance if not properly addressed. For instance, in healthcare, gaps in wearable device data may occur due to inconsistent usage (Darji et al., 2023): in financial transac- tions, data might be incomplete owing to network outages or system downtimes (Emmanuel et al., 2021); in environmental monitoring, sensor networks measuring air or water quality frequently face data loss due to device malfunctions or harsh weather scenarios (Zhang Thorburn, 2022). In these applications, the data may exhibit block missing patterns, where missing values occur consecutively rather than randomly (see Fig. 5). These gaps not only reduce the amount of available data but can also introduce biases, leading inaccurate forecasts.
Traditional approaches to handling missing data in time series typically involve a two-step pro- cess: imputing missing values and then performing forecasting on the imputed data (Cao et al., 2018; Cini et al., 2021; Marisca et al., 2022). However, in multivariate time series, the complex- ity of missing data patterns and high missing ratios make direct imputation challenging. This two-step approach often leads to error accumulation, resulting in suboptimal forecasting per- formance. Consequently, there is a growing need for end-to-end methods that integrate miss- ing data handling directly into the forecasting process. Existing methods for handling miss- ing data have notable limitations. RNN-based methods like GRU-D (Che et al., 2018) and BRITS (Cao et al., 2018) address missing data but often require long training and perform poorly. Graph models like BiTGraph (Chen et al., 2023) capture dependencies at high mem- ory cost, while ODE-based methods like Neural ODE (Chen et al., 2018), GraFITi (Yalavarthi et al., 2024) and CRUs (Schirmer et al., 2022) are computationally expensive. Fig. presents a comprehensive comparison of forecasting accuracy, training time, and memory usage across the methods. The results highlight that S4-based methods not only achieve superior performance but also offer significantly lower computational costs compared to alternative approaches. This efficiency underpins our decision focus on S4 in this work. A more detailed review of related work is provided in Appendix A.
Training Time vs Errors (with Memory Footprint)
0.54
BRITS 558 MB, 487
0.52
0.50
Transformer
To address these challenges, we propose an end-to- end method in this work that both computationally and memory efficient while maintaining robust fore- casting performance under block missing data. We build on the S4 model due to its demonstrated suc- cess in time series forecasting (Wang et al., 2024) and its ability to handle multiple inputs concurrently. This capability allows us to address missing data while simultancously learning the complex depen- dency structures inherent in the forecasting task. By integrating missing data handling directly into the S4 framework, we aim to fully leverage its strengths for multivariate time series forecasting.
778 MB, 78
Erors (MSE)
0.48
CARD 75,818,, 274
Autoformer
0.46
1105
96
GRU-D 446 MB, 244
0.44
iTransformer 2108 MB, 74
CRUs
0.42
32146 MB, 64990
$4
765
MB, 85
BITGraph 2564 MB, 268
0.38
SAM(Ours) 1071 MB, 110
10a
10a
10
Training Time (s/epoch))
Comparison of prediction MSE versus training time for various methods on the Electricity dataset. Each method is rep- resented by a dot, with size scaled accord- ing to its memory footprint. Lower values for MSE, training time, and memory indi- cate better performance. Our S4M method demonstrates superior performance across all metrics.
Figure 1:
Our method termed S4 with missing values (S4M) that explicitly considers missing values in the S4 model consists of two modules: adaptive temporal prototype mapper (ATPM) and missing-aware dual stream S4 (MDS-S4). The ATPM module is designed to use rich historical data patterns stored in a proto- type bank to learn robust and informative represen- tations of the time sequence. These representations, along with mask indicating missing values, are then processed by the MDS- module, which
2

---

Published as a conference paper at ICLR 2025
performs forecasting using dual-stream S4 architecture. As shown in Fig. 1, S4M demonstrates superior efficiency in both training and inference compared to existing baselines. We conduct exten- sive experiments real-world datasets, comparing our method with ofate-of-the-arr approaches and their variants. The results demonstrate that S4M consistently achieves top-tier performance across various settings, highlighting its robustness in handling missing data. Our work addresses the crit- ical challenge of missing data in time series forecasting, offering a scalable, efficient, and robust solution for real-world applications.
PRELIMINARY
The S4 model, introduced by Gu al. (2021), is a pioneering sequence model designed handle continuous- time data with long-range dependencies, making highly effective for tasks like time series forecasting. For completeness, we provide brief overview of S4.
Let (t),, E RP be two D-variate continuous signals. The continuous state space model (SSM) maps utt) to (t) via the following equations:
d h(t) = Ah(t) + Bu(t), y(t) = c + Du(t),
(1)
dt
where (1)) E I is an unobserved hidden state, and the system is parameterized by matrices A E RHXH B E RHDD C IR H and D HD Since real-world data is typically observed discrete time points t = 0, 1, T. the continuous model in equation can be discretized as:
h, = Ah:-1 Bus y: = Ch + Du,
based on bilinear trans- form (Gu et al., 2021) with some parameter A. By recursively applying the recurrent representation of SSM in equation 2 model over discrete time, the output yı at time t computed as a convolution of all previous inputs Noz:
where A= (I- 21-AA/21 A-/2) AA/2) and B = (I A-/2)-A/BBBBB
i y: = Σ CĀ TA 1=0
Bu, + Du.
For an input sequence u = (uo; Wi..... WT). one can observe that the output sequence y = (yo, y/....., yr) can be computed using convolution with skip connection y CK + Du, where * is the convolution operation and K = (B. A" B) is called the SSM kernel. One key challenge of Eiscrete-time SSMs is that computing the output involves repeated matrix mul- tiplications by A, which can be expensive, with computational cost of O (H²T) when implemented naively. S4 addresses two main challenges compared to basic SSMs. First, it solves the long-range dependencies modeling challenge by employing the HiPPO matrix (Gu al., 2020) for A, enabling continuous-ting memorization. Second, S4 solves the computational bottleneck by introducing specialized representation and algorithm that significantly reduces the computational cost.
PROPOSED METHOD
3.1 PROBLEM FORMULATION
We denote (1) and X (H) the look-back and horizon windows for the forecast, respectively. of corresponding lengths l, and ε.-- Given a starting time lo, they are denoted as X(L) = (a, E D : E to la + f and ( = (æ, : to + l, + 1 to + + }. We consider the case where there exist missing values in the observations due to the failure of devices or some other unexpected emrors. We use mask matrix (L) R to denote whether the value is missing or not. Specifically, the (t, d'-th element the mask matrix is binary and is given by
x=(() is observed, M - 0. otherwise.
[1. 1,
The goal of forecasting predict the horizon window X given the look-back window ( Thus, time series forecasting be framed learning mapping from (L) X (H).
3

---

Published as a conference paper at ICLR 2025
We design an approach to learn f that is parameterized by 0 in the presence of missing data. During training, let [(X(L) M () be the predicted values for the horizon window, then the parameter @ is learned by minimizing the error between the true horizon window (H) and its predicted value. Note that the input and output of f have the same length, for the foresting task where tи S Lr. we slice the last Eœ as the predicted value.
Adaptive Te Temporal Prototype Mapper (ATPM)
Missingg aware Dual Stream S4 (MDS-S4)
2 tst
Oi
h
Ro F.m,
Sankreadin
Looo Uie
y)
Ch. Do. Frn.
me
Depuer bleck
sddrection
Rigudant
RegderS4 Elass
Figure 2: Illustration of our end-to-end prediction method S4M. Our method consists of two modules. The first ATPM module uses historical data patterns to learn robust and informative rep- resentations for the current input time sequence. Specifically, we extract the local statistics 2t-sat of the time series at time point t based on raw values 2t- These statistics are then fed into the query encoder E, obtain q: which queries the prototype bank retrieve the prototype q.. Both qi andğ; are subsequently fed into linear layer to produce the final representation on. Addition- ally, the prototype encoder E, generates the prototype Pe for bank updating. In the second module MDS-S4, we model the representation o, and the mask m, using S4 to generate the forecast y::-
The pipeline of our proposed S4 with missing values (S4M) is given in Fig. 2. consists of two modules specifically designed to deal with missing values in an end-to-end man- ner. The first ATPM module focuses on representation learning with missing values, it contains prototype bank, which stores a rich set of representations of historical data in the time series, from which we can query the representation of missing values based on their local features. The second MDS- BDS-S4 $4 module directly models the missing patterns in the SSM. Our design explicitly considers missing values in the model, and the model also progressively updates the missing patterns.
Method Overview:
3.2 ADAPTIVE TEMPORAL PROTOTYPE MAPPER (ATPM)
3.2.1
OVERVIEW OF ATPM
To address missing values, we leverage a prototype bank that stores a rich set of representative pat- terns from time series. The goal is to utilize historical data patterns learn robust and informative representations for the current input time sequence. Since the raw time series input is multivari- ate and can be noisy, often containing missing values, rather than querying and storing prototypes using the raw time series data, we design encoders to extract more robust latent representations, allowing us to query and store the prototypes in the representation space. As the prototypes in the bank evolve and are adaptive to the data during training, we call this module the adaptive temporal prototype mapper (ATPM).
Specifically, recall x, E TRP is the value of the look-back window (L) time ATPM first extracts local statistics 2; at each time point t (such as its first previous non-missing value and the time difference to the first non- non-missing time point) based on the look-back window X(1) We denote this local statistics extraction as 2, fa.) and its details are given in Appendix C.I.
At the t-th time point, our hypothesis is that local statistics of single time point is insufficient to infer patterns when corresponds to a missing observation. To mitigate this, we look back over a short period of length to assist with inference at the missing time point, constructing matrix zt-s:t- = {zı | E t s t). This local statistics sequence zt-sit is then used to query and update the prototype bank in the representation space by feeding it into a query encoder E, with parameter 0, to obtain the query representation, which is used to query the prototype bank, and a prototype encoder Ep with parameter 6, to obtain the prototype representation, which is used to update the prototype bank. After querying the prototype bank, we combine the retrieved prototype and other local statistics to obtain the final representation o,, which is detailed below.
4

---

Published as a conference paper at ICLR 2025
3.2.2 DESIGN OF THE PROTOTYPE BANK
The core concept of the prototype bank is to read (query) similar representations from rich historical data stored the bank. These representations are then used input for the subsequent module. At the same time, the representations are also used to write (update) the bank adaptively. We describe the structure of the bank and how to read and write the bank below.
Bank Storage. Prototypes are organized in two-level queue. The first level represents different clusters, with each element serving as the centroid of a cluster of prototypes. Within each cluster, the second-level queue stores the corresponding prototypes that belong that cluster. To ensure efficient storage, inference, and stability, the first-level queue can hold a maximum of K₁ centroids, while each second-leves queue can accommodate up K2 prototypes per cluster. The prototype bank is (FIFO) principle, allowing
designed as a queue to facilitate updates following the First-In-First-On outdated prototypes that no longer align with the updated encoder to be filtered out efficiently. The prototype bank is initialized at its first level by applying k-means clustering on the output of the encoder of the first batch.
Bank Reading, Denote q: = (z- 02) be the query encoder that has local temporal and spatial information. We then use qi to query the prototype bank to retrieve the most similar patterns and use their weighted average as the prototype vector at the time point L. In cases where / is a missing value time point, the retrieved prototypes help account for the missing values. Specifically, let (cI, C2...., represent the cluster centroids stored in the first-level queue, and let qe be the query feature. We compute their cosine similarity as Ptj = q, c ||l|e|ll |l. Let S, (ji, (j......Jaa jx where Pt.j 2 Pt.ja 2- 2 Pt.jx. 2 be the index of the top K maximum similarities and normalize them as Wtj = exp(pts)/ L'' exp(ptr)) for j E St. These retrieved prototypes are then aggregated as: q. = Ejes. WtzCj. Chandar et al. (2016) observed that selecting the top K similar centroids, rather than using all centroids, can improve performance. Finally, we combine Zt-s:t qt and q: using a dense layer to form a single representation O,.
Bank Writing. After querying the proto- type bank, we also update it using the out- put from Pt = (x 0,) be the output of E.p. We compute the cosine similarity be- tween this representation and the prototype cen- troids to assess their closeness. If the cur- rent patterns are very similar to existing pro- totypes, we add them to the level two queue; otherwise, we add the prototype to the level one queue as a new cluster. Specifically, let wi = max, p._ e//l/px st///x:llle,) ||||c represent the sim- ilarity value of the current representation to ex- isting prototype centroids. If w; 2 Ti for some predefined syper-parameter Tı, then Pr is added to the queue of the cluster with which it shares
Algorithm Bank Reading
Input: local statistics {z ) query encoder Eg. bank prototype centroids {c1, c J. ini- tial values for parameter W and d
Output: target representation O
1: for z, in Z = { 22, .... Ze, do
22,,..,
2: Encoding: q: E, (2,,s)) s:t)
3: Similarity: Ptj = q: cj. 11 ||,]|
4: Normalization for top- maximum val-
ues: Wyj = exp(pt)) / jes, exp(pts)
5:
Aggregate prototype: 4:- Ejes, wisej
6:
Combine: e. W [21, qi q:) + d
7:
Output: 0, q: + Ws
the highest degree of similarity. If We < T2 for some predefined syper-parameter 72, indi- cating insufficient similarity with any existing centroid, P: is introduced as novel pattern the bank and also serves as the initialization of its prototypes cluster'. In both cases, the centroids are updated accordingly. In the case where Ti S Wi S T2. the prototype is not used for updating the bank. This process ensures that the prototype bank remains dynamic and capable of capturing a diverse range of patterns.
8:
end for
9: Final Output O = (01, or..
3.2.3 ENCODER UPDATE
Recall that the prototype Pt = Ez 6p) and the query feature q: = Eg 0,y) are the outputs of two distinct encoders, E., and Ep,. parameterized by O, and 0,, respectively. The archi- tecture of the encoders are given in Appendix C.2. Although both encoders take the same input, they serve different purposes: the prototype encoder E, is designed store rich set of time series
We set 71 0.9 and F2 0.6 in experiment.
5

---

Published as a conference paper at ICLR 2025
representations, while the query encoder Eg aims to obtain a representation that diverges from the prototypes. Thus, these encoders must not be identical and should be updated differ- ently. To ensure that the prototypes evolve more stably, we use momentum update for the prototype encoder Ep, while the query en- coder updates its parameters in traditional manner. Specifically, the parameter 6g the query encoder is updated using gradient de- scent based on the final loss, whereas the pa- rameter Op of the prototype encoder is updated
Algorithm Bank Writing
Input: local statistics totype encoder Ep, bank prototype centroids {c1, C2....,}
{ pro-
Output: bank with updated prototypes
Random sample n slices {zi) from Z
2: for z; in { 22, 2n do
3:
Encoding: p: = Ex(
4: Similarity: Pi: P. c ht:/|palll p
5: Maximum index: j = arg max, Pt.
6: if Piy. 2 Ti then
with momentum- maased approach, allowing for smoother updates as suggested by He et al. (2020). During the prototype bank writing pro- cess, the gradients of Op are disabled, and the parameters are updated via momentum:
7:
Add p; to the end of the j second-
level queue
8:
Update j"th prototype centriod
9:
else if Pij' < T2 then
10:
Add p: to the end of the first-level
0p = ,0, (1-7)0 7)
(3)
queue
11: else continue
where 7 E 1) is the momentum coefficient. The momentum update equation 3 makes 0, evolves more smoothly than 0g.
12: end
end for
3.3 MISSING-AWARE DUAL STREAM S4 (MDS-S4)
Drawing inspiration from the GRU-D model in (Che et al., 2018), we explicitly model the missing values by including the mask (L) in the SSM. Intuitively, with the presence of missing values, both the hidden state h. and the output of S4 depend on the mask vector m.. We therefore modify the SSM so that has two input streams: the representation and the mask. Specifically, let o, be the output from the representation learning module, and m. Yt be the ith row of (L) and (H), Our missing-awas aware dual stream SSM is:
h Ah + Bo, + Em (m; θm)
(4)
y/ Ch, Do, + FE, (m,; 0m),
where and B are the same as in equation 2 and E = (I AA/2) The encoder E., param- eterized by on, is used to ensure that also the latent representation of the mask fully utilize its information. Denote o = (oto Oto+éz ). m = (M..,.. +)o+E2) y (Yto...., Vto++E ) Given the initial hidden state, the dual stream SSM in equation
be recursively unrolled get
the following explicit convolution operation:
Therefore, our modified SSM model for missing data has an additive structure of the the same trick in $4 to efficiently calculate the y CK₁ + CK2 E., (m; + Do + FE (m, 0,.)
where K; (B, AB,
A 1 B) and K2 EE AE A t E)
A are two SSM kernels.
SSM model in equation 2. We
convolution operation and end with adding two outputs from the convolution operations. The convolution operation, together with the HiPPO matrix A. enables S4 effectively model long- term dependencies. Similarly, our dual-stream SSM incorporates convolution operation and the HiPPO matrix, preserving S4's compu- tational efficiency and capacity for modeling long-term dependencies, while simultaneously addressing missing information through dis- tinct computational kernels. Given the output from MDS-S4. S4, we can further feed into ei- ther MDS-S4 regular S4 blocks increase
Algorithm 3 Testing Pipeline
Input: Look-back window X²'),, learned proto- type bank centroids {cy}- }. query encoder Eg, learned MDS-S4 module and local statistics ex- tractor Siecal
Output: Forecasted value Y
I: Local Feature Extraction: Z flocal (X)
2: Bank Reading: O Alg. 1 (Z2
17666 C, F.g))
3: MDS-S4 Output: y -MDS $4(O)

---

Published as a conference paper at ICLR 2025
the complexity of our model. We describe the specific strueture of the encoder Em and multiple S4 blocks in Appendix C.3. Our full algorithm for training and testing is, respectively, given in Alg. and Alg. 3.
EXPERIMENTS
4.1 DATASETS AND EXPERIMENT SETUP
We select four commonly used time series datasets for forecasting: Electricity (Wu et al., 2021), ETTh1 (Zhou et al., 2021), Traffic (Wu et al., 2021), and Weather (Wu et al., 2021). complete,
Algorithm 4 Training Pipeline
Input: Batches of look-back window (x and corresponding masks {M:) initial values for model parameters
Since these benchmark datasets we manually created block missing on the train- ing and test dataset. These datasets span vari- ous domains and encompass diverse character- isties terms of magnitude ranges, sampling frequencies, and statistical properties like sea- sonality, The base statistics of the data set can be found in Tab. 7. To model practical sce- narios where sensors cannot record data for period due to failure other reasons, we de- sign block-based missing pattern for two types of missing data scenarios: time point missing and variable missing with missing rate 0.03, 0.06, 0.12, 0.14. The details of making missing pattern can be found in Appendix D.2. After obtaining the dataset with missing values, we split it chronologically into training. valida- tion, and test sets, with ratio 0.7/0.1/0.2 The horizon window for all methods is fixed at 96, while the lookback length is varied across 96, 192, 384, and 768. In addition to these benchmark datasets, we also conduct experiments on a real-world dataset, as detailed in Ap- pendix D.8.
Output: Prediction { I: Initialize:
prototype centroids C based on K-means from
[CI,.... CK]
2: for = 2 B
3:
Local Feature Extraction: Z.
f (X;:
4:
Bank Reading: O, Alg. 1(Z,, C, Eg)
5:
Bank Writing (No grad):
Alg. 2,,,, Ep)
6:
(No Gradient) Momentum
Update:
0 y0, (1-11)1 7)
7:
Backbone
Output:
Y
(4(O;, M,)
8:
Loss construction & backpropagation:
[ Y X1
9: end for
4.2 COMPETING METHODS
We compare proposed method, S4M, with two main groups of baseline methods: $4-based baselines and other state-of-the-ar and classical methods for handling missing data. The S4-based baseline group includes S4 (Mean), S4 (Ffill), S4 (Decay), and S4 (SAITS). These methods impute missing data using strategies such as global mean, last observation, a decay mechanism based on these statistics, and the superior imputation method SAITS (Du al., 2023). The other methods include classic RNN-based methods like GRUD (Che et al., 2018), LSTM- -bssed methods such as BRITS (Cao et al., 2018), the top-performing Transformer-bast based methods Transformer (Vaswani et al., 2017) and Autoformer (Wu et al., 2021), and the end-to-end method BiTGGrap (Chen et al., 2023), which is specifically designed for missing data prediction.
4.3 COMPARISON WITH BASELINES AND S4-BASED VARIANTS ON TIME POINT MISSING
Varying Input Length. The results in Tab. 1 illustrate the forecasting performance of various methods under time point missing scenarios r 0.06 across the four datasets. Our proposed 54M consistently achieves the best or second-best performance across most settings, demonstrating its robustness in handling missing data. For the Weather dataset, our method exhibits outstanding per- formance, achieving the best MSE in nearly all configurations, particularly at the 192-step length with 0.225, which significantly better than the closest competitor. For the other datasets, S4M maintains strong performance, as no competing methods can consistently outperform it across vari- datasets and settings.
7

---

Published as a conference paper at ICLR 2025
Varying Missing Ratio. Fig. 3 illustrates the performance of various methods under time point missing scenarios across four datasets: Electricity, ETThI, Weather, and Traffic. The methods are evaluated using MAR as the missing ratio (r) increases. Across all datasets, our proposed S4M (de- noted by the red line), consistently maintains lower MAF compared to other methods, particularly as the missing ratio increases. For the Electricity and Weather datasets, 54M outperforms competing methods at all missing ratios, showing a clear advantage in handling missing data. In the ETThl and Traffic datasets, while some other methods like GRU-D or BRITS perform well at lower miss- ing ratios, S 54M still demonstrates robust performance, particularly as r increases, showing strong resilience to higher levels of missing data.
Table 1: Comparison of forecasting performance of S4M (Ours) and baselines on four datasets with various look-back window length under time point missing scenario when missing ratio r 0.06. Entries with '_' indicate the experiment can not be done due to out-of-memory issue.
S4
$4 (SAITS)
S4M
Data
Metrie
BRITS
GRU-D
Trans.
Auto.
BiTGrraph
(Mean)
(Filll
(Decuy)
(Ours)
MAE
0.633
0.431
0.399
0.375
0.397
0.408
0.418
0.402
0.432
0.372
MSE
0.623
0,363
0.400
0.272
0.309
0.337
0.345
0.323
0.372
0.287
Electricity
MAE
0.636
0.437
0.402
0.366
0,388
0.387
0.384
0.381
0,394
0.367
192
MSE
0.628
0.366
0.314
0,257
0.290
0.303
0.292
0.289
0.309
0.274
MAE
0.653
0.434
0.419
0.369
0.384
0,383
0.367
0.379
0.394
0,370
384
MSF
0.659
0.363
0.339
0,272
0.295
0.298
0.272
0.285
0.307
0,277
MAE
0.644
0.437
0.416
0.379
0.387
0.378
0.384
0.379
0.393
0.373
768
MSE
0.656
0,365
0.333
0.285
0,290
0,291
0.288
0.285
0,306
0,282
MAE
0.705
0.644
0.905
0.866
0,571
0.629
0.625
0.614
0.851
0,571
MSE
0.937
0.793
0,942
0,923
0.613
0.747
0.759
0.716
0.914
0.624
ETThi
192
MSE
0.707
0.721
0.805
0.653
0.898
0.938
0.885
0.797
0.609
0.745
0.670
0.681
0.605
0.595
0.666
0.788
0
0.574
0.593
MAE
0.755
0.649
0.968
0.791
0.601
0.595
0.605
0.605
0.719
0.571
384
MSE
1.029
0.798
0.882
0.721
0.662
0.689
0.683
0.840
0.624
MAE
0.788
0.668
1.110
0.797
0.599
0.614
0.614
0.619
0.733
0.588
768
MSE
1.072
0.841
1.041
0.885
0.684
0.697
0.710
0.706
0.848
0.647
MAE
0.419
0.363
0.421
0.465
0.516
0.371
0.361
0.399
0.440
0.313
96
MSE
0.372
0.293
0.350
0.395
0,510
0.312
0.296
0,344
0.407
0.237
MAE
0.427
0.346
0.308
0.471
0.419
0.332
0.318
0.347
0.384
0.305
Weather
192
MSE
0.385
0.268
0.238
0.408
0.385
0,255
0.235
0.274
0.320
0.225
384
MAE
0.434
0.342
0.391
0.479
0.587
0.329
0.345
0.339
0.378
0.306
MSE
0.375
0.271
0.310
0.430
0.596
0.249
0.269
0.264
0.311
0.220
MAE
0.489
0.354
0.374
0.489
0.467
0,330
0.349
0,340
0.368
0.316
MSE
0.445
0,280
0.297
0.459
0.445
0,250
0.272
0.263
0,287
0.232
MAF
0.667
0.467
0.421
0.430
0.516
0.455
0.459
0.451
0.498
0,428
96
MSE
1.158
0.871
0.812
0.919
0.808
0.844
0.794
0.917
MAE
0.667
0.473
0.419
0.410
0.496
0.401
0.398
0.386
0.415
0.385
Traffic
192
MSE
I.170
0.893
0.728
0.721
0.836
0.709
0.692
0.711
0.734
0.687
384
MAE
0.675
0.483
0.452
0.496
0.527
0,400
0.398
0.381
0.412
0.385
MSE
1.193
0.918
0.746
0.817
0.913
0.690
0.682
0.702
0.711
0.702
MAE
0.697
0.490
0.410
0.465
0.394
0.392
0.381
0.407
0.388
768
MSE
1,236
0.947
0.706
0.774
0.687
0.678
0.692
0.7
0.699
BRTTS GRUD
Thersformer Autoformerr
BTCrrphh S41Mear)
5455U/55)
54/Decayl
54M0Dus))
Electricity
ETTh1
Weather
Traffic
0.45
0.8
0.6
0.40
0.7
0.5
0.5
0.35
0.03
0.06
0.32
0.24
0.03
0.05
0.12
0.24
0.03
0.06
0.32
0.24
0.03
0.06
0.12
0.24
Figure 3: The performance of different methods on four datasets under time point missing scenario when the missing ratio r varies from 0.03 to 0.24.
4.4 COMPARISON WITH BASELINES AND S4-BASED VARIANTS ON VARIABLE MISSING
Varying Input Length. Tab. 2 presents the forecasting performance of different methods under variable missing scenarios (r = 0.06) across four datasets. Our method, S S4M consistently achieves either the best second-best results across the majority of configurations, demonstrating its robust-
8

---

Published as a conference paper at ICLR 2025
ness in handling feature-missine data. On the ETThl dataset, S4M shows particularly strong results, securing the lowest MAE and MSE values in several settings. Similarly, for the Weather dataset, S 4M excels, delivering the best MAE and MSE in all configurations. Across the remaining datasets, S 84MM continues to perform competitively, consistently matching or surpassing other methods, high- lighting its general effectiveness in feature-missing scenarios.
Table 2: Comparison of forecasting performance S4M (Ours) and baselines on four datasets with various look-back window length under variable missing scenario when missing ratio 0.06. Entries with '_' indicate the experiment can not be done due to out-of ofmoomooymyy issue.
$4
$4
$4
S4 (SAITS)
S4M
Data
&x. Metric
BRITS
GRU-D
Trans.
Auto.
BiTGraph
(Mean)
(Fill)
(Decay)
(Ours)
MAE
0.439
0.426
0.400
0.373
0,383
0.387
0.387
0.396
0.432
0.369
96
MSE
0.369
0.354
0.312
0.271
0.292
0.305
0.304
0,311
0.354
0.282
Electricity
MAE
0.457
0.477
0.400
0.366
0.376
0.366
0.365
0.378
0.405
0.357
192
MSE
0.390
0.408
0,308
0.257
0,277
0.273
0.272
0.282
0,310
0.261
MAE
0.625
0.470
0.412
0.361
0.389
0.366
0.367
0.377
0,411
0.359
384
MSE
0.619
0.408
0.317
0.255
0.290
0.,270
0.272
0.279
0.317
0.264
MAE
0.635
0.487
0.411
0,363
0.387
0,367
0.376
0.374
0.402
0,362
768
MSE
0.637
0.434
0.326
0.261
0,287
0.272
0.286
0.279
0.309
0.269
MAE
0.696
0.618
0.589
0.583
0,571
0.641
0.642
0.620
0.682
0.571
96
MSE
0,905
0.727
0.658
0.648
0.653
0.761
0,763
0,717
0,851
0.624
MAE
0.820
0.617
0.647
0.583
0.599
0.619
0.619
0.598
0.658
0,568
ET1bl
192
MSC
1.165
0.725
0.817
0.640
0.719
0.687
1.619
0.665
0.788
0.598
MAE
0.821
0.607
0.614
0.585
0.602
0.607
0.606
0.607
0.633
0,584
384
MSE
1.166
0.708
0.683
0.635
0.719
0.665
0.673
0.683
0.719
0.613
MAE
0.820
0.625
0.749
0.641
0.636
0.616
0.623
0.624
0.641
0.599
768
MSE
1.163
0.734
L.029
0.733
0,811
0.676
0.706
0.721
0.733
0.649
MAE
0,408
0,409
0,427
0.498
0.543
0.413
0.394
0,388
0,439
0.336
96
MSE
0.336
0.348
0.357
0.440
0,545
0.364
0.337
0,332
0.392
0.267
Weather
192
MAE
0.417
0,383
0.426
0.507
0.444
0,363
0.352
0.347
0.403
0.320
MSE
0.357
0,311
0.351
0.454
0.418
0.296
0.275
0,275
0.335
0.261
MAE
0.452
0.381
0.405
0.517
0.654
0.359
0.345
0.338
0.405
0.334
384
MSE
0.401
0.314
0.329
0.477
0.698
0.292
0.269
0,265
0.333
0.256
768
MAE
0.470
0,392
0.401
0.529
0.623
0.349
0.349
0.340
0.395
0.341
MSE
0.427
0.323
0.337
0.508
0.663
0.272
0.272
0.263
0.321
0.266
MAE
0.676
0.483
0.428
0.439
0.516
0.443
0.438
0.440
0.504
0.442
96
MSE
1.240
0.905
0.759
0.708
0.907
0.821
0.819
0.812
0.874
0.786
MAE
0.679
0,500
0.411
0,390
0.521
0,383
0.398
0.391
0.447
0.381
Traffic
192
MSE
1.208
0.927
0.705
0.632
0.886
0.707
0.692
0.726
0.776
0.685
MAE
0.678
0.503
0.399
0.393
0.486
0.379
0,420
0,385
0.444
0.383
384
MSE
1.197
0.953
0.696
0.648
0.795
0.702
0.755
0.716
0.772
0.700
MAE
0.441
0.407
0.381
0.375
0.442
768
MSE
1.207
0.967
0.758
0.666
0.704
0.692
0.708
0.775
0.697
Varying Missing Ratio. Fig. displays the performance various methods under variable miss- ing scenarios across the four datasets. As with time point missing, MAE is used as the evaluation metric, plotted against different missing ratios (r). Our method, S S4M (indicated by the red line), con- sistently demonstrates competitive or superior performance across all datasets and missing ratios. In the Electricity dataset, 54M maintains one the lowest MAEs, showing more stability compared to methods like GRU-D, which shows sharp increase in error as the missing ratio grows. Similarly, in the ETThl and Weather datasets, S4M continues to outperform or match the best methods, par- ticularly at higher missing ratios. For the Traffic dataset, while some methods perform comparably at lower missing ratios, 54M demonstrates robust resilience, with relatively low error even as the proportion of missing features increases. Overall, S4M shows strong generalization and consistent performance, effectively handling variable missing data scenarios across multiple datasets.
4.5 ABLATION STUDY
In the previous experiment, we investigated the effects of replacing the data inputs to the S4 back- bone (blue columns in Tab. and Tab. 2). To deepen the analysis, we conducted additional ablations on ATPM and the input stream of mask indications as shown in Tab. 3.
The results demonstrate the importance of incorporating the mask as the inputs to S4 backbone, as removing it consistently increases both MAE and MSE across various prediction horizons. Notably,

---

Published as a conference paper at ICLR 2025
BRTS GRUD
Tranforrer
BitGreph Se/Meani
SAIFGI)
5415NT51
Autoformer
SAlllcay?
SAM(Ours
Electricity
ETTh1
Weather
Traffic
0.5
0.4
0.0
0.4
0.05
0.06
0.12
0.34
0.03
0.06
0.12
0.74
0.03
0.06
0.12
0.24
0.03
0.06
0.12
0.24
Figure 4: The performance of different methods four datasets under variable missing scenario when the missing ratio r varies from 0.03 to 0.24.
even when the eπror increases after removing masks appear numerically small in some entries, the overall predominantly positive red values reflect the model's ' enhanced stability and accuracy when handling missing data. This is particularly evident in the ETT and Weather datasets, where the pres- ence of the mask significantly reduces errors, affirming the effectiveness of dual-inputs in MDS-54 to capture the complex dependencies inherent in multivariate time series with missing values.
The results also highlight the significance of ATPM. The model's performance improved signifi- cantly after incorporating ATPM, as both MSE and MAE increased across various settings when ATPM was removed, particularly on the Traffic and ETThl datasets. Additionally, ATPM demon- strated substantial improvements, especially with shorter lookback windows on the Electricity and Weather datasets, further emphasizing the improvements brought by ATPM.
Table 3: Results ablation study for the mask and ATPM with blue values indicating decrease in errors, while red values representing increase in errors.
Bleomicity
4.
Metic
(wis ATPM)
ATIM)
Visiable
96
MAE
MSE
0.292
0,369
40.010
40.012
40.013
-0.011
0.571
0.624
-0.008
-0.038
+0.091
40.044
0.267
0.336
40.530
40.106
40.206
40.0221
0.786
0.442
+0.059
40.001
+0.125
+0.024
192
MAE
MSE
0.357
0.251
+0.006
+0.005
+0.009
+0.013
0.598
0.568
-0.013
.0.044
+0.090
+0.045
0.261
0.330
10.424
40.061
+0.600
+0.002
0.683
0.381
+0.003
+0.055
+00030
+0.092
384
MAN
MSE
0.359
0.264
+0.002
+0.001
+0.009
-0.009
0.613
0.584
+0.00
+0.033
+0.029
+0.055
0.334
0.256
10.019
+0.414
+0.006
+0.006
0.383
0.200
+0.062
+0.092
+0.055
+0.026
768
MAB
MSE
0.269
0.362
+0.003
-0.003
-0.000
.0.002
0.649
0.500
+0.02
+0.027
+0.033
40.05%
0.341
0.266
+0.03
40:451
+0.016
+0.031
0.383
0.697
+0.000
+0.020
+0.026
+0.074
Time point miscng
0.372
+0.071
+0035
40.011
0.42%
0.287
0.309
192
0.214
0.367
0.257
0.507
768
0.282
5 CONCLUSION
In this paper, we present S4M for time series forecasting with missing values. S4M is an end-to-end framework that first uses ATPM module to learn robust latent representation to account for missing values using rich historical data from prototype bank, and then uses missing- S4, MDMMSSS4 to directly model the mask of missing and the representation. The experimental results on four real-world benchmark datasets verify its superiority under various missing value scenarios. The ablation studies also show the importance of the masking mechanism in improving the model's robustness and accuracy. In the future, would like to explore other S4-based architectures and missing types to make our proposed method more versatile.
dual stream
ACKNOWLEDGMENTS
Jing Peng and Qiong Zhang are supported by National Key R&D Program of China grant 2024YFA1015800. Xiaoxiao Li is supported by Natural Science and Engineering Research Coun- cil Canada (NSERC), Canada CIFAR AI Chairs program, Canada Research Chair program, MITACS- Catalyst Grant Program, the Digital Research Alliance of Canada.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
George EP Box and Gwilym M Jenkins. Some recent advances in forecasting and control. Journal the the Royal Statistical Sociery. Series C (Applied Statistics), 17(2):91-109, 1968.
George EP Box, Gwilym M Jenkins, Gregory C Reinsel, and Greta M Ljung. Time series analysis: forecasting and control. John Wiley & Sons, 2015.
Wei Cao, Dong Wang, Jian Li, Hao Zhou, Lei Li, and Yitan Li. Brits: Bidirectional recurrent imputation for time series. In Advances in Neural Information Processing Systems, 2018.
Sarath Chandar, Sungjin Ahn, Hugo Larochelle, Pascal Vincent, Gerald Tesauro, and Yoshua Ben- gio. Hierarchical memory networks. arXiv preprint arXiv:1605.07427, 2016.
Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Scientific reports, 8(1))6085, 2018
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
Xiaodan Chen, Xiucheng Li, Bo Liu, and Zhijun Li. Biased temporal convolution graph network for time series forecasting with missing values. In The Twelfth International Conference on Learning Representations, 2023.
Andrea Cini, Ivan Marisca, and Cesare Alippi. Filling the G_ap_s: Multivariate time series imputa- tion by graph neural networks. arXiv preprint arXiv:2108. 00298, 2021.
Jay Darji, Nupur Biswas, Lawrence D. Jones, and Shashaanka Ashili. Handling missing data in the time-series data from wearables. In Jorge Rocha, Cláudia M. Viana, and Sandra Oliveira (eds.), Time Series Analysis, chapter 5. IntechOpen, Rijeka, 2023.
Wenjie Du, David Côté, and Yan Liu. Saits: sulf-atteetion-baee imputation for time series. Expert Systems with Applications, 19:119919, 2023.
Claude Duchon and Robert Hale. Time series analysis in meteorology and climatology: intro- duction. John Wiley & Sons, 2012.
Tlamelo Emmanuel, Thabiso Maupong, Dimane Mpoeleng, Thabo Semong, Banyatsang Mphago, and Oteng Tabona. A survey on missing data in machine learning. Journal of Big data, 8:1-37, 2021.
Vincent Fortuin, Dmitry Baranchuk, Gunnar Rätsch, and Stephan Mandt. GP-VAE: Deep proba- bilistic time series imputation. In International conference on artificial intelligence and statisties, pp. 1651-1661. PMLR, 2020.
Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. Advances in neural information processing systems, 33: 1474-1487, 2020.
Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations, 2021.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-9733 2020.
Hansika Hewamalage, Christoph Bergmeir, and Kasun Bandara. Recurrent neural networks for time series forecasting: Current status and future directions. International Journal of Forecasting, 37 (1):388-427, 2021.
Shruti Kaushik, Abhinav Choudhury, Pankaj Kumar Sheron, Nataraj Dasgupta, Sayee Natarajan, Larry A Pickett, and Varun Dutt. Ai in healthcare: time-series forecasting using statistical, neural, and ensemble architectures. Frontiers in big data, 3:4, 2020.

---

Published as a conference paper at ICLR 2025
Benjamin Lim, Simon Zohren, and Stephen Roberts. Recurrent neural filters: Learning independent bayesian filtering steps for time series prediction. In International Joint Conference on Neural Networks. IEEE, 2020.
Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X.
Liu, and Schahram Dustdar. Pyraformer: Low-complexity pyramidal attention for Long-range time series modeling and fore- casting. In International Conference on Learning Representations, 2022.
Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long. itransformer: Inverted transformers are effective for time series forecasting. arXiv preprint 0rXiv:2310.06625, 2023.
Ivan Marisca, Andrea Cini, and Cesare Alippi. Learning to reconstruct missing data from spatiotem- poral graphs with sparse observations. Advances in Neural Information Processing Systems, 35: 32069-32082, 2022.
MJ Menne, CN Williams Jr, RS Vose, and Data Files. Long-term daily climate records from stations across the contiguous united states, 2015.
Yuqi Nie, Nam Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam, time series is worth 64 words: Long-term forecasting with transformers. arXiv preprint 0rXiv:2211.14730, 2022.
Yao Qin, Dongjin Song, Haifeng Chen, Wei Cheng, Guofei Jiang, and Garrison Cottrell. A dual- stage cicentionbbaed recurrent neural network for time series prediction. In International Joint Conference on Artificial Intelligence, 2017.
Syama Sundar Rangapuram, Matthias W Seeger, Jan Gasthaus, Lorenzo Stella, Yuyang Wang, and Tim Januschowski. Deep state space models for time series forecasting. Advances in Neural Information Processing Systems, 2018.
Yulia Rubanova, Ricky TQ Chen, and David K Duvenaud. Latent ordinary differential equations for ireeullrly-sampled time series. In Advances in Neural Information Processing Systems, 2019.
David Salinas, Valentin Flunkert, and Jan Gasthaus. DeepAR: Probabilistic forecasting with autore- gressive recurrent networks. arXiv e-prints, 2017.
Mona Schirmer, Mazin Eltayeb, Stefan Lessmann, and Maja Rudolph. Modeling irregular time series with continuous recurrent units. In International conference on machine learning, pp- 19308-19405. PMLR, 2022.
Mohammad Amin Shabani, Amir Abdi, Lili Meng, and Tristan Sylvain. Scaleformer: Iterative multi-scale refining transformers for time series forecasting. In International Conference on Learning Representations, 2023.
Shun-Ya Yao Shih, Fan-Keng Sun, and Hung-yi Lee. Temporal pattern attention for multivariate time series forecasting. Machine Learning, 2019.
Xianfeng Tang, Huaxiu Yao, Yiwei Sun, Charu Aggarwal, Prasenjit Mitra, and Suhang Wang. Joint modeling of local and global temporal dynamics for multivariate time series forecasting with missing values. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 5956-5963, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, 2017.
Zihan Wang, Fanheng Kong, Shi Feng, Ming Wang. Han Zhao, Daling Wang, and Yifei Zhang, Is mamba effective for time series forecasting? arXiv preprint arXiv:2403.1II2 2024.
Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. Autoformer: Decomposition trans- formers with auto-correlatin for long-term series forecasting. In Advances Neural Information Processing Systems, 2021.
12

---

Published as a conference paper at ICLR 2025
Wang Xue, Tian Zhou, Qingsong Wen, Jinyang Gao, Bolin Ding, and Rong Jin. Card: Channel aligned robust blend transformer for time series forecasting. arXiv preprint arXiv:2305. 12095, 2023.
Vijaya Krishna Yalavarthi, Kiran Madhusudhanan, Randolf Scholz, Nourhan Ahmed, Johannes Burchert, Shayan Jawed, Stefan Born, and Lars Schmidi-Thieme. Grafiti: Graphs for forecasting irregularly sampled time series. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 16263, 2024.
Jinsung Yoon, William R Zame, and Mihaela van der Schaar. Estimating missing data in temporal data streams using multi-directional recurrent neural networks. IEEE Transactions on Biomedical Engineering, 6675):1477 2018.
Cheng Zhang, Nilam Nur Amir Sjarif, and Roslina Ibrahim. Deep learning models for price forecast- ing of financial time series: A review of recent advancements: 2020-2022. Wiley Interdisciplinary Reviews:
Data Mining and Knowledge Discovery, 14(1)=e1519 2024.
Yifan Zhang and Peter Thorburn. Handling missing data in near real-tine sal-time environmental moni- toring: A system and review of selected methods. Future Generation Computer Systems, 128: 63-72, 2022.
Haoyi Zhou, Shanghang Zhang. Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 11106-11115, 2021.
13

---

Published as a conference paper at ICLR 2025
A RELATED WORK
A.1 TIME SERIES FORECASTING
Time series forecasting has seen major improvements thanks both traditional statistical methods and modern deep learning models. The ARIMA model, for example, improves prediction accuracy by making non-stationary data more stable, which is key method in time series analysis (Box & Jenkins, 1968). Recurrent Neural Networks (RNNs) have also become important tools in this field, providing solid framework for modeling sequences and predicting time series, especially for capturing long-term patterns (Hewamalage et al., 2021). Improvements in RNN designs have led to different RNN-based approaches specifically made for forecasting (Rangapuram et al., 2018; Salinas et al., 2017; Lim et al., 2020). Attention-based models have gained attention because they can focus on key time steps, helping to capture long-term patterns that are critical for accurate fore- casts (Qin et al., 2017; Shih et al., 2019). The encoder-decoder setup, in particular, has become popular approach because of its strong forecasting ability. This has inspired various upgrades and new versions of the original Transformer model. One example is the Autoformer, which uses new architecture with an Auto-Correll Coto-Correlatinn mechanism, setting new standards for long-term forecasting accuracy (Wu et al., 2021). Similarly, the Pyraformer uses pyramidal attention strategy to model different levels of data efficiently, boosting the accuracy of long-range time series predictions (Liu al.. 2022). The Scaleformer framework refines forecasts across different scales, leading im- proved performance with little extra computation (Shabani et al., 2023). iTransformer introduces a novel approach by leveraging transformer- anisms to capture temporal dependencies in time series forecasting (Liu et al., 2023). PSt applies a patch-based technique within a transformer framework to effectively capture both short- and long-term dependencies, improving forecasting accuracy across diverse time series tasks (Nie et al., 2022). CARD leverages transformer architecture, focusing on aligning multiple temporal channels to capture dependencies effectively (Xue et al., 2023). Besides these advances, new models like the structured state space squence (S4) model combine the strengths of RNNs and CNNs, offer- ing flexible solutions for wide range of tasks, including generation, forecasting, and classification (Gu et al., 2021). S4 model combines the strengths of state-space models with modern deep learning architectures and can efficiently model long sequences.
architecture with adaptive self-attention mech-
A.2 MISSING DATA IN TIME SERIES
In many real-world scenarios, datasets can be incomplete due to unforeseen events such as equip- ment failure or communication errors, making it crucial to address time series forecasting with missing data. GRU-D (Che et al., 2018) stands out classic method to manage missing data in recurrent models. Subsequent advances such as BRITS (Cao et al., 2018) have further refined the approach for LSTMs. The field has also seen the emergence of various imputation techniques, including M-RNN, GP-VAE, and SAITS, which prioritize the estimation of missing values to im- prove the precision of forecasting (Yoon et al., 2018; Fortuin et al., 2020; Du al., 2023). Latent ODE (Rubanova et al., 2019), Neural ODE (Chen et al., 2018), CRUs (Schirmer et al., 2022), and GraFITi (Yalavarthi et al., 2024) each address missing values in time series through different mech- anisms, with Latent ODE (Rubanova et al., 2019) and Neural ODE (Chen et al., 2018) learning continuous dynamics over time, CRUs (Schirmer et al., 2022) utilizing confidence regularization to improve imputation accuracy, and GraFITi (Yalavarthi et al., 2024) applying graph-based methods to capture temporal and spatial dependencies for missing data recovery. LGNet innovatively cap- tures local and global temporal dynamics through memory network (Tang et al., 2020). BiTGraph dexterously navigates temporal dependencies and spatial structures. By explicitly incorporating the challenge of missing values into its model architecture, BiTGraph aims to optimize the information flow and mitigate the adverse effects of data incompleteness (Chen et al., 2023).
B NOTATION TABLE
A summary of key notations used in the main paper is given in Tab. 4.
14

---

Published as a conference paper at ICLR 2025
Table 4: Notations
Notations
Description
X (())
look-back time series
M
mask:
indicator for missing for look-back time series
(
horizon time series value of the time series time point t representation learning output at time point S4 hidden state at time point
Or
h
predicted value of S4 mask at time point the centroid of the jth cluster length of the look-back window length of the horizon window dimension of the time series encoder output dimension output channel ConvD1
m:
EH
D
R
F
K1
prototype bank parameter: maximum number of clusters
K2
prototype hank parameter: maximum number of elements within each cluster threshold for similarity in prototype bank writing momentum coefficient S4 model parameters { to + I,..., to + E encoder
T2
0
to : to + Ep. Eg. Em
C ADDITIONAL DETAILS OF THE PROPOSED METHOD
In this section, we provide additional details of the proposed methods. We describe the procedure for local statisties extraction in Section C.I, the encoder design in the representation learning module in Section C.2, and the design of S4 blocks in Section C.3.
C.I LOCAL STATISTICS EXTRACTION
As the first step in dealing with missing values in time series, we extract useful local statistical features using contextual information from observed parts of the time series for missing values. Specifically, we denote Fmin, Fmax E RRD respectively as the minimum and maximum of the ob- served value of X (1), Amin &D A max & are the time gap between each entry of X with minn & maxx We use the combination of two exponential weights to extract local feature information from missing data. Specifically, we let
Z (L) (L) x (L) + (1 M (1) Ømin lixwman))
be the local statistics where
Ω exp max (0, W A min b1)}
Ω. exp max (0, W + b2)}
22 2 12), Ω l22 + 122)
and W2, and be are the decay parameters. The local statistics Z are fed in the ATPM module to query from the prototype bank.
C.2 ENCODER ARCHITECTURE
The architecture of the encoder Epp. Eeg, and E. contains (1) a delay embedding layer, (2) 2D- convolutional layer with ReLU activation, (3) self-attention layer, and (4) S4 layer. We describe these layers, respectively.
Delay embedding The delay embedding layer converts the original two-dimensional matrix (1) (or (L) in Fm into third-order tensor. This technique involves recursively augmenting the
15

---

Published as a conference paper at ICLR 2025
multivariate time series by unfolding the matrix along the temporal dimension. This process signif- icantly enriches the local information at each time point by incorporating its historical time series data. Consequently, this enrichment facilitates the formalization and storage of various patterns.
Convolution We then incorporate a convolutional layer with a kernel size of W in the temporal dimension and D in the variable dimension to capture local temporal patterns and inter-variable dependencies. Subsequently, the output is passed through a Rectified Linear Unit (ReLU) layer. The ReLU layer's output is a matrix with dimensions R Te where It represents the number of filters in the convolutional layer and To = L W + 1. Additionally, a dropout layer is applied subsequent to the ReLU layer to prevent overfitting.
Attention Subsequently, we implement an attention mechanism over the temporal dimension of the sequence. enabling the model selectively emphasize salient information without changing the rank of tensor.
S4 layer The output from the attention layer is then fed into an S4 block. Unlike the layer of self- attention above, the S4 block was used to compress temporal information. Within this framework, we employ a S4 as an embedding tool, which serves to encapsulate the embedding of size T, x R at each time point into fixed-size representation vector of length R.
C.3 DESIGN OF MDS-S4 BLOCKS
Our second MDS-S4 module consists of one MDS-S4 block and multiple normal S4 blocks, each designed process sequential data efficiently. The architecture begins with an MDS-S4 block. MDS-$4 is the core and initial layer of this block, which has dual inputs, the representation or learned from ATPM and m. E, (m.), both are fed into dual-stream S4 block. To address the missing data problem in S4 models, we incorporate m, to distinguish the missing time points, enabling the model treat them differently from the observed data (e.g., by referring to data in the prototype bank). At the same time, this structure ensures that the core properties of the S4 model are preserved. To this end, we seek term that can flag missing values while preserving the HiPPO structure of S4. We find that integrating additional masking terms M, inspired by Che et al. (2018), to serve as simple yet effective indicator for the model to recognize missing values. However, since the elements of M take binary values (0 or 1), they are not naturally on the same scale as the other terms in equation 4. To address this, we design an encoder to transform the mask information to appropriate scale. Incorporating this term still preserves the HiPPO structure of S4, thereby enriching the model with additional information while maintaining its core advantages. The output of the dual-streant S4 is then fed into residual connection, coupled with layer normalization, to address gradient vanishing. Subsequently, a ID convolutional layer with a kernel size of and F output channels is applied together with ReLU. Then, it comes another convolutional layer that verts the output back to R channels. Finally, a dropout layer is integrated to introduce regularization, which is crucial for preventing overfitting. The culmination of these operations completes a single MDS-S4 block within the architecture. We list these layers of the block in Tab. 5 for easy reference.
Table 5: Architecture of MDS-S4 block. For convolutional layer (ConvID), we list parameters with sequence of input and output dimension, and kernel size.
Layer
Details
MDS- S4 model or S4 model, Residual, LayerNorm
2
Conv ID(R, F, 1), ReLU, Dropout
3
Conv2D(F, R, 1), Dropout
The following $4 blocks in MDS-S4 module have the same architecture with MDS-S4 block, except for the initial MDS-S4 model replaced with traditional S4 model. Begin with the MDS S4 block, the output of one block is fed directly as input to the subsequent block, This iterative process allows for increasingly complex feature extraction and integration. The final output from the last block in the sequence represents S4M's prediction.
16

---

Published as a conference paper at ICLR 2025
D EXPERIMENT DETAILS AND MORE RESULTS
D.1 BASELINE METHODS
In this section, we describe the baseline methods that we compare with. The baselines include latest state-of-art methods and some classic methods. For models not specifically designed for missing data forecasting, we impute the missing observations with the mean value and conduct experiments on the imputed dataset.
GRU-D: It is time series model that extends the Gated Recurrent Unit (GRU) by incorpo- rating decay mechanisms to handle missing data and capture temporal dependencies (Che et al., 2018).
BRITS: It's time series imputation model that integrates Bidirectional Recurrent Neural Network (RNN) with a time decay mechanism to capture the relationships between missing values and observed data (Cao et al., 2018).
Autoformer: It incorporates seasonal decomposition mechanism that captures both long- term trends and short-term seasonal patterns. Autoformer also leverages auto-correlation capture the dependencies between different time steps, allowing it model the temporal relationships effectively (Wu et al., 2021).
Transformer: It's foundational sequential model that utilizes stacked self-attention blocks effectively capture temporal dependencies in time series data (Vaswani et al., 2017).
iTransformer: It introduces novel methodology by integrating transformer- based architec- ture with adaptive self-attentio mechanisms, enabling more efficient handling of complex temporal dependencies in time series forecasting tasks (Liu et al., 2023).
CARD: It leverages transformer architecture, focusing on aligning multiple temporal channels to capture dependencies effectively. Also, CARD incorporates a token blend module, which efficiently utilizes multi-scale knowledge to enhance the model's predic- tive power (Xue et al., 2023).
CRUs: introduces a unique method for handling missing or irregularly spaced data points, incorporating confidence-bel based regularization to improve the robustness and accuracy of time series forecasting models (Schirmer et al., 2022).
GraFITi: A novel approach that models irregularly sampled time series data using graph- based techniques (Yalavarthi et al., 2024).
BiTGraph: A state-of-ther art method that performs end-to-end prediction with biased tem- poral convolutional graph networks when missing data is present (Chen et al., 2023).
S4 (Mean): Impute missing data using the global mean and employ S4 blocks as the back- bone.
S4 (Ffill): Impute missing data by forward filling with the latest observation, using S4 blocks the backbone.
S4 (Decay): Impute missing data by combining the global mean and the latest observation, with a decay factor controlling the weighting, and use S4 blocks as the backbone.
S4 (SAITS): Fill missing entries with the state-of-the-arl imputation method SAITS, using the imputed data input for S4 blocks. SAITS is time series forecasting method that em- ploys self-attention mechanism to capture long-term dependencies and trends, enabling more accurate imputation across various temporal patterns (Du et al., 2023).
We also provide detailed comparisons and computational cost analysis for above methods in Tab. 6. To measure the training and inference time, conducted performance experiments using the elec- tricity dataset, with batch size of 16 and a hidden size of 512. The maximum memory usage, along with the training and inference times, were recorded for single epoch.
We observe that the ODE-based method incurs an extraordinarily high computational cost com- pared to other approaches. Notably, $4M demonstrates significantly lower memory usage compared to other state-of ofte-oo--the-rn art transformer-basee methods, particularly iTransformer and CARD. Addi- tionally, ba-based methods generally exhibit shorter training times compared to recurrent methods
17

---

Published as a conference paper at ICLR 2025
designed for forecasting with missing observations, such BRITS and GRU-D. These results rein- force our decision to focus on S4-based architectures due to their superior efficiency.
Table 6: Computation Cost for Different Methods ('OOM' refers "Out-of- -omoof---mo""")
Method
GRU-D
CRUs
GraFITi
S4(Mean)
S4(Decay)
S4M(Ours)
(MB)
445.70
32146.06
OOM
764.75
887.28
1071.39
Training Time(s)
244.26
64990.81
OOM
84.71
90.07
110.65
Inference Time(s)
32.64
1524.16
OOM
12.16
13.23
14.55
Method
Autoformer
BiTGraph
Transformer
iTransformer
CARD
BRITS
MeMory(MB)
1106.41
2564.35
777.69
2107.67
7517.68
$57.68
Training Time(s)
95.98
267.74
78.00
74.16
274.29
486.68
Inference Time(s)
17.88
24.88
10.35
12.31
34.40
57.19
D.2 DATASET DETAILS
In Tab. 7, we present the number of variables (Variables), the total length of the time series (Time steps), and the frequency that observations are made (Granularity). We select these four datasets because they exhibit significant variation size, number variables, and the presence or absence of seasonality.
Table 7: Dataset statistics.
Data
Variables
Time steps
Granularity
Electricity
321
26.304
hour
ETTh1
7
17,420
15 min
Weather
21
52,696
10 min
Traffic
862
17,544
hour
For all datasets in our experiment, consider different missing scenarios: time point missing and variable missing, which is illustrated in Fig. 5. Under the time point missing scenario, we first
Time Point Missing
Variable
Missing
Figure 5: Illustration block missing patterns: Time Point Missing (Left) and Variable Missing (Right). Each column represents variable in the time series, and each corresponds to observa- tions at a specific time point. Red blocks indicate missing observations, while white blocks represent observed data. Missing values are consecutive in both patterns. For time point missing, all variables are missing at a given time point. For variable missing, some variables may remain observed at the same time point.
randomly select a ratio r of time points, and for each selected time point, we remove following consecutive time points of length 5 and eliminate all variables at those time points. In the variable missing scenario, we perform the same procedure independently for each variable. When generating the missing data, the missing ratio ranges from 0.03, 0.06, 0.12, 0.24. Due the design of the
18

---

Published as a conference paper at ICLR 2025
consecutive missing points, the overall missing ratio (the percentage of missing entries in the times series matrix) is higher than r, and we report these values Tab. under different values r.
Table 8: Overall missing ratio statistics.
Missing pattern
0.030
0.060
0.120
0.240
Electricity
0.139
0.260
0.450
0.694
Time point missing
ETThl
0.122
0.231
0.399
0.616
Traffic
0.139
0.256
0.447
0.705
Weather
0.132
0.247
0.432
0.667
Electricity
0.139
0.258
0.450
0.696
Variable
ETTh1
0.122
0.228
0.395
0.613
missing
Traffic
0.139
0,259
0,451
0.698
Weather
0.133
0.258
0.431
0.667
D.3 HYPER-PARANEE PARAMETER DETAILS
The learning rates are set to 0.01 for the Electricity and Traffic datasets, 0.005 for the ETTh1 dataset, and 0.001 for the Weather dataset. The dimensions of the hidden layers tricity and Traffic datasets, and 256 for the ETTh and Weather datasets. The number of basic blocks layers selected from {2, 4, 4, The batch size set for all experiments optimizer and implement an early stopping strategy across all experiments. Other hyperparameters for both the proposed method and baseline methods are adjusted based on their performance on the validation set. The performance of different methods is evaluated using Mean Squared Error (MSE) and Mean Absolute Error (MAF). For both metrics, lower values indicate better performance.
512 for the Elec-
16. We the Adam
D.4 SENSITIVITY ANALYSIS
In this section, evaluate the sensitivity of our method with respect the size of queue Ki, K2. threshold 7ı, 72, the dimension R of encoder output, the size of the short period retrieve window and the number of memory centroids K choose. All of the experiments are done under the time point missing scenario with r = 0.06, look-back window H 96, which is a representative scenario make analysis on.
D.4.1 BANK INITIALIZATION
In experiment, find the performance is insensitive to the initial cluster configuration, the clusters are updated continuously throughout the training process. To provide evidence, we present the experimental results four datasets using different cluster numbers for initialization, shown below in Tab. 9, In practice, we recommend using to clusters for initialization or determining the optimal number of clusters based the within-cluster sum of squares.
Table 9: Performance of S4M (Ours) with different number of clusters for initialization with other parameters fixed.
number clusters
12
16
MAE MSE
0.415 0.356
0.415
0.415
0.415
0.415
0.415
0.415
Electricity
0.356
0.356
0.356
0.357
0.358
0.356
MAE
0.647
0.647
0.648
0.648
0.647
0.648
0.65
ETTh1
MSE
0.768
0.767
0.77
0.77
0.767
0.767
0.773
MAE
0.386
0.39
0.387
0.385
0.388
0.388
0.385
Weather
MSE
0.307
0.31
0.308
0.306
0.31
0.31
0.307
MAE
0.510
0.505
0.504
0.515
0.509
0.513
0.509
Traffic
MSE
0.966
0.954
0.944
0.999
0.974
0.992
0.985
19

---

Published as a conference paper at ICLR 2025
D.4.2 ANALYSIS OF K1 AND K₂
We fix 71 0.95, 72 = 0.6, R 256, and 8 = 32. K1 represents the size of the maximum centroid, which governs the storage of prototype clusters. Choosing an appropriate value for K₁ allows the bank to effectively filter out outdated representations, especially cases with large number of patterns in the original time series. Tab. 10 indicates that suitable value for K is below 50.
For the analysis of K2, we set 1 A smaller K2 allows the bank to store only newly generated representations, ensuring that it remains aligned with the model's updates. Tab. 11 shows the performance changes across different values of K2, suggesting that relatively smaller value is more beneficial. We do not include results for ETThl because its shorter time series length and variable dimensions result in a significantly smaller pattern size, which does not require constraint the number of clusters.
K₂ controls the size of each cluster in the prototype bank.
Table 10: Performance of S4M (Ours) when K fixed.
19, 30, 50, and 100 with other parameters
Data
Metric 4
5
10
30
50
100
MSE
0.372
0.377
0.377
0.376
0.376
Electricity
MAE
0.287
0.293
0.293
0.293
0.290
MSE
0.347
0.345
0.345
0.347
0.347
Weather
MAE
0.270
0.267
0.267
0.268
0.268
MSE
0.442
0.438
0.437
0.436
0.439
Traffic
MSE
0.863
0.823
0.819
0.817
0.830
Table 11: Performance of S4M (Ours) when K₂ = 5, 10, 20, 50, and 100 with other parameters
fixed.
Data
Metric
3
5
10
20
50
100
MAE
0.393
0.377
0.393
0.398
0.393
0.394
Electricity
MSE
0.313
0.299
0.312
0.319
0.312
0.313
MAE
0.606
0.610
0.607
0.603
0.605
0.601
ETTh1
MSE
0.695
0.700
0.694
0.655
0.659
0.655
MAE
0.347
0.345
0.343
0.346
0.346
0.347
Weather
MSE
0,268
0.267
0.268
0.268
0.268
0.268
MAE
0.438
0.436
0.438
0.437
0.438
0.438
Traffic
MSE
0.815
0.818
0.827
0.828
0.830
0.822
D.4.3 ANALYSIS R AND
In this section, we performe sensitivity analysis when the dimension of the encoder R and the short period window size varies. We set K1 30, K2 50, Tı 0.95, and T2 0.6. For the analysis of R, we fix s = and vary the values of R from 16 to 1024. Similarly, for the analysis of s, set A 256 and vary the values of from to 48. Tab. 12 shows that It significantly affeets performance, with values larger than 128 benefiting the model. Tab. 13 shows that increasing generally improves performance.
D.4.4 ANALYSIS 71 AND 72
We set K; 30, K2 50, x 16, and R 256, and then vary the values of Tı and 72 across four datasets to observe how changes in these thresholds affect model performance in Tab. 14 1 Tab. 17. Overall, the forecasting performance is less sensitive to Ti and 72 compared to other hyperparameters previously analyzed. Specifically, model performance ETThl and Traffic is more sensitive to these threshold values than on the other two datasets. ETTh I achieves its best performance when < 0.95 and 72 < 0.9, while Traffic performs optimally T 0.9 and 72 0.5. Electricity and Weather exhibit similar patterns, with slight performance improvements when т = 0.975 and
72 0.5.
20

---

Published as a conference paper at ICLR 2025
Table 12: Performance of s S4M (Ours) when R ranging from 16
1024 with other parameters fixed.
Dataset
Metric J
16
32
64
128
236
512
1024
MAE
0.409
0.400
0.406
0.388
0.376
0.379
0.375
Electricity
MSE
0.358
0.335
0.339
0.308
0.279
0.295
0.292
MAE
0,480
0,438
0,465
0,444
0.418
0,418
0.400
ETThl
MSE
0.895
0.826
0.846
0.855
0,363
0.351
0.332
MAE
0.585
0.591
0.603
0.571
0.571
0.610
0.609
Weather
MSE
0.654
0.656
0.680
0.624
0.621
0.699
0.690
MAE
0.329
0.320
0.329
0.315
0.352
0.317
0.349
Traffic
MSE
0.257
0.246
0.255
0.243
0.277
0.244
0.274
Table 13: Performance of S4M (Ours) when varies with other parameters fixed.
Dataset
8
16
32
48
MAE
0.379
0.379
0.378
0.378
Electricity
MSE
0.297
0.296
0.295
0.293
MAE
0.596
0.570
0.584
0.582
ETThl
MSE
0.660
0.624
0.656
0.649
MAE
0.345
0.350
0.351
0.332
Weather
MSE
0.267
0.275
0.279
0.253
MAE
0.453
0.438
0.443
0,436
Traffic
MSE
0.847
0.826
0.853
0.786
D.4.5 MORE DISCUSSION ON HYPERPARAMETERS K1, K2, AND 72-
Based on our results, we recommend setting K₂ between and this range is both effective and relatively insensitive to performance variations. The suggested ranges for Tı and 72 are [0.3, 0.6) and [0.8, 1.0), respectively, with both parameters selectable using the validation set. Most experiments robust to choices of K₁ 30 K1 50. If dataset contains very large number of clusters, as determined by preliminary experiment with a higher K₃ value, the number of clusters can be adjusted accordingly.
Table 14: Performance under different values of Tj and T2 on Electricity. Entries with --' mean the
experiment is not meaningful in setting because 7i 72.
\a
Metric J
0.050
0.100
0.300
0.500
0.700
0.900
MAE 0.393
0.392
0.392
0.393
0.500
MSE
0.312
0.311
0.311
0.311
MAE 0.393
0.393
0.393
0.393
0.393
0.700
MSE
0.311
0.312
0.312
0.312
0.312
MAE
0.392
0.391
0.392
0.392
0.393
0.393
0.900
MSE
0.311
0.310
0.311
0.311
0.312
0.311
MAE
0.395
0.395
0.394
0.393
0.393
0.393
0.950
MSE
0.316
0.315
0.315
0.313
0.314
0.312
MAE
0.392
0.395
0.395
0.394
0.393
0.393
0.975
MSE
0.311
0.315
0.316
0.314
0.314
0.312
21

---

Published as a conference paper at ICLR 2025
Table 15: Performance under different values of and 72 on Weather. Entries with mean the experiment not meaningful in our setting because we set 7j 2 72.
T 72
Metric +
0.050
0.100
0.300
0.500
0.700
0.900
MAE
0.342
0.342
0.342
0.342
0.500
MSE
0.269
0.269
0.269
0,268
MAE
0.343
0.342
0.343
0.343
0.342
0.700
MSE
0.270
0.270
0.270
0.270
0.269
MAE
0.343
0.343
0.343
0.343
0.343
0.343
0.900
MSE
0,271
0.271
0.271
0.271
0.270
0.270
MAE
0.340
0.341
0.341
0.340
0.341
0.344
0.950
MSE
0,267
0.268
0.268
0.267
0.268
0.270
MAE
0.339
0.339
0.339
0.340
0.342
0.345
0.975
MSE
0.266
0.266
0.266
0.267
0.269
0.270
Table 16: Performance under different values of 71 and T2 on ETTh1. Entries with + mean the experiment is not meaningful in our setting because we set 71 2 T2.
Ti 772
Metric
0.050
0.100
0.300
0.500
0.700
0.900
MAE
0,591
0.591
0.591
0.591
0.500
MSE
0.662
0.662
0.663
0.663
MAE
0.591
0.591
0.591
0.591
0.591
0.700
MSE
0.663
0.662
0.663
0.662
662
MAE
0.591
0.591
0.591
0.591
0.591
0.591
0.900
MSE
0.659
0.659
0.659
0.659
0.659
MAE
0.591
0.591
0.591
0.591
0.591
0.597
0.950
MSE
0.651
0.651
0.651
0.651
0.651
0.671
MAE
0.580
0.582
0.583
0.582
0.586
0.599
0.975
MSE
0.643
0.647
0.647
0.644
0.650
0.680
Table 17: Performance under different values of and 72 Traffic. Entries with mean the experiment is not meaningful in our setting because we set Ti 2 T2.
Ti Aa
Metric
0.050
0.100
0.300
0.500
0.700
0.900
MAE
0.444
0.442
0.445
0.442
0.500
MSE
0.870
0.855
0.870
0.856
MAE
0.442
0.439
0.441
0.440
0.441
0.700
MSE
0.854
0.836
0.848
0.833
0.857
MAE
0.441
0.441
0.440
0.438
0.440
0,441
0.900
MSE
0.851
0.840
0.849
0.808
0.852
0.857
MAE
0.439
0.439
0.446
0.440
0.444
0.439
0.950
MSE
0.837
0.834
0.869
0.842
0.859
0.838
MAE
0.445
0.443
0.442
0.438
0.445
0.439
0.975
MSE
0.865
0.848
0.857
0.825
0.872
0.851
22

---

Published as a conference paper at ICLR 2025
D.5 EXPERIMENT RESULT ON DATASET WITH NO MISSING VALUE
If there is no missing data, as expected in Tab. 18, our method does not show a significant advantage over the other methods but still maintains very competitive performance. For this experiment, we report the results using horizon window of size 96 and lookback window of size 96, with no missing values in the original dataset.
Table 18: Comparison of forecasting performance of S4M (Ours) and baselines with t 96, l =
96 on four datasets with no missing value.
Dataset
Metric +
BRITS
GRU-D
Transformer
Autoformer
S4
BiTGGrap
S4M(Ours)
MAE
0.398
0.413
0,411
0.352
0.386
0.348
0.383
Electricity
MSE
0.318
0.332
0.242
0.301
0.254
0,295
MAE
0.676
0.571
0.604
0.556
0.538
0.530
0,538
ETThl
MSE
0.867
0.636
0.677
0,588
0.560
0,571
0,560
MAE
0.373
0.370
0.383
0,306
0,363
0,504
0.332
Weather
MSE
0.291
0,301
0.298
0.235
0,301
0.494
0.259
MAE
0.428
0.446
0.405
0.454
0.425
0.504
0.414
Traffic
MSE
0.770
0.840
0.707
0.705
0.425
0.879
0.761
D.6 ADDITIONAL EXPERIMENT RESULTS
In the main text of the manuscript, we include the comparison of $4M with different baselines under the missing ratio 0.06. In this section, provide the complete additional results in Tab. 19 to Tab. 24 when r = 0.03, = 0.12, and r 0.24. Similar to the = 0.06 case, Our proposed S4M consistently achieves the best second-best performance across most settings, demonstrating its robustness in handling missing data.
Table 19: Comparison of forecasting performance of S4M (Ours) and baselines four datasets with
various look- back window length under time point missing scenario when missing ratio = 0.03.
Data
f2
Merric
BRITS
GRU-D
Trans.
Auto.
BiTGraph
(Mcan)
$4
(Pill)
$4
(Decay)
$4
(SAITS)
$4
(Ours)
54M
MAE
0.606
0.419
0.413
0.374
0.390
0.395
0.409
0.397
0.409
0.370
96
MSE
0.272
0.334
0.281
Elcetricit
192
MAE
MSE
0.616
0.421
0.409
0.348
0.380
0.380
0.383
0.372
0.395
0. 0.369
0.271
MAE
0.627
0.420
0.420
0.346
0.366
0.377
0.384
0.378
0.392
0.371
384
MSE
0.619
0.339
0.333
0.240
0.264
0.285
0.289
0.278
0.300
0.273
768
MAE
0.635
0.419
0.409
0.353
0.391
0.378
0.382
0.375
0.392
0.372
MSE
0.632
0.338
0.324
0.251
0.289
0,286
0.286
0.276
0.299
0.273
MAE
0.696
0.624
0.681
0.624
0.528
0.618
0.625
0.603
0.632
0. 0.565
MSE
0.917
0.734
0.885
0.752
0.556
0.721
0.732
0.689
0.757
0.603
MAE
0.731
0.629
0.669
0.619
0.607
0,596
0.599
0.588
0.619
0,355
EITH
MSE
0.971
0.742
0.883
0.739
0.736
0.661
0.663
0.650
0.710
0.566
MAE
0.625
0.545
0.557
384
MSE
LO10
0.734
0.933
0.746
0.599
0.663
0.669
0.669
0.697
0.586
0,580
768
MSE
L,61
0,780
1.157
0.768
0.760
0.695
0.696
0.700
0.711
0.624
MAE
0.408
0.402
0.436
0.400
0.534
0.372
0.366
0.388
0.424
0.345
96
MSF
0.327
0.336
0.365
0.327
0.531
0.305
0.298
0.331
0.375
0.281
Weather
192
MAE
0.378
0.378
0.420
0.412
0.433
0.350
0.337
0.345
0.374
0.315
MSF
0.303
0.303
0.351
0.342
0.401
0.268
0.255
0.271
0.297
0.246
384
MAE
MSF
0.305
0.375
0.305
0.375
0.345
0.414
0.421
0.363
0.694
0.653
0.338
0.263
0.251
0.326
0.261
0.337
0.294
0.373
0.256
0.333
MAE
0.385
0.385
0.394
0.448
0.618
0.351
0.340
0.333
0.370
0.336
768
0.314
0.273
0.255
0.291
0.259
96
MAE
0.677
0.504
0.449
0.471
0.516
0.455
0.444
0.433
0.455
0,420
MSE
1.198
0.767
0.915
0.837
0.822
0.81T
0.837
0.
MAE
0.681
0.501
0.437
0.405
0.537
0.404
0.399
0.391
0.413
0.381
Trafic
MSE
1.225
0.927
0.754
0.648
0.944
0.698
0.710
0.710
0.711
0.697
384
MAE
0.680
0.507
0.417
0.390
0.527
0.401
0.392
0.385
0.406
0.380
MSE
1.216
0.940
0.715
0.632
0.908
0.695
0.700
0.703
0.701
0.6 0.689
MAE
0.680
0.507
0.456
0.435
0.391
0.389
0.388
0.396
0.380
MSE
1,223
0.882
0.772
0.715
0.693
0.707
0.731
0.694
0. 6.696
23

---

Published as a conference paper at ICLR 2025
Table 20: Comparison of forecasting performance of S (Ours) and baselines four datasets with
various look- bbck window length under variable missing scenario when missing ratio r = 0.03.
Data
Metric
BRITS
GRU-D
Trans.
Auto.
BiTGraph
(Mean)
S4
(Fill)
S4
(Decay)
54
(SAITS)
(Ours)
S4M
96
MAE
0.415
0.424
0.401
0.356
0.373
0.384
0.386
0.389
0.403
0.379
MSF
0.339
0.351
0.312
0.250
0.280
0.301
0.303
0.305
0.321
0.290
Electricity
192
MAE
0.423
0.408
0.415
0,338
0.381
0.370
0,371
0.376
0.385
0.358
MSE
0.349
0.327
0.326
0.226
0.281
0.275
0,281
0.280
0.289
0.260
MAE
0.439
0.429
0.409
0.355
0.380
0.364
0,368
0.374
0.384
0.362
384
MSF
0.368
0.316
0.251
0.279
0.268
0.274
768
MAE
0,437
0.445
0.406
0.358
0.376
0.364
0.373
0.373
0.388
0.362
MSF
0.370
0.378
0.323
0.253
0.274
0.267
0.284
0.277
0.294
MAE
0.691
0.599
0.607
0.593
0.544
0.645
0.623
0.606
0.644
0.560
MSE
0.892
0.678
0.683
0.681
0.600
0.757
0.714
0.686
0.786
0.598
MAE
0.725
0.601
0.686
0.564
0.580
0.605
0.603
0.584
0.628
0.547
ETTM
192
MSE
0.943
0.679
0.890
0.614
0.682
0.605
0.668
0.631
0.732
0.574
384
MAE
0.738
0.600
0.596
0.581
0.600
0.591
0.601
0.627
0.556
MSE
0.982
0.680
0.672
0.673
0.680
0.661
0.636
0.676
0.730
0,593
MAE
0.771
0.759
0.619
0.619
0.600
0.606
0.642
0.569
768
MSF
1,024
0.689
0.967
0.672
0.744
0.661
0.665
0.690
0,766
0.599
96
MAE
0.375
0.373
0.384
0.377
0.511
0.375
0.362
0.360
0.388
0.340
MSE
0.298
0.308
0.306
0.296
0.505
0.319
0.302
0.300
0.329
0.272
Weather
192
MAE
0,380
0.349
0.406
0.388
0.410
0.325
0,317
0.314
0.349
0.308
MSE
0.317
0.278
0.332
0.311
0.374
0.249
0.237
0.239
0,270
0.227
MAE
0.417
0.357
0.369
0.403
0.626
0.324
0,315
0.306
0.347
0.302
384
MSE
0.358
0.287
0.288
0.338
0.662
0.247
0,234
0.230
0.266
0.222
MAE
0.437
0.362
0.372
0.421
0.603
0.325
0.317
D.306
0.342
0.3I0
768
MSE
0.387
0.294
0.306
0.374
0.635
0.246
0.235
0,225
0.342
0.220
MAE
0.680
0.459
0.439
0.452
0.510
0.427
0.431
0.437
0.479
0.431
MSE
1.211
0.845
0.756
0.746
0.894
0.779
0.8035
0.791
0.847
0.798
MAE
0.669
0.475
0.434
0.386
0.504
0.377
0.387
0.380
0.419
0.360
Trafii
MSE
1.181
0.873
0.741
0.619
0.839
0.694
0.727
0.694
0.738
0.598
MAE
0.669
0.467
0.397
0.389
0.482
0.373
0,383
0.375
0.372
384
MSE
1.181
0.843
0.689
0.632
0.790
0.685
0.722
0.691
0.732
0.675
MAE
0.670
0.460
0.401
0.404
0.374
0.385
0.376
768
MSE
1.182
0.832
0.702
0.655
0.688
0.732
0.687
0,740
0.680
Table 21: Comparison of forecasting performance of S 4M (Ours) and baselines on four datasets with various look- window length under time point missing scenario when missing ratio 0.12.
S4
S4
S4
S4
S4M
Data
Metric
BRITS
GRU-D
Trans.
Auto.
BiTGraph
(Mean)
(Fill)
(Decay)
(SAITS)
(Ours)
MAE
0.655
0.458
0.419
0.405
0.414
0.429
0.450
0.421
0.478
0.402
MSE
0.666
0.394
0.339
0,313
0.351
0.369
0.390
0.347
0.442
0.328
Hlcctricity
192
MAE
0.652
0.450
0.411
0.378
0.401
0.398
0.404
0.399
0.414
0.374
MSE
0.666
0.386
0.326
0.279
0.307
0.320
0,317
0.307
0.340
0.281
MAE
0.659
0.450
0.430
0.395
0.437
0.395
0.398
0.397
384
MSE
0.682
0.383
0.344
0.301
0.347
0.317
0.310
0.304
0.336
0.286
MAE
0. 0.659
0.450
0.432
0,390
0.437
0.397
0,397
0.395
0.413
0.382
768
MSE
0.680
0.384
0.348
0.299
0.347
0.319
0.310
0.303
0.337
0.299
MAE
0.733
0.680
0.701
0.675
0.566
0.673
0.663
0.637
0.752
0.591
96
MSE
0.983
0.853
0.909
0.831
0.627
0.830
0.810
0.749
1.004
0.674
ET1bl
192
MAE
MSF
0.759
1.022
0.865
0.687
0.906
0.686
0.780
0.662
0.764
0.628
0.616
0.695
0.691
0.615
0.675
0.605
0.711
0.883
0.595
0.648
384
MAE
0.764
0.689
0.713
0.669
0.678
0.608
0.614
0.613
0.701
0.588
MSF
1.042
0.869
0.949
0.794
0.905
0.679
0.686
0.688
0.841
0.638
768
MAE
0.793
0.701
L.099
0.663
0.654
0.711
0.624
0.633
0.711
0.611
MSE
L.078
0.890
1.118
0.768
0.802
0.863
0.709
0.729
0.663
MAE
0.413
0.402
0.471
0.698
0.556
0.401
0.385
0.385
0.536
0.355
MSE
0.341
0.336
0.492
0.767
0.562
0.348
0.323
0.325
0.540
0.278
MAE
0.426
0.377
0.468
0.706
0.455
0.368
0.344
0.348
0.447
0.335
Weather
MSE
0.365
0.303
0.414
0.789
0.431
0.299
0.27
0.275
0.386
0.253
MAE
0.454
0.383
0.451
0.708
0.656
0.359
0.345
0.337
0.453
0.331
MSE
0. 0.405
0.313
0.396
0. 806
0.699
0.286
0. 0,266
0.263
0.393
0.256
MAE
0,480
0.382
0.418
0.719
0.633
0.364
0.340
0.336
0,446
0,350
768
MSE
0.439
0.312
0.362
0.838
0.673
0.288
0,264
0.261
0,381
0.276
96
MAE
0.693
0.531
0.464
0.516
0.554
0.469
0,495
0.463
0.527
0.454
MSE
1.221
0.915
0.812
0.850
1.025
0.827
0.872
0.806
0.951
0.841
MAE
0.685
0.521
0.448
0.413
0.501
0.401
0,441
0.419
0.426
0.397
Trafiic
192
MSE
1.201
0.904
0.779
0.667
0.835
0.716
0.744
0.720
0.756
0.703
384
MAF
0.686
0.576
0.499
0.445
0.533
0.394
0.439
0.397
0.416
0.393
MSE
1.222
0.962
0.839
0.744
0.908
0.692
0.740
0.694
0.731
0.702
MAE
0.688
0.563
0.486
0.530
0.386
0.425
0.397
0.415
0.389
768
MSE
1.226
0.949
0.801
0.920
0.687
0.722
0.694
0.732
24

---

Published as a conference paper at ICLR 2025
Table 22: Comparison of forecasting performance of S4M (Ours) and baselines on four datasets with
various look-back window length under variable missing scenario when missing ratio r
0.12.
S4
S4
S4
S4
S4M
Data
Metric
BRITS
GRU-D
Trans.
Auto.
BiTGraph
(Mean)
(PAll)
(Decay)
(SAITS)
(Ours)
MAE
0.641
0.452
0.402
0.395
0.426
0.396
0.403
0.410
0.387
96
MSE
0.642
0.395
0.320
0.300
0.343
0.324
0.329
0.337
0.454
0.307
Electricity
192
MAE
0.644
0.432
0.412
0.368
0.407
0.374
0.373
0.391
0.465
0.362
MSE
0.649
0.368
0.331
0.262
0.315
0.284
0.281
0.304
0.388
0.270
384
MAE
0.625
0.453
0.413
0.390
0.405
0.372
0,376
0.391
0.462
0.364
MSE
0.619
0.396
0.329
0.291
0.309
0.279
0.283
0.304
0.459
0.271
MAE
0.643
0.466
0.442
0. 0.369
0.397
0.377
0.381
0.385
0.381
0.365
768
MSF
0.649
0,412
0.363
0.266
0.298
0.288
0.296
0.291
0.758
0.274
96
MAE
0.727
0.646
0.611
0.625
0.599
0.678
0.684
0.642
1,000
0.590
MSE
0.960
0.778
0.718
0.707
0.836
0,830
0.766
0.718
0.651
ETThl
192
MAE
MSE
0.754
0.995
0.772
0.643
0.683
0.877
0.674
0.611
0.640
0.807
0.601
0.625
0.637
0.726
0.670
0.603
0.699
0.920
0.610
0.581
384
MAE
0.757
0.645
0.626
0.662
0.623
0.623
0.607
0.605
0.868
0,594
MSE
1.012
0.781
0.687
0.791
0.765
0.648
0.664
0.673
0.702
0.642
MAE
0.784
0.656
0.802
0.665
0.656
0.698
0.621
0.625
0,873
0.635
MSE
L.455
0.792
1.061
0.787
0.810
0.848
0.701
0.726
1 15.503
0.721
MAE
0.384
0.417
0.678
0.530
0.393
0,394
0.389
0,444
0.350
96
MSE
0.314
0.305
0.353
0.749
0.530
0.348
0,336
0.332
0.401
0.276
Weather
192
MAE
0.397
0.362
0.684
D.433
0.362
0.350
0.347
MSE
0.340
0.290
0.363
0.764
0.404
0.294
0.274
0.275
0.360
0.244
384
MAE
0,428
0.354
0.386
0.691
0.626
0.359
0,344
0.338
0.427
0.342
MSE
0.379
0.282
0.316
0.789
0.663
0.291
0.268
0.265
0.365
0.264
MAE
0,445
0.359
0.392
0.699
0.605
0.359
0,348
0.336
0.417
0.332
768
MSE
0.402
0,286
0.337
0.818
0.638
0.290
0,270
0.260
0,351
0,250
96
MAE
0.686
0.502
0.433
0.447
0.519
0.457
0.455
0.459
0.630
0.447
MSE
1.232
0.750
0.727
0.924
0.875
1.082
MAE
0.681
0.542
0.430
0.398
0.540
0.389
0.392
0.410
0,542
0.387
Trffie
192
MSE
1.221
1.047
0.753
0.661
0.948
0.703
0,744
0.795
0.891
0.725
384
MAE
0.683
0.534
0.415
0.406
0.485
0.387
0.392
0.405
0.558
0.387
MSE
1,229
1.019
0.730
0.693
0.811
0.692
0,744
0.786
0.901
0.726
MAE
0.684
0.540
0.491
0.416
0.387
0.389
0.398
0,541
0.400
MSE
1,228
1.036
0.817
0.706
0.695
0.740
0.763
0.885
0.749
Table 23: Comparison of forecasting performance of S4M (Ours) and baselines on four datasets with various look-back window length under time point missing scenario when missing ratio 0.24.
S4
S4
S4
S4
S4M
Data
82
Metric
BRITS
GRU-D
Trans.
Auto.
BiTGraph
(Meun)
(Fill)
(Decay)
(SAITS)
(Ours)
MAE
0.673
0.481
0.422
0.441
0.436
0.556
0.501
0.460
0.556
0.418
96
MSE
0.698
0.437
0.344
0_363
0.367
0.570
0.479
0.409
0.570
0.366
Electricity
192
MAE
0.681
0.456
0.434
0.453
0,410
0.464
0.410
0.420
0.464
0.391
MSE
0.713
0.394
0.356
0.396
0.322
0.409
0.324
0.336
0.409
0.305
384
MAE
0.656
0.464
0.432
0.418
0.425
0.472
0.420
0.424
0.472
0.389
MSE
0.671
0.404
0.357
0.343
0.340
0.417
0.334
0.341
0.417
0,304
MAE
0.665
0.464
0.447
0.433
0.823
0.469
0,413
0.415
0.469
0.399
768
MSE
0.690
0.406
0.376
0.356
0.998
0.413
0.328
0.331
0.413
0.318
MAE
0.765
0.733
0.695
0.749
0.654
0.710
0.717
0.681
0.841
0.627
96
MSF
1.043
0.992
0.898
0.976
0.851
0.908
0,946
0.879
MAE
0.776
0.739
0.685
0.707
0.650
0.644
0.659
0.640
0.817
0.609
E11hl
192
MSF
1.047
1.004
0.893
0.856
0.815
0.739
0.792
0.782
1.076
0.703
384
MAE
0.772
0.738
0.702
0.712
0.677
0.632
0.648
0.648
0.814
0.628
MSE
1 1,058
1.001
0.917
0.870
0.908
0.710
0.768
0.779
1.059
0.710
MAE
0,800
0.744
0.793
0.702
0.630
0.639
0.661
0.672
0.801
0.632
MSE
.087
1.007
1.067
0.825
0.738
0.714
0.800
0.827
1.018
0.744
MAE
0.448
0.397
0.606
1.022
0.585
0.421
0.381
0.378
0.710
0.362
MSE
0,389
0.328
0.602
1.571
0.598
0.379
0,321
0.317
0.866
0.286
Weather
192
MAE
0.459
0.372
0.593
1.034
0.488
0.386
0.357
0.353
MSF
0.413
0.296
0.604
1.615
0.473
0.324
0,283
0.282
0.644
0.269
384
MAE
0.489
0.375
0.563
1.024
0.656
0.381
0,349
0.343
0.607
0.358
MSE
0.451
0.303
0.562
1.594
0.697
0.315
0.273
0.270
0.638
0.276
768
MAE
0.517
0.375
0.512
1.017
0.645
0.381
0.351
0.342
0.584
0.375
MSE
0.489
0.304
0.490
1.586
0.683
0.312
0.276
0.268
0.592
0,300
MAE
0.705
0.641
0.490
0.607
0.554
0.487
0,569
0.529
0.658
0.485
96
MSE
L.300
1.142
0.920
1.073
1.025
0.910
1.063
0.984
1.282
0.933
Taffic
192
MAE
MSE
0.695
.2667
0.617
1.110
0.512
0.950
0.472
8.904
0.949
0.533
0.826
0.442
0,870
0.480
0.452
0.812
0.539
1.014
0,433
0.787
384
MAE
0.698
0.623
0.487
0.466
0.541
0.431
0.456
0.440
0.547
0.433
MSE
1.133
0.896
808
0.952
0.795
0.842
0.809
1.031
0.788
MAE
0.700
0.628
0.509
0.463
0.432
0.449
0.434
0.560
0.429
768
MSE
1,270
1.158
0.872
0.798
0.799
0.823
0.789
1.030
0.789
25

---

Published as a conference paper at ICLR 2025
Table 24: Comparison of forecasting performance of S S4M (Ours) and baselines four datasets with
various look- back window length under variable missing scenario when missing ratio r = 0.24
$4
S4
$4
$4
S4M
Data
Metric
BRITS
GRU-D
Trans.
Auto.
BiTGraph
(Mean)
(Pill)
(Decay)
(SAITS)
(Ours)
MAE
0.647
0.497
0.424
0.421
0.436
0.407
0.431
0.430
0.621
0,4012
96
MSE
0.654
0.346
0.342
0.362
0.340
0,364
0.367
Electricity
192
MAF
0.649
0.454
0.421
0.412
0.425
0.382
0.391
0.401
0.575
0.373
MSE
0.659
0.388
0.348
0.326
0.341
0.299
0,301
0.316
0.281
MAE
0.652
0.482
0.424
0.416
0.446
0.383
0.392
0.413
0.573
0.377
MSE
0.667
0.434
0.347
0.335
0.358
0.299
0,298
0.329
0.557
0.290
MAE
0.654
0.509
0.469
0.411
0.415
0.380
0.398
0.410
0.569
0.383
MSE
0.672
0.473
0.413
0.320
0,320
0.293
0.311
0.324
0.549
0.298
MAE
0.757
0.654
0.712
0.637
0.708
0.728
0.671
0.622
MSE
1,016
0.874
0.742
0.875
0.807
0.916
0.959
0.847
1.161
0.766
ET1hI
192
MSE
0.768
1,025
0.871
0.775
0.836
0.867
0.776
0.873
0.765
0.637
1.022
0.654
384
MAE
0.774
0.681
0.630
0.691
0.661
0.648
0.657
0.669
0.753
0.630
MSE
1.061
0.879
0.708
0.806
0.868
0.767
0.785
0.843
0.961
0.713
MAE
0,798
0.692
0.746
0.687
0.660
0.665
0,682
0.677
0.750
0.682
768
MSE
1,072
0.895
1,004
0.782
0.868
0.808
0.842
0.852
0.955
0.829
MAE
0.430
0.396
0.529
0.544
0.584
0.442
0,386
0.384
0.544
0.370
96
0.373
0.595
0.318
Weather
MAE
0.454
0.385
0.514
0.505
0.490
0.385
0.355
0.356
0.505
0.355
MSE
0.405
0.309
0.484
0.468
0.473
0.324
0.272
0.276
0.468
0.270
384
MAE
0.485
0.376
0.479
0.506
0.655
0.385
0.351
0,348
0.506
0.359
MSE
0,443
0.300
0.436
0.469
0.693
0.320
0.269
0.269
0.278
MAE
0.492
0.379
0.461
0.494
0.640
0.384
0.356
0,345
0.494
0.377
MSE
0.459
0.305
0.418
0.583
0.674
0.317
D.273
0.264
0.447
0.301
0.699
0.464
0.547
0.462
0,507
0.524
0,473
MSF
1 .2666
1.074
0.891
0L778
0.998
0.850
0.977
0.969
1.245
0.896
MAE
0.689
0.427
0.546
0,412
Traffie
MSF
1.241
1.203
0.827
0.734
0.971
0.747
0.755
0.831
1.114
0.747
384
MAE
0.690
0.643
0.509
0.428
0.401
0,408
MSE
1.245
1.199
0.857
0.748
0.813
0.741
0.742
0.823
1.109
0.753
MAE
0.692
0.639
0.526
0.434
0.389
0,408
0.436
0.633
0.438
768
MSE
1.247
1.174
0.906
0.714
0.713
0.750
0.826
1.102
0.796
D.7 VISUALIZATION
To provide a clear comparison among different models, we list supplementary prediction showcases of Electricity dataset in Fig. 6. The results demonstrate that S 4M surpasses other methods in captur- ing future trends in the presence of missing observations.
D.8 REAL WORLD APPLICATION
For more general evaluation, also include the real world dataset, USHCN climate dataset (Menne et al., 2015), with 271728 time steps and 10 variables total. We set the lookback window &,. = 96 and horizon window εи 96. The results shown in Tab. 25 further confirm that S4M outperforms other methods on the real-world dataset.
Table 25: Comparison of forecasting performance of S 34M (Ours) and baselines USHCN climate dataset for fixed look-back length and horizon window under time point missing scenario when missing ratio 0.12.
Metric
BRITS
GRU-D
S4(Mean)
S4(Ffill)
S4(Decay)
S4M(Ours)
MAE
0.644
0.477
0.477
0.489
0.466
0.447
0.12
MSE
0.668
0.452
0.455
0.414
0.447
0.417
MAE
0.644
0.499
0.507
0.522
0.502
0.473
0.24
MSE
0.689
0.484
0.503
0.517
0.503
0.433
Metric|
Transformer
Autoformer
BiTGraph
iTransformer
CARD
MAE
0.461
0,511
0.474
0.478
0.451
MSE
0.406
0.499
0.439
0,460
0.411
MAE
0.475
0.534
0.495
0.504
0.477
0.24
MSE
0.403
0.530
0.469
0.502
0.444
26

---

Published as a conference paper at ICLR 2025
S4M (Ours)
94 (SAITS)
54 (Decay)
MAE: 0.372
MAE: D. 0.432
MAE: D. 40402
Promusted
(Mirsing
(Mioiigg
(Mioing)
210
54 (FTIII)
54 (Mean)
BITGraph
D.
080
060
A00
230
XC8
Autoforrerr
Transformer
GRU-D
390
rediimed Ground (Mhmangg Ground TUU?
Hedimedd
Ground Thith (Masning
Cround Butt (Venang) cround TUS?
Ground m2thh
300
112
680
MAE: 0.633
mOd cood
Grounn Trith (Ntesing)
Ground Thith
800
Figure 6: Visualization of S4M (Ours) and baselines with 6 96, l 96 on the Electricity dataset under time point missing scenario when missing ratio r = 0.06.
27


Reviews:
[{"review_id": "FI1O8vZMHD", "reviewer": "Reviewer_UFK6", "summary": "The paper presents S4M, an innovative end-to-end framework consisting of the Adaptive Temporal Prototype Mapper (ATPM) and the Missing-Aware Dual Stream S4 (MDS-S4) for multivariate time series forecasting that addresses the challenge of missing data.", "questions": "1. What are the computational costs and scalability of the S4M model, especially when dealing with large-scale multivariate time series data with high missing ratios? How does it compare to the baseline models in terms of training and inference time?\n2. How does the dual stream processing impact the model's ability to capture temporal dependencies?\n3. Can the authors confirm whether they implemented these baseline methods using official code or leveraged existing unified Python libraries, such as the Time-Series-Library [1] or PyPOTS [2]? It's important to note that data processing varies significantly among different imputation algorithms. Utilizing unified interfaces could help ensure that the experimental comparisons are conducted fairly. \n\n### References\n[1] https://github.com/thuml/Time-Series-Library\n\n[2] Wenjie Du. PyPOTS: a Python toolbox for data mining on Partially-Observed Time Series. In KDD MiLeTS Workshop, 2023. https://github.com/WenjieDu/PyPOTS", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The proposed S4M model is innovative, integrating missing data handling within the model architecture.", "weaknesses": "1. Although the results are promising, the authors did not provide their source code which results in very low reproducibility;\n2. Computational efficiency comparison is missing;", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tRbWZwkRmg", "reviewer": "Reviewer_iDLq", "summary": "The paper introduces S4M, an extension of the S4 framework to multivariate time series forecasting with missing values. It combines a prototype-based representation learning module (ATPM) with a dual-stream S4 architecture (MDS-S4) to handle missing values directly rather than through preprocessing. The method is evaluated on four datasets under various missing data scenarios.", "questions": "What is the computational complexity of ATPM vs traditional approaches?\nCan you provide theoretical justification for the prototype bank design?\nHow sensitive is performance to prototype bank initialization?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper addresses a practical and relevant problem. Real-world data often contains missing values (or mis-recorded values) which makes developing principled methods to handle them in forecasting models a worthwhile endeavor. The paper is well written and relatively easy to follow. The empirical evaluation does employ a set of strong baselines for comparison. The use of a \"prototype bank\" for backfilling is new to this reviewer and represents an interesting practical way to tackle missing values.", "weaknesses": "The paper introduces a few new key components, notably the prototype bank and the MDS-S4 architecture which, while well described, are only subjected to limited analysis and theoretical justification. For instance, complexity analysis is missing and ablation studies are partial. Some architectural choices seem arbitrary. The datasets chosen in the empirical evaluation (Traffic, Electricity, ETTh1, Weather) are all fairly simple datasets. Given that there are many more publicly available time-series evaluation datasets, I would like to see a more comprehensive evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YpjBaOmVaM", "reviewer": "Reviewer_iyXX", "summary": "In this paper, the authors propose S4M. S4M is an adaption of S4 which can handle missing values by:\n- Using prototype clusters, look-back information and an encoder to find representations also for time-points where values are missing\n- By explicitly also incorporating the masking matrix M into the S4-Layers.\n \n They evaluate S4M on the standard data for regular time-series forecasting when some data is hidden and show\n that i) S4M is competitive or often outperforms other S4 and Missing-Value approaches.", "questions": "Additionally to the critique points mentioned above, I have the following comments/question:\n\n- Have you tested S4M when there are no missing values at all? I would be curious whether your prototype bank is also useful if no values are missing.\n- Table 3: Do I understand correctly, that you are comparing: Prototyping Bank + Masking (i.e. having m_t in (4)) against only having the prototype bank? I would also like to see Only Masking, i.e. having the mask m_t in (4) but no prototype bank, i.e. replacing o_t with X_t. Is your prototype-bank really needed for irregular time-series forecasting?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "+ The idea of the prototype bank is very compelling and thoughtful, I like it a lot.\n+ The presentation is very good. The paper is written in a manner which makes it comfortable to follow.\n  Especially having an algorithm for each of the crucial parts helped me a lot.\n+ The results are not looking like fundamental break-throughs, but they are very promising for such a novel approach and there are a lot of ablations studies/hyperparameter experiments.", "weaknesses": "The two main weaknesses I identified:\n\n- My largest critique point is, that the authors are not comparing at all with recent results from the \"Irregular Sampled Time-Series wit Missing Values\" Literature. There is a plentitude of recent works solving irregular time-series forecasting in an end-to-end manner via ODEs, modelling latent dynamics or graph modelling.[1-5]. Furthermore, these papers provide a set of standard datasets for time-series forecasting with missing values, thus no need to synthetically make the normal regular datasets irregular.\n\n- There are a lot of standard methods missing, where one could do simply linear interpolation to use them in the experiments on which S4 is tested. For example, this work is not referring to important forecasting works like PatchTST or iTransformer at all. The results of S4M in Table 1, are way worse then the results of PatchTST (see Table at https://github.com/yuqinie98/PatchTST?tab=readme-ov-file) that it may be the case that PatchTST with 0.06 missing values indeed outperforms S4M.\n\n[1] De Brouwer, E., Simm, J., Arany, A., & Moreau, Y. (2019). GRU-ODE-Bayes: Continuous modeling of sporadically-observed time series. Advances in neural information processing systems, 32.\n\n[2] Yalavarthi, Vijaya Krishna, et al. \"GraFITi: Graphs for Forecasting Irregularly Sampled Time Series.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 15. 2024.\n\n[3] Schirmer, Mona, et al. \"Modeling irregular time series with continuous recurrent units.\" International conference on machine learning. PMLR, 2022.\n\n[4] Biloš, Marin, et al. \"Neural flows: Efficient alternative to neural ODEs.\" Advances in neural information processing systems 34 (2021): 21325-21337.\n\n[5] Klötergens, Christian, et al. \"Functional Latent Dynamics for Irregularly Sampled Time Series Forecasting.\" Joint European Conference on Machine Learning and Knowledge Discovery in Databases.\n\n## My Current Rating\nI do really like the idea and think that it has potential, even beyond S4 models for irregular time-series forecasting. However, for a top conference like ICLR, the amount of missing comparison to important related work is too high for recommending acceptance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "roDxECSeKZ", "reviewer": "Reviewer_EZj4", "summary": "This paper explicitly models missing patterns within the Structured State Space Sequence (S4) architecture, developing two key modules: the Adaptive Temporal Prototype Mapper and the Missing-Aware Dual Stream. The experiments demonstrate that S4M achieves state-of-the-art performance.", "questions": "1. The proposed model struggles when the input length is shorter than the output length. In addition, it is general to fix the lookback length and adapt to various prediction lengths [1]. Thus, it would strengthen this paper to add experimental results with various horizons.\n\n2. Baselines such as Transformer and Autoformer are designed for complete data. The authors should clarify how these models can be adapted for partially observed data. Besides, the experimental analysis should include state-of-the-art baselines specifically designed for long-term forecasting tasks, such as iTransformer [1], CARD [2], and Crossformer [3].\n\n[1]  Liu Y, Hu T, Zhang H, et al. itransformer: Inverted transformers are effective for time series forecasting[C]. The eleventh international conference on learning representations. 2023.\n\n[2]  Wang X, Zhou T, Wen Q, et al. CARD: Channel aligned robust blend transformer for time series forecasting[C]. The Twelfth International Conference on Learning Representations. 2024.\n\n[3]  Zhang Y, Yan J. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting[C]. The eleventh international conference on learning representations. 2023.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper explores the integration of State Space Models with missing patterns for long-term time series forecasting.\n\n2. The proposed adaptive temporal prototype mapper and missing aware dual stream S4 modules effectively capture rich historical patterns and learn robust representations.\n\n3. Experimental results illustrate that the proposed model achieves state-of-the-art performance in handling missing data.", "weaknesses": "1. The authors should further explain the motivation for introducing $\\mathbf{\\bar{E}}E_m(m_t;\\theta_m)$ to the SSM model in Equation 4.\n\n2. This paper lacks a discussion between the S4M and existing methods designed for handling missing values, which diminishes the significance of the proposed model.\n\n3. The settings of hyperparameters $K_1$, $K_2$, $\\tau_1$, and $\\tau_2$ are emprical. The authors should provide guidance on how to set these hyperparameters across different datasets with varying characteristics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)