Information Bottleneck-guided MLPs for Robust Spatial-temporal Forecasting
Authors: Min Chen
Guansong Pang
Wenjun Wang
Cheng Yan
Venue: ICML 2025 poster (2025)
Keywords: Robust spatial-temporal forecasting
Multi-Layer Perceptron
Information bottleneck
Primary area: social_aspects->robustness
TL;DR: Derive a novel and general information bottleneck based principle, along with its instantiation for robust spatial-temporal forecasting under dual noise effect in MLP networks.

Abstract: Spatial-temporal forecasting (STF) plays a pivotal role in urban planning and computing. Spatial-Temporal Graph Neural Networks (STGNNs) excel at modeling spatial-temporal dynamics, thus being robust against noise perturbations. However, they often suffer from relatively poor computational efficiency. Simplifying the architectures can improve efficiency but also weakens robustness with respect to noise interference. In this study, we investigate the problem: *can simple neural networks such as Multi-Layer Perceptrons (MLPs) achieve robust spatial-temporal forecasting while remaining efficient?* To this end, we first reveal the *dual noise effect* in spatial-temporal data and propose a theoretically grounded principle termed *Robust Spatial-Temporal Information Bottleneck* (RSTIB), which holds strong potential for improving model robustness. We then design an implementation named *RSTIB-MLP*, together with a new training regime incorporating a knowledge distillation module, to enhance the robustness of MLPs for STF while maintaining their efficiency. Comprehensive experiments demonstrate that *RSTIB-MLP* achieves an excellent trade-off between robustness and efficiency, outperforming state-of-the-art STGNNs and MLP-based models. Our code is publicly available at: [https://github.com/mchen644/RSTIB](https://github.com/mchen644/RSTIB).

Spatial-temporea forecasting (STF) holds great significance in modeling complex dynamic systems (Bai al., 2020; Guo et al., 2021a; Deng et al., 2021). Previous works in Information caalleneck-guided MLPs for Robust Spatial-temporal Forecasting
Min Chen 1
Guansong Pang"
Wenjun Wang'
Cheng Yan
Abstract
Flow Data in PEMS06 Dataset with No Noise
Spatial-temporal forecasting (STF) plays pivotal role in urban planning and computing. Spatial- Temporal Graph Neural Networks (STGNNs) ex- dynamics, thus being robust against noise perturbation. How- ever, they often suffer from relatively poor com- putational efficiency. Simplifying the architec- tures can speed up these methods but it also weakens the robustness w.r.t. noise interfer- ence. In this study, we aim to investigate the problem - can simple neural networks such as Layri-Layer Perceptrons (MLPs) achieve robust ceatial-temprall forecasting yet still be efficient? To this end, we first disclose the dual noise ef- fect behind the -empiall data noise, and propose theoretically-groune Robust Tempooal Information Bottleneck (RSTIB), which preserves wide potentials for en- hancing the robustness of different types of mod- els. We then meticulously design an implemen- tation, termed RST1B-MIP, along with new training regime incorporating a knowledge dis- tillation module, to enhance the robustness of MLPs for STF while maintaining its efficiency. Comprehensive experimental results show that an excellent trade-off between the robustness and the efficiency can be achieved by RSTIB- MLP compared state-of-the-art STGNNs and MLP models. Our code is publicly available http://gjitabbccommcccene4444119
Flow
cel in modeling
Sensor Sensor 10
02:0119 Q140 25 0100 03:9 01.01 01402 co 02.021 02 0LA00R
Flow Data in PEMSOS Dataset with Noise Ratio=40%
Flow
(b)
Sensor
Sensor 10
094119 01.1 28
01.0
03-01 22
00:43
01.42 01.02 03:02
03:22
Time
Figure 1. (a) Two time series are distinguishable at both historical input (P area) and forecasting target (F area), but (b) they become indistinguishable in both cases in the presence of noise.
principle termed
spatial- forecasting have effectively adopted convo- lutional neural networks (CNNs) (Lai et al., 2018), recurrent neural networks (RNNs) (Meng al., 2020), temporal con- volution networks (TCNs) (Wu et al., 2019) model spatial- temporal relations. Lately, there has been growing interest in spatial-tempont graph neural networks (STGNNs) (Shao et al., 2022b;e; Wu et al., 2019) due to their strong capacity. Though achieving exceptional performance, STGNN-based methods suffer from slow computational efficiency.
To alleviate this, few recent works (Shao al., 2022a; Qin et al., 2023; Wang al., 2023b; Yi et al., 2024) adopt Multi-Layer Perceptrons (MLPs) due to its advantagcous ef- ficiency. However, these simple yet efficient baselines lack effective architectures to combat against -emtial-emmpoal noise perturbation, which common in the real world (Jiang et al., 2023b; Tang et al., 2023; Fang et al., 2021; Liu et al., 2024c: Zhang al., 2023). Noise perturbation would result in sample- and feature-level harmful impacts, leading is- sues like sample indistinguishability (Shao et al., 2022a) and feature collapse respectively. As shown in Fig. 1, two time series may become indistinguishable in both the historical input end and the forecasting target end due to the presence of noise perturbation; besides, one can also observe severe feature collapse reflected by much lower feature variance,
1. Introduction
Eqqual contribution 'Tianjin University "Singapore Man- agement University. Correspondence to: Cheng Yan <yancheng wwork@tj..dd..c>>>
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Information Bottleneck-guided MLPs for Robust Spatial-temporal Forecasting
which is used for quantitative analysis of the diversity among the learned features (Papyan et al., 2020; Zhu et al., 2023a; Bardes et al., 2021) (see Section 3). Naturally, one may come across with the problem can simple neural networks such as MLPs achieve robust STF yet still be efficient?
that 1) RSTIB-MLP achieves better, or comparably good. ro- bustness, compared with state-of-the-an (SOTA) STGNNs, while being substantially more computationally efficient, and 2) RSTIB-MI ATTI-MMPP1 is much more robust than SOTA MLP. based models while being comparably efficient.
To investigate this problem, our focus lies on how one could effectively make MLPs robust, yet not suffer much effi- ciency loss. Methods based on adversarial training (Jiang et al., 2023b), graph information bottleneck (GIB) (Tang et al., 2023), mathematical tools (Choi et al., 2022), fre- quency domain MLPs (Yi et al., 2024), Biased TCN (Chen et al., 2024), Spatial-temporal Curriculum Dropout (Wang et al., 2023a) have been proposed to combat the noise for robust representation learning. However, they often rely on models/modules that can impair much on the efficiency, lacking a guiding principle targeting on spatial-temporal forecasting and theoretically grounded implementations on MLPs. To tackle this issue, we first reveal that spatial- temporal data often undergoes preprocessing through slid- ing window mechanism, where a sequence can serve as either the input or the target when residing in different win- dows, leading to a problem termed as "dual noise effect", i.e., noise potentially harms both ends. While Robust Graph In- formation Bottleneck (RGIB) (Zhou et al., 2023) effectively combats bilateral edge noise for link prediction, we reveal that generalizing RGIB directly to MLPs for STF is difficult: the GNN model architecture and graph data assumption it relies upon are very different from the spatial-tempol data that is continuous multivariate time series.
2. Related Work
Spatial-Tempont Forecasting (STF), Efforts in STF have led to the development of sophisticated models such as AGCRN (Bai et al., 2020), Graph WaveNet (Wu et al., 2019), and STExplainer (Tang et al., 2023), which lever- age STGNN-basee methodologies to model series-wise de- pendencies over time. Recent explorations have integrated Neural-ODE- (Jin et al., 2022) and self-supervised learning paradigms (Li et al., 2022) enhance spatial- temporal modeling. Despite their predictive capabilities, these methods often suffer from computational efficiency issues when compared with MLP-based approaches.
MLP-based Approaches for STE. In response to the effi- ciency challenge. MLP-based approaches have gained at- tention. Notable works include STID (Shao et al., 2022a), which incorporates spatial-temporal identity information to achieve superior performance over STGNN-basel meth- ods, and STHMLP (Qin et al., 2023), which employs a hierarchical MLP structure to capture various aspects of spatial-temporal data. FreTS (Yi et al., 2024) applies ML.s in the frequency domain. Specifically, its advantage of the energy compaction can help MLPs to preserve clearer pat- terns while filtering out inffuence of noises. However, these methods have not yet explored the dual noise effect in the face of comprehensive noise perturbations.
The main contributions of this paper are follows. (i) We introduce new theoretically sound principle, named Robust Information Botrleneck (RSTIB), general- izing the RGIB principle to mitigate the dual noise effect in spatial-temporal data. Particularly, it lifts the Markov as- sumption typically assumed in IB while not impairing the IB nature. In doing so, the derived additional noisy information and the original redundant information explicitly refor- mulated and minimized. This is general theoretical frame- work that has wide potentials for different types of models. (ii) We further instantiate RSTIB on simple networks MLP, modeling
Robust Representation Learning with Information Bot- tleneck Principle. The Information Bottleneck (IB) prin- ciple has emerged as a guiding framework for robust rep- resentation learning. Initially applied in Deep Variational Information Bottleneck (DVIB) (Alemi et al., 2017), IB has since found applications in diverse domains (Peng et al., 2018; 8; Higgins et al., 2016). Notably, GIB (Wu et al., 2020) extends IB to graph-structured data for supervised learn- ing. Subsequent advancements, such as STExplainer (Tang et al., 2023), build upon the GIB principle for explainable representations. While these methods can enhance robust- ness to some extent, they overlook the presence of noise in the forecasting target. RGIB (Zhou et al., 2023) takes step forward by decoupling mutual information to enhance such robustness, but generalizing it to MLP networks for spatial-temporal forecasting remains unexplored.
namely RSTIB-MLP. for robust while maintaining its efficiency. Subsequently, combined with the instantiation, we propose training regime to han- dle the dynamic relations between different time series via an innovative knowledge distillation module. The key idea is to balance the informative terms within the objective by accounting for the quantified noise impact, thereby being better balanced and less impacted by noisy information. We quantify the noise impact to each time series by defining a new noise impact indicator (Definition 4.9) and incorporate this knowledge for each time series. (iii) Comprehensive experiments on STF benchmark datasets from various do- mains under both noisy and clean evaluations demonstrate
3. Notations and Preliminaries
Spatial-temporal Forecasting (STF). STF aims at predict- ing the future target spatial-temporal data Y e R xxxXx
2

---

Information Bottleneck-guided MLPs for Robust Spatial-temporal Forecasting
framework designed to be more general for enhancing ro- bust spatial-temporal modeling. Following this, detail novel instantiation termed RSTIB-MLP, which leverages data reparameterization techniques for continuous multivari- ate time series. We also design a training regime, incorporat- ing a knowledge distillation module, to further enhance the performance of spatial-temporal forecasting. This approach capitalizes on the dynamic spatial-temporal relationships in- herent in the data, resulting in better balance of informative terms within the objective.
Table 1. Feature Variance of Single-End and Dual Noise Effect
Under Differrnt Noise Ratios
Noise Ratio
10%
30%
50%
70%
90%
Festure Variance (Single Knd)
T.9462
1.0325
0.8579
0.6523
0.6142
Fenture Variance (Dual Noise Effeet)
1,6900
0.6582
0,4859
0,4350
0,4332
with N time series of C' features in each time series within F nearest future time slots, based on historical input data X e PxXC from the past time slots. Additionally, we denote the sample from time series at time step l as X E RC and Y E IC for the historical and future data respectively.
4.1. Deriving the RSTIB Principle
Feature Variance. Drawing inspiration from prior stud- ies (Bardes et al., 2021; Zhu et al., 2023a), we aim for the learned representations in spatial-temporal forecasting to display significant diversity, capturing complex spatial- temporal patterns effectively. We quantify feature variance as follows:
X
Y
X
Y
Consider a set of latent spatial-temporal representations where each 2; E IRd for i = I,...,N. N. The feature variance defined as:
(z,, ;2, =,),
(a)
(b)
d
Figure 2. Comparison of IB(a) and DVIB with lifted Markov as- sumption - x Y(b). (a) (1) H(X information Z covers, i.e. [(x; Z Y) (2) the minimum sufficient information preserved by the expected optimal representation Z, i.e., 1(X; ) (b) By lifting& 2 X Y. () exists (3), i.e., | infor- mation Z covers.
Σ (VCovs)) = d 1=1
Var(z1, 22, IN)
(1)
where Cov;: denotes the variance of the i-th feature across the set of representations, defined as the diagonal elements of the covariance matrix Cov. Cov is computed as Cov N E>>(" (z: - (=; 2)²", with - = + > repre- senting the mean vector of the representations (see (Bardes et al., 2021; Zhu et al., 2023a) for detailed derivation and theoretical grounding). In Appendix 1.3, we link the feature variance with the spatial-temporal heterogeneity discussed in (Chen& & Liang. 2025).
Let X represents the input to the IB model and its variants, obtained from X and the attachment of spatial-temport information from a specially designed module. Formally, given the imput X, target Y. and encoding from X, the learning objective of the standard IB principle can be for- mulated follows:
Noise Perturbation vs. Feature Variance. We conduct an
min LIB - ((ZY) + Z),
(2)
empirical study to assess the impact of noise perturbation on feature variance, where the STID model (Shao et al., 2022a) used. During training, we inject random noise into the signals from the single input end and both ends, with a 50% probability across varying noise ratios- 10%, 30%, 50%, 70%, and 90%. The evaluation focuses on the diversity of extracted features by measuring the variance under these conditions. As shown in Table 1, a significant degradation in feature variance is observed with increasing noise pertur- bation. Besides, faster degradation can be observed when injecting to both ends, highlighting the detrimental effects of the noise and the dual noise effect on the effectiveness of capturing spatial-temporal patterns.
where I(:))) denotes mutual information (MI), and 3 2 0 is a Lagrange multiplier for controlling the trade-off be- tween the compression of X and the preservation of Y. The Markov chain Z - X - Y is assumed in IB (Alemi et al., 2017). We can use the information diagram (Fig.2) to de- pict the IB, where we represent information of X and Y as circles. Then IB encourages to cover as much of T(X; Y) and as little of H(|)Y) as possible.
However, the vanilla IB is sub-optimal in our scenario. By drawing inspirations from (Jiang et al., 2023b; Choi et al., 2022; Tang et al., 2024; Yuan et al., 2024; Liu et al., 2024b), we firstly have the following assumptions about spatial- temporal data:
4. Methodology
Assumption 4.1. Noisy Nature of spatial-temporal Data. We focus on spatial-tempores data that inherently exhibits noisy characteristics. Under the sliding window mecha-
In this section, we introduce the Robust Temporl-e Information Bottleneck (RSTIB) principle, a theoretical
3

---

Information Bottlencck-guided MLPs for Robust Spatial-temporal Forecasting
nism, a sequence can serve different purposes when re- siding in different windows, either the input or the tar- get. Consequently, the noise elements can potentially re- side in the input and target areas. For simplicity, we pre- sume the noise type in our analysis to adhere to Additive White Gaussian Noise (AWGN), a prevalent and empiri- cally approximated noise model in practical applications (Lim & Puthusserypady, 2007).
reformulations:
Proposition 4.3. Reformulate I(Z;Y | (X) and I [(X; Z Y). The sum of I X) and T(X; ZYY) can be refor- mulated as: I(Z; Y X) + I Z Y) = I(Z;X,Y) I(X;Y; 2). Proof. See Appendix F.I.
Leveraging this reformulation, we aim to minimize the in- fluence of noisy information Z captures, encapsulated by H and (X
Assumption 4.2. Invariant and Variant spatial-tempore Patterns. A dynamic spatial-temporal graph exhibits a dual nature, wherein each node, representing a time series, em- bodies both spatial-temporal invariant patterns conducive to generalized predictions across all time windows and spatial- temporal variant patterns reflecting underlying time-varying and node-specific dynamics.
Definition 4.4. Robust
Information Bottleneck Principle. Under the Markov chain condition
X 7 Y, the learning objective is encapsulated by the following optimization:
min LRSTIB = - 1(2,
(3)
Regarding Assumption 4.1, the sliding window mechanism a technique for processing spatial-temporal data, extract- ing fixed-length subsequences from raw data by progres- sively sliding window over temporal spatial-temporal dimensions. A critical feature is that the same data win- dow can flexibly serve as either input or target, creating a dual noise effect-when a noisy sequence serves as both the input (X) in one window and the target (Y) in another, noise propagates bidirectionally. If (Z- X - Y) holds, then [(Z; Y X)= = 0: the noisy information behind I(ZYY) X) is directly ignored. Ignoring noise in Y is therefore problematic. The dual noise effect allows noise influences both input and target across overlapping windows. Regarding Assumption 4.2, spatial-temporal graphs exhibit invariant patterns (generalizable across time) and variant patterns (node-specific, time-varying dynamics). Invariant patterns might represent structural dependencies (e.g., road connectivity in traffic prediction), whereas variant patterns could reflect transient events (e.g., traffic congestion due to accidents). Data dynamics thus also depend on the cur- rent window's 's characteristics, meaning the prediction for Y is not entirely determined by X, but also by Y's unique dynamics.
82 2 is the respective Lagrange multipliers to where ,1,33 control the balance within this objective.
4.2. Instantiating RSTIB
Here, we introduce the instantiation, termed RSTIB-MLP, in the order of [(X; Y; Z). I(Z and Y(Z,Y).
Instantiating I(X;Y; Z). Per definition (Definition B.11), the expression 2) = I(X;Y) - T(X;Y Z) in- dicates that maximizing Y;; 2) is equivalent to min- imizing 1 I(;Y 2), given I(X;Y) remains constant. It is important to note that the Markov chain condition X - 7 - Y is only approximated by reaching the opti- mal joint distribution of X, Z. Therefore, by explicitly minimizing I(X;Y Z), we aim to learn a sufficient Z while reaching objective simultancously. To this end, Z is initially encoded from X. Then, we aim to reduce the relative knowledge between X and Y by observing Z. To achieve this objective, we employ data reparameterizzatio to obtain the reparameterized X and Y while assuming inde- pendent and identically distributed (i.i.)) prior distributions of them, thereby reducing the overlapped information con- ditioned on X, Z, and Y. We effectuate the instantiation by directly imposing input regularization I(X; X)) and target regularization I I(;; Y). While mutual information terms are typically intractable, we introduce upper bounds for I [(;;) and I(;; Y), as elucidated in Proposition 4.5.
Following these assumptions, it is naive to assume the Markov assumption 7 - X - Y. which results in T(Z; Y X) 0. This implies that we directly overlook the noisy information behind H(|IX) (i.e., the noisy information conveyed by the target data). Fortunately, (Wieczorek & Roth, 2020) demonstrate that a lower bound of I(Z: Y) can be derived without relying on the Z - X Y assump- tion, thus lifting the Z- X Y Markov restriction. It is achieved in the DVIB model, which assumes X Z Y assumption by its construction. We apply it to our spe- cific scenario by assuming only the Markov chain condition X- --Z-Y Y (see Proposition 4.8). The introduced additional term, I(Z; Y X), as represented in 2 must be mini- mized as well, along with the original irrelevant information, ie., T(X; YY). Accordingly, we introduce the following
Proposition 4.5. The Upper Bounds of I(;; X) and I(;:)) Assuming the prior distribution of X and Y. denoted as Q(X) and Q(Y), to be iidd unit Gaussian (0, The upper bounds for XX;: and 1\Y; ,Y) are given by I(X; X) S E [KL (Po, (XXx)11((0) and T(::Y) T. KL. (MMI))))))))) where KL de- divergence, P and Pe, denote
notes the the parameterized distributions. Proof. See Appendix F22
According to Proposition 4.5, we first utilize simple MLP
4

---

Information Bottlencck-guided MLPs for Robust Spatial-temporal Forecasting
layers to parameterize the posterior distribution Po. (Z)X). This parameterization yields the posterior Gaussian distribu- tion of Z, represented as Po. ~ (Nz, o Subsequently, layers, one
(L = = ( o + 1 + 02
1). where Ls2 denotes the upper bound of representation regularization. Proof. See Appendix F.4.
we employ two additional mally-Connectedrfe for X and the other for Y, to facilitate dimension transfor- mation for aligning the dimensions of X and Y respectively. This process parameterizes two distributions, denoted as Po_ ~ N[ 2) and Po, ~ -Niyi oyy)) According to this, we establish Po. ~ (N,, o.)) and Poy ~ ~N(g, of) where Hxx + fix. My = y Fly o2 = oz² and og = oy respectively. Then, we adopt data reparameterization to obtain ã- = μr, + Tne and y = My + oye, where E and y represent the reparameterized signals, with each & E X and ý E Y respectively, and 1).
Instantiating I(Z;Y). Given the explicit reparameteriza- tion of Y to obtain Y. we aim to optimize [Z; Y) instead of ((; Y). However, directly computing ((2; Y) is also intractable. Therefore, we introduce Proposition 4.8 below to provide an approximated lower bound.
Proposition 4.8. The Lower Bound of I(Z; Y). The varia- tional lower bound of I(Z;Y) can be derived and approxi- mated by minimizing the typical regression loss while with- out being restricted to the Markov assumption Z - X - -Y, as follows:
Proposition 4.6. Analytical Solution for the Upper Bounds of the Input and Target Regularization. The Kullback- Leibler (KL) divergence between two Gaussian distri- butions, given their means and variances, can be an- alytically determined. Specifically, in our context, the KL. divergence is computed for the input and target respectively, as Lg_ = KL(N o²) (I,(,, 1)) (- loggo'' oh + + 00 fy (L 21 1). 2 (-log n -1), where Lu de- notes the upper bound of the input regularization, and Ey denotes the upper bound of the target regularization. Proof. See Appendix F.3.
I(Z;Y) N E Ep(x) B((2x)x(,, log Q(1IZ)
(4)
-reg Y),
where Ereg represents the regression loss and Y signifies the prediction. Proof. See Appendix F.5.
Specifically, predictions are made through a regression layer to obtain Y based on Z. We employ a standard regression loss, such as Mean Absolute Error (MAB), to maximize the variational lower bound.
Furthermore, findings from (Burgess et al., 2018) under- score the significance of the training regime concerning the 3 hyperparameter value for robust representation learning rather than adhering to fixed (-weighted term. In light of Assumption 4.2, it evident that conventional IB methods designed for static relations are not directly applicable to the domain of spatial-temporal forecasting, which inherently relies on dynamic relationships. Consequently, we adopt a novel approach by designing a training regime tailored for dynamic relations.
Instantiating T(Z; X, Y). According to the mutual infor- mation w.r.t three random variables (Definition B.8), we can express I XYY) = (Z) - (Z X.Y). Thus, our objective is to minimize the overlap of the entropy of Z with respect to X and Y. Given the condition of the Markov chain X 2 - Y. this can be implemented by reducing the entropy overlap between 2 and X through the use of data reparameterization. Specifically, reparameter- ized data X serves as the input to the MLP encoders. The encoders maintain the same network structure and param- eters as used in the instantiation of I (X;;;; Z). Then, the posterior distribution Po. (ZIX) is parameterized through the encoding process, as denoted Pa_ ()). The encoding= z = μ4= + a=E is obtained through reparameteriza- tion, where = E Z. Besides, we impose the representation regularization I [[X;; Z). and by assuming the prior distribu- tion of Z to be similar i.i.d unit Gaussian (0, 1), out goal can be reached. The analytical solution for the upper bound of T (X; Z) is also given in Proposition 4.7.
Training Regime. In our training regime, we adapt the regu- larization strategy (i.e., the balance of the informative terms within the objective) to accommodate the noise impact on different time series quantified in each time window. When noise impact is low, we relax the regularization. When there is a significant noise impact, we intensify the regularization. To quantify it, we leverage a trained model with no assump- tion on the model type and treat as the teacher. Then, the noise impact indicators, defined in Definition 4.9, are com- puted based the teacher model's predictive performance. By leveraging this knowledge, we dynamically balance the ASTIB-MLP'S robust representation learning in different time series within different time windows. The noise impact indicator is formally defined as follows:
Proposition 4.7. The Upper Bound of the Representation Regularization and Its Analytical Solution. The Upper bound of the representation regularization can be similarly given by T(x: 2) < E
}. [K KL Poo (7) { with
Definition 4.9. Noise Impact Indicator. Given the his- torical data X E TxXNC and a teacher model fr with trained and fixed parameters, we define the noise impac
Q(Z) being an i. unit Gaussian (0, 1). Specifically, we have the analytical solution for this upper bound: L.- =
5

---

Information Bottleneck-guided MLPs for Robust Spatial-temporal Forecasting
indicator to quantify the noise impact on each time series. It is calculated follows:
Besides, For Electricity dataset, we adopt the same training, validation, and testing split ratio as in (Deng al., 2021), and for other datasets, we adopt 6:2:2 for all datasets to ensure consistency.
exp (D (YF', Y:))
Σ N exp (D (YF x)))
Robust Baselines for Clean and Noisy Spatial-temporal
exp (D
Forecasting. (1) MLP-based Baseline: STID (Shao
E (J,...,N.
N).
j== exp (D (fr(A, YJ))
et al., 2022a); (2) STGNN-based Methods: GWN (Wu
(5) where A E N represents the adjacency matrix, uti-
et al., 2019) (3) IB-based Method: STGKD (Tang et al..
2024) (4) GIB-based Baselines: STExplainer (Tang et al., 2023) and STExplainer-CGiB (STExplainer with Conven- tional GIB); (5) Adversarial Training-based Method: TrendGCN (Jiang et al., 2023b) (6) Mathematical Tools- based Method: STG-NCDE (Choi et al., 2022). (7) Energy Compaction Enhanced Method: FreTS (Yi et al., 2024)
lized optionally depending on the modeling approach of the teacher. -)) denotes the distance function, such as mean squared error (MSE) or mean absolute error (MAE), to indi- cate the predictive performance. The computed é; for each time series reflects the relative impact of noise within the current time window, with higher values indicating greater susceptibility to noise.
(8) Biased TCN-based Method: BiTGraph (Chen et al.. 2024). (9) Spatial-temporal Curriculum Learning-basee
Method: STC-Dropout (Wang et al., 2023a).
Learning Framework. The final objective for the robust representation learning in RSTIB-MLP is formalized as follows:
Extra Baselines Designed for Clean Spatial-tempon Traffic Forecasting. We also dedicate to utilize PEMS datasets to compare RSTIB-MLP with three types of base- lines proposed for clean spatial-temporal traffic forecast- ing: (1) Attention-based Method: DSTAGNN (Lan et al.,
LNSTIB MLP
Leg (Y,S, 8:)]
2022); (2) MLP-based method: STHMLP (Qin et al.,
2023); (3) STGNN-based Methods: STGCN (Yu et al.,
+ Σ (1 + ás)) + Xyb + XeGra)
2018), AGCRN (Bai et al., 2020), GMSDR (Liu et al.,
2022), FOGS (Rao et al., 2022), and TrendGCN (Jiang
(6) The balance among all terms is controlled by the noise impact indicator â, and the Lagrange multipliers Ag. Ay and A₂ for input, target, and representation regularization, respectively. This learning objective highlights the relation- ship between RSTIB-MLP and the proposed training regime. The control over the balance of the informative terms is achieved not only by setting the hyperparameters, namely the Lagrange multipliers, but also by leveraging knowledge from noise impact indicators computed for different time series.
et al., 2023b);
Implementation Details. For the basic settings, we employ hidden dimension d 64 and utilize an MLP architecture with L=3 layers. For PEMS and TSSD) benchmark datasets, we use historical traffic flow data with window length P P= 12 to forecast future traffic flow data with win- dow length F == 12, while for the Electricity dataset, we follow the default settings in (Deng et al., 2021), i.e., we set P=16 and F=3, and calculate the average predictive ac- curacy by averaging over 2, 3 hours. Since there is no pre-defined graph structure in the Electricity dataset, some results are denoted as "_" meaning Not Available. The model performance is evaluated using three metrics: MAE, R.SE,, and MAPE. The learning rate is initialized as 17 = 0.002 with a decay factor r = 0.5. Baselines with recom- mended hyperparameter settings are used (See Appendix: D). Our method teacher model agnostic (Appendix.K) KepeendxxK.1100 10), where set the default teacher model to STGCN. Spatial- temporal prompts (Tang et al., 2024) are utilized to attach the spatial-temporal information.
5. Experiments
Datasets. For demonstrating universality, we consider six datasets from different domains, including PEMS04, PEMS07, PEMSOS (Fang et al., 2021; Guo et al., 2019; Song et al., 2020; Yu et al., 2018), LSDD)ST(SD) (Liu et al., 2024a), K-a (Zhu et al., 2023b), Elec- tricity (Deng et al., 2021). The diverse sample rates ensure the exploration of short-term, mid-term and long-term fore- casting evaluations. Detailed statistics and public accesses provided in Appendix.E. For PEMS and LSrgeST(SD) benchmark datasets, we choose the traffic flow (vehicles per hour) as the metric. For Weather2K-P dataset, We select vertical visibility from 20 meteorological factors as the experimental variable. For Electricity dataset, we select the average electricity consumption (Deng et al., 2021).
5.1. Main Results
Robustness Study. We evaluate the robustness of RSTIB- MLP by injecting noise into both the input and the target area, similar to the empirical study we conduct for evaluat- ing the harmful aspect of dual noise effect. As presented
6

---

Information Bottleneck-guided MLPs for Robust Spatial-temporal Forecasting
Table 2. Predictive Accuracy Comparison Under Various Noise Ratios in Different Datasets
Noise Ratio
0% (clea))
10%
30%
50%
Metrics
MAE
RMSE
MAPE
MAE
RMSE
MAPE
MAE
RMSE
MAPE
MAE
RMSE
MAPE
Dataset
PEMS04
STID
18.79
30.37
12.51%
27.83
41.34
17.31%
36.53
52.74
21.11%
36.22
52.15
21.45%
GWN
19.22
30.74
12.52%
30.03
43.27
19.32%
56.78
22.60%
40.87
55.13
23 23.02%
TrendGCN
18.81
30.68
12.25%
23.83
37.10
17.53%
27.35
43.10
19.32%
27.90
44.83
20.38
STExphiner-CCiB
19.14
30.77
12.91%
25.76
38.36
16.05%
31.72
48.51
17.98%
28.43
44.69
16.85%
STExplainer
18.57
30.14
12.13%
24.48
36.78
89%
31.39
47.18
18.05%
29.60
46.41
17.37%
STGKD
18.69
30.46
12.34%
24.35
37.06
16.31%
28.53
44.74
17.66%
29.24
46.28
18.63%
BiTGruph
18.82
30.44
12.25%
24.73
37.08
16.03%
31.65
47.52
18.20%
29.85
46.75
17.50%
STC-Dropout
18.75
30.38
12.33%
26.85
39.32
16.50%
34.15
51.22
20.54%%
33.74
50.37
19. 19.98%
STG-NCDE
19.21
31,09
12.76%
24.82
37.41
17.30%
29.24
44.17
19.44%
30.97
47.19
20. 20.86%
FreTS
18.77
30.45
12.25%
24.68
37.05
16.00%
31,60
47.45
18.17%
29,82
46.65
17.48%
RSTIB-MLP
18.46
30.14
12.221%
23.64
36.44
15.22%
27.15
42.85
17.19%
27.16
43,43
17.76%
STID
20.41
33.68
8.74%
27.99
45.02
12.37%
31.83
55.26
13.62%
32.38
57.29
14.07%
GWN
20.25
33.32
8.6.3%
28.25
45.47
12.51
32.15
55.81
13.76%
37.71
59.86
25. 11%
TrendGON
20.43
34.32
8.51%
44.65
14.59%
31.94
55.28
20.78%
36,78
57.89
23.22%
STExplaine--CGIB
20.55
35.12
8.61%
28.14
44.07
12.18%
34.92
57.60
14.22%
35.12
59.17
16.78%
STExplainer
20.00
33.45
5.50%
28.30
44,21
12.22%
31.58
54,03
14.82%
32.52
$7.64
15.48%
STGKD
20.30
34.30
8.87%
27.04
43.83
12.23%
55.16
13.69%
32.16
56.89
14.08%
BiTGraph
20.25
33.75
8.60%
28.55
44,55
12.35%
31.84
54.37
14.94%
32.77
58.00
15.60%
STC-Dropout
20.47
33.91
8.75%
28.63
44.67
12.41
31.72
$4.42
15.01%
32,91
58.13
15.691%
STG-NCDE
20.53
33.84
8.80%
28.79
44.62
14.22%
32.21
56.23
15.78%
33.48
58.83
16.78%
FreTS
19.92
33.65
8.70%
28.60
44.40
14.10%
32.05
56.00
15.65%
33.30
58.60
16.65%
RSTIB-MLP
19.84
33.90
8.33%
26.55
43.77
11.37%
30.15
54.08
12.65%
30.94
56.79
12,91
Dataset
PEMSO8
STID
14.87
23.97
10.43%
20,26
32.24
14.05%
45.73
15.63%
27.76
48 48.64
16.45%
GWN
14.67
23.49
9.52%
20.52
32.65
14.19%
26.91
46.19
15.78%
28.04
49.13
16.61%
TrendGCN
15.15
24.26
9.51%
20.81
32.49
14.92%
24.74
41.46
23.74%
26.
45.69
22 95%
STExplainer-CGIB
14.87
24.07
10.200 10.26%
23,66
35,49
24.34%
24.87
43.14
15.32%
26.50
44.62
15.54%
STExplainer
14.59
23,91
9.80%
20,28
32,86
13.17%
25.42
43.41
16.77%
27.17
45.79
15.26%
STGKD
15.13
24.80
10.66%
20.62
32.45
14.99%
25.63
43.29
16.03%
25.93
44.03
16.59%
BITGraph
14.85
24.20
9.90%
20.55
33.15
13.50%
25.70
43.75
16.90%
27.45
46.10
15.40%
STC-Dropont
14.70
24.32
9.75%
20.35
32.25
13.95%
25.55
44.82
15.25%
26.75
47.15
15%
STG-NCDE
15.45
24.81
9.92%
21.36
33.25
15.23%
28.35
41.89
16.33%
29.44
47.32
18.62%
FreTS
14.85
24.15
9.89%
20.52
33.12
13.45%
25.68
43.75
16.88%
27.41
46.05
15.35%
RSTIB-MLP
14.51
24.18
9.44%
19.90
31,86
12.92%
23.16
40.46
14.26%
24.37
43.77
14.36
Dataset
(SSg)
STID
17.60
29.05
11.92%
26.53
40.35
16.91
34,82
$4,03
20.62%
35,21
55.26
21.52%
GWN
17.74
29.62
11.88%
27.39
40.95
17.81%
32.87
55.64
18.84%
37.32
23.23%
TrenGGCN
17.39
29.63
11.64%
25.84
39.64
16.23%
31.45
51.71
17.83%
33.63
52.18
18.85%
STExplaine--CGIB
18.60
30.29
12.69%
26.17
40.46
17.55%
32.11
$3.39
18.37%
34.56
$2.8%
19.43%
STExplainer
17.51
11.57%
25.68
39.48
16.24%
31.41
51.49
17.87%
33.39
51.96
18 80%
STGKD
17.60
29.42
11.62%
25.85
39.71
16.08%
31.52
51.37
17.67%
33.93
52.67
18.97%
BiTGruph
18.85
29.80
12.68%
25.88
39.34
16.23%
31.16
$2.13
17.74%
33.74
$1.98
18.86/%
STC-Dropout
17.55
29.36
11.68%
25.78
39.64
16.14%
31.48
$1.42
17.73%
33.87
52.73
18.92%
STG-NCDE
17.58
29,14
11.87%
26.24
40.39
16.52%
31.83
52.67
17.86%
33.76
52.23
18.97%
FreTS
17.54
29.01
11.98%
26.16
40.21
16.53%
31.60
$2.99
17.71%
34.09
52,51
18.8615
RSTIB-MLP
17.50
28.75
11.20%
25.02
38.37
15.42%
30.60
50.52
16.85%
32.78
50.38
17.92%
Weather2K-R
STID
3997.92
6199.77
65.34%
4950.47
6610.89
67.00%
6301,76
1071..43
76.34%
7654.47
9660011
82.08%
GWN
3991.24
6207.50
66.00%
5218.42 42
6896.87
66.72%
68%3.U7
8681.24
74.27%
8324 2224.23
9K32.49
83.08%
TrendGCN
3987.92
6223.53
30%
4589.00 04
6274.23
63.31%
5982.71
7688. 16
72.99%
7108 36
8964.54
81.53%
STExplaine--CGIB
3994.82
6200.83
65.35
4789.03
6540.73
67.12%
6215.86
7985.24
75.72%
7775.50 50
98 18.12.4
81.61%
STExplainer
3992.57
6198.33
65. 65.22%
4786.53
6537.73
67.18%
6213.36
7982.24
75.58%
7773.00
9809.44
81.47%
STGKD
3990.07
6195.83
08%
4784.08
6534.73
67.02%
6210.80
7979924
75.44%
7770.50
9805.94
81.33%
BITGraph
3989.32
6216.03
65.21%
4588.67
6274.45
63.15%
1,81,76
7680.34
72.91%
7103.92
8964487
81.38%
STC-Dropuut
3986.43
6205.21
65.35%
4792.87
6543.11
67.25%
6205.49
7975.08
75.45%
7782.56
98177.33
81.60%
STG-NCDE
3992.57
6199.03
65.22%
4787.5
6538.53
65.15%
6214.36
7983.04
75.58%
7774.00
9810.24
81.46%
FreTS
3984,37
6219.03
65. 12%
4585.09 09
0269.53
63.13%
5978.28
7683.56
72.81%
7104.06
8959.99 9999994
81.35%
RSTIB-MLP
3964.53
6191.08
64.94%
4561.97
6239.33
62.96 %
5948.52
7645.33
72.64%
7073.69
8914.37
81.17%
Dataset
Electricity
28.23%
45.78%
GWN
TrendGCN
19.98
39.62
15.72%
25.23
46.48
20.37%
34.35
63.82
26.78%
47.26
78.65
42.38
STEsplaine--GGI
STExplainer
STGKD
20.15
40.05
15.89%
25.40
46.75
20.65%
34.90
64.90
27.65%
48.75
79.80
44,90%5
BITGraph
19.98
29.87
16.12%
25.52
47.10
21.05%
35.68
65.82
28.55%
49.78
78.95
43.78%
STC-Dropout
19.92
39.85
16.47%
26.12
48.25
21.69%
36.68
68.32
30.27%
49.30
78.03
42.78%
STG-NCDE
19.85
39.92
16.52%
26.05
48.12
21.78%
36.75
68.45
30%1
49,23
77.90
42.85%
FreIS
20.12
40.45
16.22%
26.15
47.88
21.95%
37.30
69.25
31.05%
52.98
78.75
43.78%
RSTIB-MLP
19.80
39.67
15.72%
24.50
45.85
19.95%
33,80
62.50
25.85%
45.30
74.60
40.75%
7

---

Information Bottleneck-guided MLPs for Robust Spatial-temporal Forecasting
ance Perspective: A Case Study. In this case study, we examine the superiority of our method from the perspective of feature variance, a crucial aspect for effective model eval- uation. As discussed in (Bardes et al., 2021), maintaining feature diversity is essential to mitigate feature collapse and enhance model robustness. The quantitative case findings in Fig.3 indicate that our proposed knowledge distillation module significantly boosts feature variance, a critical factor in capturing the intricate and dynamic spatial-temporal pat- terns. This observation underscores the effectiveness of ac- counting for the noise impact on different time series when balancing the informative terms in the learning objective, which is achieved by incorporating knowledge distillation into the training regime. We also provide a model inter- pretation case study to visualize the distribution of learned representation in Appendix. K.8.
PEMS04
2,0
STID VAR
STGKD VAR
1.8
Irol's VAR
STG-NCDE VAR
reature Variance
1.6
STC-Dropoul VAR
1.4
STExplainer VAR
STExpiaine-CGIB VAR
TrandGiA VAR
BiTGraph VAR
1.0
AESTIB-MLP.. www.oooo....... VAR
RSTIB MLP VAR
0.6
0.1
0.3 Noise Ratio
0.5
Figure 3. Feature Variance (Var) of Different Methods w.r.t differ- ent noise ratio (7) in PEMS04 Dataset
WD-ED
IST.DAG
5.2. Ablation Study
24
We assess our proposed components within RSTIB-MLL through its various variants: i) "w/o IB & KD": Excludes any Information Bottleneck (IB)-based enhancement and the knowledge distillation module. ii) "w/o RSTIB & KD": Similar to the first variant but implements vanilla IB. iii) "w/o KD": Removes the knowledge distillation mod- ule while instantiating the RSTIB principle. The ablation study, as shown in Fig.4, reveals significant performance degradation without IB instantiation most scenarios, em- phasizing its role in mitigating the detrimental effects of spatial-temporal data noise. However, the results also show some cireumstances where implementing the vanilla IB prin- ciple results in even worse performance. The potential rea- son is that vanilla IB has not considered noisy information conveyed by the target, while also challenging to balance the informative terms within its objective. Besides, instantiating RSTIB can make significant performance improvements compared with vanilla IB instantiation non-IB enhanced instantiation. This underscores the importance of minimiz- ing the noisy information conveyed by both input and target data ends in the ppatio-temporal forecasting scenario. Fur- thermore, the knowledge distillation module contributes to performance enhancement, due the better balance of the informative terms during robust representation learning.
Roias Wesmerkk
Ri on
17%
10%
Figure 4. Ablation Study Results on Different Benchmark Datasets When Combating Noises with Different Noise Ratios
in Table 2, the results demonstrate that RSTIB-MLP can achieve better or comparably good predictive performance when combating noise perturbation, even when comparing with robust baselines with dedicate model design. This finding underscores the robustness of RSTIB-MLP, which can be attributed to its consideration of both noisy input and target information conveyed with theoretical guarantee. In contrast, previous approaches typically consider only a single noisy area. Our method's ability to handle both noisy patterns contributes to its enhanced robustness.
Learning with Clean Data. In this analysis, we investi- gate the behavior of RSTIB-MLP when learning with clean data. As depicted in Table 2 and Table 10 in Appendix K. we observe some improvements in performance metrics on clean datasets, although not significant. It is noteworthy that while our primary objective is to enhance robustness on noisy datasets, the observed slight improvement on clean datasets suggests potential benefits. However, we hypoth- esize that the marginal improvement could stem from the challenge of effectively balancing the informative terms within the learning objective, thus impacting the overall performance.
5.3. Computational Efficiency
This section compares the efficiency of IRSTI--MLL with some representative state-of-the-art STGNN-based meth- ods. We also include an MLP-based method, STID (Shao et al., 2022a), as an ML.P-asse baseline. We measure the efficiency by recording the average training time per epoch of all methods on the PEMS04 dataset. All evaluations are conducted on an NVIDIA RTX 3090Ti GPU. Fig.5 displays the results. We can see that prior STGNN- based works re- quire more time due to the sophisticated model design (See
Inspecting Representation I Learning from a Feature Vari-
8

---

Information Bottlencck-guided MLPs for Robust Spatial-temporal Forecasting
Fund (No. 24-S1S-SMU-008), and the Lee Kong Chian Fellowship.
Efficiency Study: Seconds/Epoch MAE
19.2
Model Type
Impact Statement
Model Type
This paper presents work whose goal is to advance the field of machine learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
19.0
RSTIB-MLP
MAA
STID
18.8
GWN
DSTAGNN
18.6
STG-NCDE
References
STExplainer
Alemi, A. A., Fischer, I., Dillon, J. V., and Murphy, K. Deep variational information bottleneck. In Proceedings of the 5th International Conference Learning Representa- tions, pp. 1-19, 2017.
50
100
150
Seconds/Eooch
Figure 5. Computational Efficiency Results on PEMS04
Bai, L., Yao, L., Li, C., Wang, X., and Wang, C. Adaptive graph convolutional recurrent network for traffic forecast- ing. Advances in neural information processing systems, 33: 17804-4-778, 2020.
Section H). By contrast, work utilizes computationally more efficient MLP networks, resulting in more stream- lined model architecture, allowing faster processing and shorter training time. However, the RSTIB-MLP implemen- tation reduces the efficiency compared to STID. Therefore, our method is inherently trade-off. Combining the results from Table this trade-off proves be well-balanced.
Bardes, A., Ponce, J., and LeCun, Y. Viereg: Variance- regularization for self-superrised learning. In Proceedings of the 9th International Confer- ence Learning Representations, pp. 1-23, 2021.
Burgess, C. P., Higgins, I., Pal, A., Matthey, L.,, Watters, N., Desjardins, G., and Lerchner, A. Understanding disentan- gling in beta-vae. In Proceedings of the 32th International Conference on Neural Information Processing Systems Workshop, pp. 1-11, 2018.
6. Conclusion
The problem we aim to investigate this study is: can sim- ple neural networks achieve robust apatial-tepporal fore- casting yet still be efficient? To this end, we first propose the Robust Tempoolll Information Bottleneck (RSTIB) principle for guiding robust representation learning. We then instantiate the computationally efficient and robust RSTIB- MLP using pure MLP networks. Additionally, we incor- porate knowledge distillation module into our training regime. Knowledge distillation can enhance feature diver- sity and improve predictive accuracy by better leveraging the knowledge from previously trained teacher models bal- ance informative terms within the objective of RSTIB-MIPP Through comprehensive evaluation encompassing feature variance and predictive performance metrics, our approach demonstrates a superior trade-off between efficiency and robustness. achieves robust forecasting accuracy under challenging conditions while still being computationally more efficient than state-of-the-ae tfrt STGNN-baseee methods.
Chen, W. and Liang, Expand and compress: Explor- ing tuning principles for continual spatio-temporal graph forecasting. In The Thirteenth International Conference Learning Representations, 2025.
Chen, X., Li, X., Liu, B., and Li, Z. Biased temporal convolution graph network for time series forecasting with missing values. In International Conference on Learning Representations (ICLR), 2024.
Cheng, H., Wen, Q., Liu, Y., and Sun, L. Robustisf: To- wards theory and design of robust time series forecasting with anomalies. Proceedings of the 12th International Conference on Learning Representations, pp. 1-27, 2024.
Choi, J., Choi, H., Hwang, J., and Park, N. Graph neural controlled differential equations for traffic forecasting. In Proceedings of the AAAI Conference Artificial Intelli- gence, volume 36, pp. 6367-6374, 2022.
Acknowledgements
Deng, J., Chen, X., Jiang, R., Song, X., and Tsang, I. W. St-norm: Spatial and temporal normalization for multi- variate time series forecasting. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pp. 269-278, 2021.
This research is supported by the National Key R&D Pro- gram of China NN.20233FF3880522 MTC YIRG Grant (No. M24N8c0103), the Ministry of Education, Singapore under its Tier-1 Academic Research
A*STAR under its

---

Information Bottleneck-guided MLPs for Robust Spatial-temporal Forecasting
Fang, Z.,, Long, Q. Song, G., and Xie. K. Spatial-temporal graph ode networks for traffic flow forecasting. In Pro- ceedings of the 27th ACM SIGKDD conference knowl- edge discovery & data mining, pp. 364-373 2021.
network for traffic flow forecasting. In International con- ference on machine learning, pp. 11906-11917. PMLR, 2022.
Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
Guo, K., Hu. Y. Sun, Y., Qian, S., Gao, J., and Yin, B. Hier- archical graph convolution network for traffic forecasting. In Proceedings of the AAAI conference on artificial intel- ligence, volume 35, pp. 151-159, 2021a.
Li, Z., Huang, C., Xia, L.,, Xu, Y., and Pei, J. Spatial- temporal hypergraph self-supervised learning for crime prediction. In 2022 IEEE 38th International Confer- ence on Data Engineering (ICDE), pp. 2984 2996. IEEE, 2022.
Guo, S., Lin, Y., Feng, N., Song, C., and Wan, H. Attention based spatial-temports graph convolutional networks for traffic flow forecasting. In Proceedings of the AAAI con- Serence on artificial intelligence, volume 33, pp. 922-929, 2019.
Lim, T. P. and Puthusserypady, S. Chaotic time series pre- diction and additive white gaussian noise. Physics letters A, 365(4):309-314 2007.
Guo, S., Lin, Y., Wan, H., X.,, and Cong, G. Learning dy- namics and heterogeneity of spatial-temporal graph data for traffic forecasting. IEEE Transactions on Knowledge and Data Engineering. 34(11):54155544 5428,, 2021b.
Liu, D., Wang, J., Shang, S., and Han, P. Msdr: Multi- step dependency relation networks for spatial temporal forecasting, In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1042-1050, 2022.
Han, L., Du B., Sun, L., Fu, Y., Lv, Y., and Xiong, H. Dynamic and multi-faceted spatio-temporal deep learning for traffic speed forecasting. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining, pp. 547-555, 2021.
Liu, H., Dong, Z., Jiang, R., Deng, J., Deng, J., Chen, Q. and Song. X. Spatio-temporal adaptive embedding makes vanilla transformer sota for traffic forecasting. In Proceedings of the 32nd ACM International Conference Information and Knowledge Management, pp. 4125- 4129, 2023.
Higgins, I., Matthey, I.., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., and Lerchner, A. beta- vae: Learning basic visual concepts with constrained variational framework. In International conference learning representations, 2016.
Liu, X., Xia, Y., Liang, Y., Hu, J., Wang, Y., Bai, L.,, Huang, C., Liu, Z., Hooi, B., and Zimmermann, R. Largest: A benchmark dataset for large-scale traffic forecasting. Advances in Neural Information Processing Systems, 36, 2024a.
Jiang, J., Han, C., Zhao, W. X., and Wang, J. Pdformer: Propagation delay-aware dynamic long-range transformer for traffic flow prediction. In AAAI AAAI Press, 2023a.
Jangg J., Wu, B., Chen, L., Zhang, K., and Kim, S. En- hancing the robustness via adversarial learning and joint spatial-temporal embeddings in traffic forecasting. In Pro- ceedings of the 32nd ACM International Conference on Information and Knowledge Management, pp. 987-996, 2023b.
Liu, Y., Li, C., Wang. J., and Long, M. Koopa: Learning non- stationary time series dynamics with koopman predictors. Advances in Neural Information Processing Systems, 36, 2024b.
Jin, M., Zheng, Y., Li, Y.-F., Chen, S., Yang, B., and Pan, S. Multivariate time series forecasting with dynamic graph neural odes. IEEE Transactions on Knowledge and Data Engineering, 2022.
Liu, Y., Rasouli, S., Wong. M., Feng, T., and Huang, T. RI- gen: Gaussian based spatiotemporal graph convolutional network for robust traffic prediction. Information Fusion, 102: 2024c.
Lai, G., Chang, W.-C., Yang, Y., and Liu, H. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st international ACM SIGIR confer- ence on research & development in information retrieval, pp. 95-104, 2018
Meng. X.,, Fu, H., Peng, L., Liu, G., Yu, Y., Wang, Z., and Chen, E. D-Istm: Short-term road traffic speed prediction model based on gps positioning data. IEFE Transactions on Intelligent Transportation Systems, 2303):2021- 2020.
Lan, S., Ma, Y., Huang, W., Wang, W., Yang, H., and Li, P. Dstagnn: Dynamic
Palomar, D. P. and Verdú, S. Lautum information. JEEE transactions on information theory, 54(3):964 975,,, 2008
aware graph neural
10

---

Information Bottleneck-guided MLPs for Robust Spatial-temporal Forecasting
Papyan, V., Han, X., and Donoho, D. L. Prevalence of neural collapse during the terminal phase of deep learn- ing training. Proceedings of the National Academy of 2020.
Tishby, N., Pereira, F. C., and Bialek, W. The informa- tion bottleneck method. arXiv preprint physics/0004057, 2000.
Sciences, 117(40):246522444)
Wang. H., Chen, J., Pan, T., Fan, Z.,, Zhang. B., Jiang, R., Zhang. L., Xie, Y., Wang, Z., and Song, X. Easy begun is half done: Spatial-temporal graph modeling with st- curriculum dropout. 2023a.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L.., et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Wang. Z.., Nie, Y., Sun, P., Nguyen, N. H., Mulvey, J., and Poor, H. V. St-mlp: A cascaded spatio-temporal linear framework with channel-independence strategy for traffic forecasting. arXiv preprint arXiv:2308.07450 2023b.
Peng. X. B., Kanazawa, A., Toyer, S., Abbeel, P., and Levine, S. Variational discriminator bottleneck: Improv- ing imitation learning, inverse rl, and gans by constrain- ing information flow. arXiv preprint arKiv:1810.003: 2018.
Wieczorek, A. and Roth, V. On the difference between the information bottleneck and the deep information bottle- neck. Entropy, 22(2):131, 2020.
Qin, Y., Luo, H., Zhao, F., Fang, Y., Tao, X., and Wang, C. Spatio-temporal hierarchical mlp network for traffic forecasting. Information Sciences, 632:543 554,, 2023.
Wu, T., Ren, H., Li, P., and Leskovec, J. Graph information bottleneck. Advances in Neural Information Processing Systems, 33:20437-20448, 2020.
Rao, X., Wang, H.. Zhang, L., Li. J., Shang, S., and Han, P. Fogs: First-order gradient supervision with learning- based graph for traffic flow forecasting. In Proceedings of International Joint Conference on Artificial Intelligence, IJCAI. ijeai. org, 2022.
Wu, Z., Pan, S., Long, G., Jiang, J., and Zhang, C. Graph wavenet for deep spatial-temporal graph modeling. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pp. 1907-1913, 2019.
Shao, Z., Zhang, Z., Wang, F., Wei, W., and Xu, Y. Spatial- temporal identity: simple yet effective baseline for multivariate time series forecasting. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pp. 4454-4458, 2022a.
Yi, K., Zhang, Q., Fan, W., Wang, S., Wang, P., He, H., An, N., Lian, D., Cao, L., and Niu, Z. Frequency-domee mips are more effective learners in time series forecasting. Advances in Neural Information Processing Systems, 36, 2024.
Shao, Z., Zhang, Z.. Wang, F., and Xu, Y. Pre-training enhanced spatial-temporal graph neural network for mul- tivariate time series forecasting. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1567-1577, 2022b.
Yu, B., Yin, H., and Zhu, Z. Spatio-tempores graph convo- lutional networks: A deep learning framework for traffic forecasting. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pp. 3634-3640 2018
Shao, Z., Zhang, Z., Wei, W., Wang. F., Xu, Y., Cao, X., and Jensen, C. S. Decoupled dynamic spatial-temporal graph neural network for traffic forecasting. Proceedings of the VLDB Endowment, 15(1 1):2733-2745 2022c.
Yuan, H., Sun,Q.. Fu, X., Zhang, Z., Ji, C., Peng, H., and Li, J. Environment-aww aware dynamic graph learning for distt--iitttitttion generalization. Advances in Neural Information Processing Systems, 36. 2024.
Song, C., Lin, Y., Guo, S., and Wan, H. Spatial-temporal synchronous graph convolutional networks: A new frame- work for spatial-temporal network data forecasting. In Proceedings of the AAAI conference on artificial intelli- gence, volume 34, pp. 914-921, 2020.
Zhang, Q., Huang, C., Xia, L., Wang, Z., Yiu, S. M., and
Han, R. Spatial-temporaa graph learning with adversarial contrastive adaptation. In International Conference on Machine Learning, pp. 41151-41163, PMLR, 2023.
Zhou, K., Wang, W., Hu, T., and Deng, K. Time series fore- casting and classification models based on recurrent with attention mechanism and generative adversarial networks. Sensors, 20(24):7211, 2020.
Tang, J., Xia, L., and Huang, C. Explainable spatio-temporal graph neural networks. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pp. 2432-2441, 2023.
Zhou, Z., Yao, J., Liu, J., Guo, X., Yao, Q., He, L., Wang, L., Zheng, B., and Han, B. Combating bilateral edge noise for robust link prediction. In Advances in Neural Information Processing Systems, 2023.
Tang, J., Wei, W., Xia, L.,, and Huang, C. Spatio-temporal graph knowledge distillation, 2024. URL https:// openrevi www.ieew.eeeeeeeeeeeeeeeeeeeeeee formm//rummmm d=iiddakkkiii KKNGWerr.

---

Information Bottleneck-guided MLPs for Robust Spatial-temporal Forecasting
Zhu, J., Shwartz-Ziv, R., Chen, Y., and LeCun, Y. Variance- covariance regularization improves representation learn- ing. arXiv preprint 2tt.::333..12222 2023a.
Zhu, X., Xiong, Y., Wu, M., Nie, G., Zhang. B., and Yang, Z. Weather2k: A multivariate spatio-temporal benchmark dataset for meteorological forecasting based real-time observation data from ground weather stations. Interna- tional Conference on Artificial Intelligence and Statistics, (C):2704-2722, 2023b.
12

---

Information Bottlencck-guided MLPs for Robust Spatial-temports Forecasting
Appendix
A. Notations
Our notations are elaborated in Table 3.
B. Additional Preliminaries
B.1. Mathematical Preliminaries and Definitions
This section provides mathematical preliminaries concerning entropy and mutual information using three discrete random variables X, Y, and Z for illustrative purposes. It is important to note that these variables do not carry specific meanings within this context and the notations used here are distinct from those in the main discussion. Additionally, this section offers an intuitive understanding of each term.
Definition B.1. Entropy. We define the entropy (X) of a discrete random variable X as a measure of its uncertainty, using its marginal distribution p(x). Mathematicallyy entropy is expressed as:
H(X) = E
- p(x) log p(x),
zEX
where the summation extends over all possible outcomes of the random variable X. The function (X) quantifies the expected information content or uncertainty inherent in X's outcomes.
Definition B.2. Joint Entropy. The entropy of two random variables X and Y can be jointly considered by viewing them as components of single vector-valued random variable. This joint entropy is defined as:
H(X,Y)=
Σ
p(a, v) logp(r;
TEX,EEY
E. #EX BEY
p(a, v) log
(8)
where p(z, y) represents the joint probability distribution of X and Y. This definition encapsulates the total uncertainty present when considering the distribution of both variables simultaneously.
Definition B.3. Conditional Entropy. Given two discrete random variables X and Y, the conditional entropy of X given Y is defined as:
H(X = Σ p(v) E((l))) (ogl)))
VEY
=EX
where p(#ly) is the conditional probability of X given Y, and p(v) the marginal distribution of Y. value of H(X(Y) = 0 implies that knowing completely determines X, signifying no remaining uncertainty about X once Y observed.
This concept allows us to understand H (X) tropy-reflecting the uncertainty in X after Y is known. The reduction in entropy, H(X) H(XJY), quantifies the amount of information Y provides about X, which is formally termed mutual information in Definition B.S.
priori entropy of X, while HXX)Y) represents posteriori en-
Remark B.4. Conditional Entropy w.r. three variables. The conditional entropy H HZ quantifies the residual uncertainty in random variable Z when the values of other variables X and Y are known. It is mathematically defined as:
H(ZIX,Y)=
Σ p(a, v)
Σ 2EZ
(10)
TEXWEYY
Here, p(z, y) represents the joint distribution X and Y, and p(=|z, y) is the conditional probability of Z given that X and Y take the values and respectively. This measure effectively describes how much uncertainty in remains after observing both X and Y.
13

---

Information Bottlencck-guided MLPs for Robust Spatial-temporal Forecasting
Table Notations
Symbol
Description
N
The number of time series(i.e, nodes).
P
The length of historical input.
F
The length of forecasting target.
C
The number of features in each input target time series at specific time slot. The historical spatial-temporal data X E RPNxx with N time series of C' features in each time series within P nearest historical time slots, with each X E R The input to RSTIB-MLP model, generated from X². The dimension of X, along with Xt,5. depends on X,, and the attachment apatial-tempore information from specially designed module.
X²
X
Y
The forecasting target data Y FXNxC with time series of features in each time series within F nearest future time slots, with each Y E RCC
X
The reparameterized imput.
The reparameterized target.
yr
The teacher model's 's output.
ys
The RSTIB-MIP model's output.
d
The hidden dimension of each i... E Rd for i 1,...,
Z
The encoded spatial-temporal representation, comprised of series of latent spatial- temporal representations (21, 22...... 2N where I2 for i I....,N.. N. Z E Nxd
Cov
The covariance matrix of Z, with each Cov; representing the variance of the i-th feature across the representations, Lee, the diagonal elements of Cov.
Var
The feature variance defined in (q.
D
The distance function for calculating the noise impact indicators. The noise impact indicator, where d; is computed for each time series. The Lagrange multiplier defining the trade-off between the compression of X and preservation of Y in the IB objective.
á
f
B1, B2 Ary A A2
The Lagrange multipliers defining the informative terms within the RSTIB objective.
The Lagrange multipliers defining the balance between the informative terms within the RSTIB-MLP objective.
LIB CRSTIB LASTIB-ME MLP
The original objective of IB principle.
The objective of RSTIB principle.
The learning objective of RSTIB-MLP.
Ereg Cas Cy Lz
The typical regression loss.
The upper bounds of input regularization, target regularization, representation regular- ization.
L
The number of layers.
Na
The time slots in day.
No
The number of days in week.
fr
The teacher model.
The learning rate.
The expectation of random variable, i.e., the mean the possible values a random variable can take, weighted by the probability those outcomes.
E
The maximum epoch number.
B
The batch size.
The decay factor.
The non-linear activation.
The noise ratio.
H())
The entropy of a discrete random variable, e.g., H(X) represents the entropy of X. The mutual information between two discrete random variables, e.g., I(X;Y) repre- sents the mutual information between X and Y.
I(s))
L((3))
The lautum information between discrete random variables. e.g., LI(: repre- sents the lautum information between X and Y.
14

---

Information Bottlencck-guided MLPs for Robust Spatial-temporal Forecasting
Definition B.5. Mutual Information. Given two discrete random variables X and Y, their mutual information (MI).
denoted as I(X;)) is defined by:
I(X - -(X)) - - H(II))
- Σ p(x) logp(o)) +
Σ
p(a, y) logp(z)))
BEX
=EX,NEY
Σ p()
Σ p(a,y) +
Σ
p(a, v) logp(zlv)
FEX
NEY
ZEX.NEY
p(xxl))
Σ
p(a, v) log
(11)
p(x)
ZEX,EY
where p(r, y) is the joint distribution between X and Y. p(x) is the marginal distribution of X and p(xly) is the conditional probability distribution of X given Y, respectively.
Definition B.6. Relative Entropy. The relative entropy, or Kullback-Leible (KL) distance, between two probability mass functions p(z) and ((x) is defined as follows:
p(z) log p(x) q(x))
LL(pll = Ep(z)
(12)
zEX
The mutual information between X and Y can also be expressed as [(: = K L(p(a, v))p(())(()), which implies that mutual information relative entropy between the joint distribution p(r,, y) and the product of the marginal distributions p(z)p(y).
Remark B.7. Mutual information satisfies the following identities:
I(x:Y) = H(X) -(XX))((H()) - H((])) -(X:))
(13)
I(X;Y) H(X) - H(XI)) = H(X)+ + +(Y)) H(X,Y).
(14)
The relationships among (X), HYY), H(XI)) X)) ((X;)) and H (XX Y) can be visualized in Venn diagram, as shown in Fig. 6.
Figure 6. Relationship between ((X), H(Y), H(X[Y), H(IIX), ((XY)) H(X,Y). (I): H(X (2): I(X;Y): (3): (1+2): (X): (2+3): H(Y): (1+2+3): H..
Remark B.8. Mutual Information w.r.t to three variables I(Z;X,Y) Y). The mutual information (Z; quantifies the shared information between the variable Z and the variables consisted of both X and Y. It defined mathematically
p(z, y. log p(2(2,8) v)
I() = Σ
(15)
p(z)
zEX.VEY,:EZ
where p(z.y. y, 2) represents the joint probability distribution the variables X, Y, and Z. This expression highlights how much uncertainty in Z is reduced by knowing both X and Y.
15

---

Information Bottlencck-guided MLPs for Robust Spatial-temporal Forecasting
Definition B.9. Lautum Information. Because of the non-symmetry of the KL divergence, Lautum information is defined the divergence from the product of the marginal distributions the joint distribution of two random variables X and Y and is given by:
LI(XYY) (L(p(=)p(v)) p(z, y)),
(16)
where p(z, y) is the joint distribution of X and Y. p(x) and ply) are the marginal distributions of and respectively, This concept was introduced by (Palomar & Verdú, 2008).
Definition B.10. Conditional Mutual Information. The conditional mutual information between and Y given Z, denoted as ((;; YIZ), measures the amount of information shared between and Y that unique and not already explained by Z. is defined as:
= (X
p(a, 8/2)
Σ
p(z, y, log
p(<l=))(ullz)
=EX.VEY,EEZ
(17)
[p(rlu,2)
Σ
p(a, v. log
p(æ[2)
DEX.
which quantifies the additional information about X obtained by observing Y when the influence of Z is already known.
Definition B.11. Interaction Information. The interaction information concerning the variables X, Y, and Z quantifies the unique information shared by these three variables. It is formally defined as:
T(X;Y; = I(X;)) - T(X;
(18)
This measure reveals whether the mutual information between X and Y is increased or decreased by conditioning on Z.
B.2. Preliminaries for IB and DVIB
Here, we detail the preliminaries regarding the Information Bottleneck(1BB) and Deep Variational Information Bottle- neck(DVIB). We denote X as the input to different IB models, Z as the encoding from X, and Y as the target.
X
Y
X
Y
(a)
(b)
Figure 7. Comparison of (B(a) and DVIB with lifted markov assumption Z - X Y(b).. Refer to Fig. 2 for more details.
In Fig. 7, the entropy of X, i.e., HX). and the entropy of Y, i.e., H(Y,, are depicted as circles, with their mutual information Y( represented in the overlapping area. The representation learning guided by the IB principle aims to optimize the information flow by retaining as much relevant information about in Z as possible while minimizing the redundant information from X. This principle targets reducing irrelevant information H(XY) Z captures, namely [XX; Y)),, aiming for what is termed the "minimal sufficient representation", ideally encapsulating solely I(X; Y). Achieving this optimal representation presents substantial challenges due the intrinsic complexities of the models and the varied selection of parameters and hyperparameters, such as 3 in Eq. (2).
Incorporating the IB model with deep learning, where mutual information terms are modeled using deep neural networks (DNNs), has proven successful. The DVIB method leverages deep learning to approximate the IB model, finding a sufficient statistic Z given X while retaining pertinent side information about Y. The approach involves parameterizing the conditional probabilities P(Z X) and ((Y|Z) using DNNs, thus enabling direct recovery of the terms in the original IB objectives.
16

---

Information Bottlencck-guided MLPs for Robust Spatial-temporal Forecasting
Regarding the assumptions of the Markov chain, the typical practice in the original IB formulation assumes Z X- This assumption is also utilized derive DVIB. Additionally, by its construction, the DVIB model satisfies the data generating process, which implies that the Markov assumption X - Z Y holds. Adhering to both Markov chain restrictions in DVIB may seem overly restrictive, and as pointed out by (Wieczorek & Roth, 2020), no directed acyclic graph (DAG) with three vertices can faithfully represent such a distribution. Consequently, (Wieczorek & Roth, 2020) theoretically explore the possibility of relaxing the Z circumventing the necessity for the Z X Y configuration. As illustrated in Fig. ((a), the original method does not encompass a region representing [2; )| owing to reliance on the Z X Markov chain assumption. Conversely, in the DVIB approach with this assumption lifted, the proposed RSTIB approach, the term T TZZ is represented in Fig. 7(b). Detailed explanations regarding the derivation are provided in Proofs. F.5.
Y restriction by demonstrating how (2; Y) can be lower bounded, thus potentially
Algorithm 1 RRTIB--MIL MLP for Spatial- Temporal Forecasting
Input: Historical
data X input adjacency matrix (optional), trained teacher model fr. N time series, Na time slots in a day, N₂ 7, Lagrange multipliers Ags Au Ag max epochs E, learning rate r)
Output: Trained model parameters e
1: for to F. do
2: // Obtain noise impact indicator
3: Obtain Y fA, x")
4: Compute a; for each series i (Eq. 5)
5: // Prepare RS11B-MLP input
6: Add spatial-temporal info to X using Eq. 20
7: // Data reparameterizatiin for X,Y
8: Encode to get 62, 6y ~N( 6²)
9:
Derive dz Gy ~ NN,, o² )²), with p= x + i
10:
Sample ē. ÿ from reparameterization using (0, 1)
11:
// Input/Target regularization
12:
Compute upper bounds via Proposition 4.6
13:
// Latent Z representation
14:
Reuse encoder to get Pæ. NN(1,, o?)
Sample Z by μs + agE
16:
// Representation regularization
17:
Compute upper bound via Proposition 4.7 // Decoder and optimization
18:
19:
Decode Z to get Y
20:
Compute loss CRSTIB- (Eq, 6)
21:
Update = θ 7) VOLRSTIB-ME MLP
22:
end for
23:
return +
B.3. Preliminaries for Sample Indistinguishabiiiy
recent work (Deng et al., 2021) identifies that the essential element for the efficacy of STGNNs lies in the capability of GCN mitigate the issue of spatial indistinguishabiiity. Thus, in MLP for spatial-temporal forecasting, additional modules needed to alleviate the sample indistinguishability bottleneck by attaching the ppatial-temporal information. In this study, spatial-temporal prompts (Tang et al., 2024), which is extension of spatial-temporal identity (Shao et al., 2022a), are adopted to attach this information to the historical input data X for obtaining the input X to the models.
B. TEMPORAL IDENTITIES
With the spatial-temporal identities technique, inputs can be attached with spatial-tempora identity information, which is as follows:
X FC E. T.T.
(19)
where FC refers to fully connected layers that map the dimension the historical input data X from R PxNXC to the dimension PxXC', Assuming N time series, No time slots day and N,, = 7 days week, the spatial-temporal
17

---

Information Bottlencck-guided MLPs for Robust Spatial- Forecasting
Target Regularization
I(Z; Y) Supervision
Teacher Model
I(Y; Y)
ys
YT
Y
Y
Y
Noise Impact Indicators
Input Regularization
I(X:X)
X
Embedding Layer
X
X
Z
Regression Layer
MLP Layers
Attach the Spatial-temporal Information
I(;; 2)
Representation Regularization
Figure The framework of RSTIB-MLP The historical input data X attached with the spatial-temporal information to generate RSTIB-MLP's input X. Then, input regularization, target regulan rggularization and representation regularization imposed, along with the optimization for supervision. X is also used to calculate the noise impact indicators to quantity the noise impact on each time series to balance the informative terms within this framework better.
identities are in three trainable embedding matrices, i.e., E E NxC with each E, E RC, TTiD E N with each TTDD E TRC", and TDW E x with each TDDWW E IRC", The input to the model will be X E PPNxAC by concatenating (l) each term.
B.3.2. SPATIAL- PROMPTS
With the spatial-temporal prompts technique, inputs can be attached with spatial-temporal contextual information, including which is follows:
X = FC, (FCC), (FCC (T(DD) ||FC (E (DoW))
(20)
Here, the terms ((a) E RNx with each E(() E RC represents learnable spatial prompt, E(TaD) R NNxx with each E, (ToD) C and D(Do)) E Nwx with each E, (DDo) E R represent the learnable temporal prompts, with the same
settings that we have N time series, Na time slots in day and N., days in week. Et-P:t E PxNx with each E E C represents the dynamic spatio-temporal transitional prompt, inherent from (Han et al., 2021). FC,, where 1= 5, refers to fully connected layers that map the data and all the embeddings to the same dimension C². In this case, the input fitted into the MLP networks will be X E PxcX""'' with each Xt, E IRSC"
C. Algorithm
Our learning framework shown in Figure 8. Our algorithm is detailed in Algorithm 1
D. Baselines
All baselines for comparisons are based on their original implementations. We list their source links here.
STID, http://ggthhbbcomm/cccsshaaSSStt
TrendGCN, http://hhiiiooo////oooggg/////ea
18

---

Information Bottleneck-guided MLPs for Robust Spatial-temporal Forecasting
D http:///hhhhbooommSS//////12///0
S http://gghhbooooo////iiiiiiitttt
GWN, h http:////ggghhooooooooo/oo///o//
AGC SGCGKNN/ssss/shhhiboommmm////ccc
GMSDR, http://ggitnb.ccom////////////h
FOGS, http://gglabbboom////ciiiuuaaaof
BiTGraph, http:///ggtaabboooo/oooooooooaaa
FreTS, https//gtthhbccmm/aaayyyyytttS
E. Datasets
Table 4. Statistics of Datasets
Dataset
Node
#Time Steps
#Sample Rate
#Time Span
PEMS04
307
16992
5min
01/2018 02/2018
PEMS07
883
28224
5min
05/2017 - 08/2017
PEMS08
170
17856
5min
07/2016 08/2016
LargeST(SD)
716
35040
15min
01/2017 12/2021
Weather2K-R
1866
40896
Thour
01/2017 08/2021
Electricity
336
2184
hour
10/2014 12/2014
The statistical information for six datasets is summarized in Table 4.
The PEMS04/07/08 datasets comprehensive collection of traffic data gathered from Districts 4, 7, and 8 of Caltrans, respectively. These datasets typically include flow (vehicles per hour), speed (miles hour), and occupancy (percentage of time the detector is occupied), recorded across multiple lanes and aggregated into 5-minute intervals. Public accessed data can be found in (Guo et al., 2021b);
The versions of the datasets are the same as the sources' default versions.
LargeST (Liu et al., 2024a): It is publicly available at https//giithbbccmm/munn////gggg Weather2K-R (Zhu et al., 2023b): It is publicly available at https//ggiiibb.coomomyy///////ee
Electricity (Deng et al., 2021): It is publicly available at https//gihabbboo//DDDnn//SYooo
F. Theoretical Proofs
F.1. Proof for Proposition 4.3
Proof. We firstly provide the proof for I(X;Y) Z) H(X, H(X,Y. H(Z).
By utilizing the definition of conditional mutual information, 1(X;Y] Z) can be expressed follows:
T(X: Y Z) HXX Z) H(Y Z) H(X,Y Z)
(21)
By expanding each term using the definition of conditional entropy, we can obtain:
H(X Z) H(X, - H(Z)
(22)
19

---

Information Bottlencck-guided MLPs for Robust Spatial- Forecasting
H(Y 2) = -
(23)
N(X,Y 2) - HXX,Y, -
(24)
Then have:
(X;YI2)
(H(Y 2) +
- H(())
(25)
Simplifying the equation, we can obtain:
- H(X, #(, -
(26)
Proofs of Y(X;Y| -) = H Z) H(Y, 2) H(X,Y, (Z) have been completed. Then, we have the following
equivalent expression:
I(X:Y = -
= H
(27)
+(,,) -(X,) -
+ (((,) - - H(X)
By using the following definitions:
I(X:Y)
(28)
[[7;X,Y) -
(29)
[2; = - H(Y),
(30)
| = - - H(,,,, H(X),
(31)
We have:
I(X;Y = - - ( - (2;Y1X)
(32)
According to Definition B.11, we draw the conclusion as follows:
I(Z;Y x) + 1( Y) = -
(33)
F.2. Proof for Proposition 4.5
Proof. Consider the mutual information I(X;X) defined as follows:
P(XIX)) P(X) ]
x(X;x) = =E log
(34)
We parameterize the conditional distribution P(XJX) by utilizing Po. (xIX), and substituting the marginal distribution P(X) with variational approximation Q(x), which introduces an extra K KL (((()))II((})) term, we get:
Po Pxx(xxx))
x(x;x)= = Ex,x
log
- KL(P(())QQ()).
(35)
Q(X)
20

---

Information Bottlencck-guided MLPs for Robust Spatial-temporal Forecasting
Using the non-negativity of the Kullback-Leible divergence, we establish an upper bound:
((x;)) S (XLPI
(36)
Similarly, for the mutual information I(Y;Y) Y), we have:
1(;;Y) (KKK,,,
(37)
F.3. Proof for Proposition 4.6
Proof. We demonstrate the proofs by utilizing L2 as an example, which is the upper bound of the input regularization. Considering C2 the Kullback-Libler divergence from normal distribution N(uz, of)) the standard normal distribution (0, 1), the divergence is given by:
C. KL (N(u,, 2 (0)
(2 (3-1,1
pz)
exp
20
exp
log
dr
2TT
2
/2mo3 cxp
(-2²) 22)
exp
-
11,))
Log( (og((2T0²))
(x.
M
+
x22
da
(38)
2TT
20 11
2
20
2
1
log(o2)
+1
1
exp
(x (bz
mx) do
exp
(- (x
M2
x² dx
2
02
20
20
1
log(o
2 + 0 + 11 - 1
2
Analogously, the upper bound of the target regularization, denoted as Cy, can be similarly derived and results in:
Ly 2 (-log log 0² to + 1).
(39)
F.4. Proof for Proposition 4.7
Proof. The proof can be found the similar Proof. F.2 and Proof. F.3.
F.5. Proof for Proposition 4.8
Proof. Without holding the Markov chain condition X Y, we cannot derive the lower bound I(Z:Y) in (Alemi et al., 2017). Therefore, We re-derive the substituted lower bound of ((Z with the additional term I(Z;YJ)) that arises upon relaxing the constraint X- Y. Then, we establish the lower bound for our objective.
21

---

Information Bottlencck-guided MLPs for Robust Spatial-temporal Forecasting
((;Y)) KL (fpeav.)RUU.,)a)P((2)))
PI(XX) P(Y)Z)P(Z)
dz dx dÿ
P(Z)P(Y)
- -BP(x / P(ZIx, log d(
- EEXIE log P(P)) / P(ZIX,Y)ds
P() H(()
EINN P(Y|Z, (Y)
EP(X) / P(Z,Y P(Z, PIx) dz H())
Ep(X) /
P(Z P()I))((2))II)
dz dij + H(V)
P(YIX)P(ZIX)
EP(x)KL (P(Ý.ZIX) (()1X)((ZI))) + / |
dz dr) H())
(40)
|L (P(Ý. x (((1X)(((()))
H
// (( ((z de dzj + H(Y)
[P (P(Y. xI) (((1X)((ZI)))
EP(x)(KL (P(.,2I)) (()1X)P(ZIX))
+ / ((V1X))(()X))0 P +dji+H(0)
P(Z,V1x)P((1X))
EPXy)KL (( z1x)
(()1))(((X)) KKL (P(YIX)P(ZIX) p(Ý p()2/)))) H())
=E(x)/KL (P(Ý) x ((X1X))(ZIX)) +KL (P(71X))((XX)) p(F p(F,2(x))
+ (P T(V; 1TTY; x|X) + P log P() + H())
> EP(()) log + +(Y)
Let Q((IZ) be the variational approximation of the intractable 2() similar to Eq.(3 By the non-negativity of the KL divergence, we have:
KL(P(Ý12) || (( 2
(41)
Thus, the inequality simplifies to:
I(Z:Ý) 2
E 1 log P(7 - HÝ²) 2 B log Q( 2) + H()).
(42)
Since the entropy H(Y) is independent of the optimization, P(2xx112)))))) log (() x -Ereg Y), where y represents the predictive outputs of RRSTI--MLP model.
maximize T(Z,Ý) by maximizing
G. Sanity Check for RSTIB-MLP
In this section, we perform a sanity check on the RSTIB-MLE model to determine whether the instantiation impairs the Information Bottleneck(IIB) nature. By conducting this analysis, we aim to theoretically ensure that the RSTIB principle, as
22

---

Information Bottleneck-guided MLPs for Robust Spatial-temporal Forecasting
an extension of the IB, does reduce undesirable degenerate solutions.
Table 5. Comparison of Assumed Markov Chains, Structural Equations, and Corresponding Directed Acyclic Graphs (DAGs)
Assumed Markov chain
Z
X
X
Z
Z (2(,nn2
Z fz 72)
Possible set of structural equations
Y fr My)
Y fr
Corresponding DAG
X
Y
X
7
Y
Z
As articulated by (Wieczorek & Roth, 2020), the assumptions underlying different Information Bottleneck (IB) principles correspond to different admissible information flows, which can be effectively represented using Directed Acyclic Graphs (DAGs). This approach allows for a convenient elucidation of the properties in different IB models. The arrows in the DAGs explicitly symbolize the data generation process rigorously defined by a corresponding set of equations.
Z-x-y
X
Y
Z~X-1
Y
Z-X
Y
--X--
Y
(b)
a
Figure Admissible DAGs Under Different Markov Assumptions while not impairing IB nature. (a) YY: X -
x~
Information Flow
Z
Y
X-Z-Y
Figure 10. DAGs of RSSTI-MLP.
As depicted in Fig.9, the Markov chain assumption Z X Y serves as sufficient condition to preclude the model from deriving the trivial solution Z = Y. Nonetheless, the necessary condition that 7 should not directly depend on Y. Consequently, the requirement Z --X X Y can be relaxed to merely prohibiting a direct edge from to Z in the DAGs, i.e., Y + Z. This relaxation is achieved by adhering to admissible DAGs under the Markov assumption X depicted in Fig.9. Moreover, Z must encapsulate information about both X and Y. necessitating the exclusion of structures Z X < Y and Z Y < X in the DAGs, which would otherwise result in I(Z; Y) = 0 and [(X; Z) = O, respectively. To summarize, since we also lift the Markov restriction Z - X Y by just holding X - Z Y condition, it is imperative to adhere to the DAGs outlined in Fig. 9. This mandates thorough sanity check of the RSTIB-MIPP model.
Y, as
The DAGs of RSTIB-MLA are presented in Fig. 10. As is shown the figure, our model effectively ensures that it does not reduce to the solution = Y while simultaneously guaranteeing the preservation of information from both X and Y, thereby maintaining the nature of the information bottleneck principle.
23

---

Information Bottleneck-guided MLPs for Robust Spatial-temporal Forecasting
H. Computational Complexity Analysis
In this analysis, we theoretically compare the computational complexities of our RSTIB-MLP with other leading baselines in spatial-tempont forecasting. Many advanced "TGNN-baser methods integrate Temporal Convolutional Networks (TCNs) and Graph Convolutional Networks (GCNs) with self-attention mechanisms to effectively capture temporal and spatial dependencies, respectively. In contrast, ESTIB-MLI employs Multi-Layer Perceptrons (MLPs) alone, simplifying the model architecture. This section provides detailed analysis of the computational complexity associated with these fundamental model architectures.
Table 6. Notation for Computational Complexity Analysis of GCNs and Self-Attenion Mechanisms.
Symbol
Description
N
The number of time series
The edge matrix
[E]
The number of edges
A
The adjacency matrix, where A NRN The hidden dimension of each time series The average degree of the time series The adjacency matrix with self-loops, Á= A+ where / is the identity matrix The diagonal degree matrix corresponding to Å, where D.s = Σ, Aij
d
deg
A
D
A'
The normalized adjacency matrix, A'' = D- AD The feature matrix, where Z E xdd
Z
WO
The feature transformation matrix for the 1-th layer, E Rdxd A non-linear activation function
T()
The total number of layers in the network
w-O
The query matrix for the I-th layer of the self-attention mechanism, W Rddd
The key matrix for the 1-th layer of the self-attention mechanism, w
K E Rdxd
The value matrix for the I-th layer of the self-attention mechanism, w E Rdddd
Computational Complexity of GCN. We detail the computational complexities of GCNs based the notations provided in Table 6. The computation at the 1-th layer of GCN can be expressed as:
z ((+1) 1) = 7(A'
(43)
which can typically be divided into two primary operations:
Feature Transformation: zn0 (1().
Neighborhood Aggregation: /(-1) 7 (A)
Thus, naively, the computational complexity of GCN can be expressed as:
OLL (N dl + N2 dd)
(44)
In practice, the seatter function from Pytorch (Paszke al., 2019) can efficiently handle the graph structure's 's sparsity. Given that the average degree of nodes is denoted by deg, the complexity for neighborhood aggregation per node is O(deg x d), resulting in total of O(N deg d) O(EE) d). Thus, the practical computational complexity of GCN is:
O(L (N df + (8)-d)
(45)
Generally, the complexity of the activation function 7(.), being an element-wise operation, is negligible and can be approximated as O(N).
When combining GCNs with a Self-Attention mechanism, the query, key, and value matrices in the I-th layer, denoted as W Wi. and Wtt. respectively, are all d x d matrices. The self-attention mechanism involves the following computations:
24

---

Information Bottleneck-guided MLPs for Robust Spatial-temporal Forecasting
1. Compute Q = Z K Z K and V 2 w each with computational cost O(Nd)).
2. Compute the product Q())
K(O), which incurs a cost of dN.
3. Compute the final attention scores, requiring O(N2d) time.
Therefore, the total computational complexity when incorporating self-attention is:
O NNd))
(46)
Computational Complexity of TCNs. We detail the computational complexities of TCNs based the notations provided in Table 7. TCNs integrated with attention mechanisms are often benchmarked against sequential models such as RNNs and LSTMs. The computational complexity for these sequence models is typically O(L x T x N² x d'). However, similar to the above analysis, TCNs equipped with attention mechanisms generally incur lower computational costs, estimated at OLL X N x T x d). The reduced complexity is attributed to the faster learning dynamics of T compared to (N x d)². Although TCNs have been demonstrated to enhance efficiency significantly (Zhou et al., 2020), they are still considered sub-optimal compared MLP networks.
Table 7. Notation for Computational Complexity Analysis of TCNs with Attention Mechanisms.
Symbol
Description
L
The number of layers in the model
T
The length of the time series
N
The number of time series
The hidden dimension
Computational Complexity of RSTIB-MLP Networks. The RSTIB-MLP architecture employs a straightforward encoder- decoder MI.P network design. We denote d as the input dimension, dout as the output dimension, and as the dimension of the hidden layer. The computational complexity of the model can be succinetly expressed as O (din d + dout x d)), where N represents the number of time series being processed.
Table 8. Notation for Computational Complexity Analysis of RSTIB-MLP Networks.
Symbol
Description
N
The number of time series
din
The dimension the input
dout
The dimension
output
The dimension of the hidden layer in the MLP network
Thus, theoretically, ASTIB-MLP computational complexity is considerably more efficient than that of methods, primarily due its streamlined MI.P-based model architecture.
Further Discussions
I.1. Limitations
Our general framework leaves many interesting questions for future investigation. For example, could we automatically search for better regularization coefficients with theoretical and empirical efficiency guarantees? Besides, MLPs with specially designed modules have been proven to be effective. Could we instantiate the RSTIB principle to incorporate more reliable spatial-temporal information from the module design? These are all the limitations and future directions that we are attempting to explore.
I.2. A further comparison of RSTIB and other IB principles
This section elaborates on the comparative analysis between the Information Bottleneck(IBB, its variants and our proposed Robust Ppatial-Tempore Information Bottleneck (RSTIB) principle. Notably, RSTIB extends the capabilities of Deep
25

---

Information Bottlencck-guided MLPs for Robust Spatial-temporal Forecasting
Variational Information Bottleneck (DVIB) and Robust Graph Information Bottleneck (RGIB). Given that RGIB itself generalizes the Graph Information Bottleneck (GIB), our comparison primarily focuses on the IB model as introduced by (Tishby et al., 2000), alongside its significant extensions: DVIB (Alemi et al., 2017), GIB (Wu et al., 2020), RGIB (Zhou et al., 2023), and our RSTIB.
Briefly speaking, compared to IB, as well as DVIB and GIB derivatives, the RSTIB introduces significant advancements by accounting for spatial-temporal data noise present both in input and target regions, enhancing robustness both theoretical constructs and instantiation. While the RSTIB extends the RGIB principle. diverges by considering lifting specific Markov assumption typically held to explicitly minimize the irrelevant information terms. The subsequent reformulations can ensure the integrity of the IB principle. In other words, RSTIB ensures, both theoretically and practically, that encoding Z does not reduce to the trivial solution Z = Y and preserves information from both X and Y. This enhancement is meticulously analyzed in Section G, a thoroughness not typically found in RGIB's 's analysis. Furthermore, the instantiation of the RSTIB principle does not depend on specific data structural assumptions inherent to the instantiations of RGIB, which are based on graph data and assume that the number of edges in the pruned graph, denoted as [Zal, does not exceed those in the original graph, ||]. Therefore, the RSTIB framework demonstrates more general potential applications and robustness, making it suitable for instantiating in Multi-Layer Perceptron (MLP) networks for spatial-temporal forecasting.
Analytically, traditional models such as IB, DVIB, and GIB predominantly focus on minimizing the conditional entropy H YY) while maximally preserving (Y))). These models operate under the implicit assumption that T(Z; ([Z;Y|X) 0, adhering strictly to the Markov chain condition Z - X - Y. This approach proves effective for specific applications, such as classification tasks. However, in pratial-temporaa forecasting, with the Assumption 4. and 4.2 proposed about spatial-temporal data, such Markov assumption is too restrictive. The noise-related irrelevant information could be obscured within this restriction, thereby questioning the direct adoption of the Markov assumption Z X Y. Besides, RGIB, by its definition, considers an explicit relationship between the information terms and attempts to balance them in self- controlled way. Some of its derived terms, such H X, Y), is minimized by controlling H (Z) to be within the range TH < (Z) < <Tt given that (Z) 2 max [ H (2 x), (Z 2 HZ||X,Y) Y). This requires a delicate balance within the RGIB objective. In comparison to these. RSTIB adopts distinct approach. It lifts the Markov condition of Z - - X-Y by adhering to only the X - 2 - Y assumption, which is less restrictive while not impairing the bottleneck nature of the representation Z. This formulation introduces [(2; X * O, with the existing 1 (2; X (Y) to be minimized, enhancing robustness against noise perturbations in both input and target. Besides, RSTIB focuses on learning the "minimal sufficient representation" while minimizing explicitly expressed and reformulated irrelevant information under the X - Y Markov assumption. This strategic orientation provides theoretical guarantee that the encoding of Z neither reduces the trivial solution Z = Y nor compromises the information from X and Y which results in 1(X; Z) 0 and [(: Y) = 0 respectively.
Regarding instantiations, the GIB is inherently intertwined with the Graph Attention Network (GAT) architecture. While the RGIB mitigates this constraint by eliminating the need to modify the Graph Neural Network (GNN) architecture, it still necessitates reliance on GNNs and their inherent graph structures. This dependence is under the assumption that the number of edges in the pruned graph, denoted as Al,, does not exceed those in the original graph, denoted as |-A]. However, such assumption can not hold when generalizing to Multi-Layer Perceptron (MLP) networks, where graph structures are inapplicable. Meanwhile, spatial-temporal data often comes with no pre-defined graph structure. GIB/RGIB -baeed method can not directly be applied to such scenario. Besides, the DVIB adheres strictly to both Markov chain assumptions, which imposes overly restrictive constraints on the optimization process for the potential set of joint distributions P(X,Y, Z). In response to these limitations, we propose the ASTIB-MLA instantiation, as outlined in Section 4.2. The information flow of RSTIB-MIP follows Fig. 10, which under less restrictive Markov assumption for a wider array of potential joint distributions while not impairing IB nature with theoretical guarantees. Besides, integrates three independent and identically distributed (i.idd) Gaussian distributions as the prior distributions for the input, representation, and target regions, which play the role of minimizing the irrelevant information during the optimization. These circumvent the dependency on graph structures, which can be instantiated in MLPs for spatial-temporal forecasting.
I.3. Linking feature diversity with spatial-temporal heterogeneity
To link feature diversity with spatial-temporal heterogeneity, we propose the Average Node Deviation (AND) metric (Chen & Liang, 2025), which measures the average pairwise squared difference across all node features. Given a feature matrix
26

---

Information Bottlencck-guided MLPs for Robust Spatial-temporal Forecasting
X E Ruxd where the number of samples (e.g., nodes) and is the feature dimension, the AND metric is defined as:
( =
2skk 2zb)
(47)
n2
To connect the AND metric with the overall variance, we derive as follows:
D(X)
Tss xz
n
Lik
+
2z - -2011
n
K.J.k
2
2 Σ
Σ
3
Dikk
n
1
2
2
Σ Dite
71
n
2(n
1)
1
Σ
n
n
(48)
n
n
2(n 1)
tr (Cov)
n
where 2g Ex= Fik is the sample mean of the k-th feature dimension, and Cov E Rdddd denotes the sample covariance
matrix of X. The trace of the covariance matrix is given by:
d
tr(Cov) =
2kc
n
(49)
This derivation shows that the AND metric is proportional the total variance across all feature dimensions:
D(X) x tr(Cov)
(50)
The AND metric emphasizes total variance, but it does not distinguish whether such variance is evenly distributed across features. In contrast, our investigated Var metric (see Eq.(1)) emphasizes balanced standard deviations across feature dimensions, thus promoting uniform diversity.
L.4. The Role of Knowledge Distillation in Enhancing Feature Diversity and Robustness
In our framework, the Lagrange multipliers associated with regularization terms are fixed as constants. This implies that the trade-off between the preservation of the target and regularization strength remains uniform across all time series, which may be suboptimal in scenarios with heterogeneous noise patterns. Moreover, the use of strong regularization can restrict the representational capacity of the MLP backbone, potentially impeding ability to capture complex temporal-spatial dependencies.
Based the above, introduce dynamic regularization mechanism based noise impact indicator, implemented via KD module. Our motivation stems from the empirical observation that the feature variance tends to decrease significantly under higher noise ratios. Therefore, we propose to adaptively adjust the KD-based regularization strength according to the estimated noise impact of each individual time series. Specifically, when the noise impact is low, we relax the regularization imposed by KD to avoid unnecessary suppression of informative features; conversely, when the noise impact is high, we intensify the regularization to encourage robustness by suppressing noise-sensitive components.
27

---

Information Bottleneck-guided MLPs for Robust Spatial- Forecasting
This noise-aware KD mechanism allows the model dynamically modulate the regularization pressure, thereby improving feature variance and mitigating the negative effects of noise. Notably, the noise impact indicator, which quantifies the corruption level of each input time series, plays a crucial role in guiding this adaptive behavior during training.
Empirically, we observe consistent performance improvements when incorporating the KD module. These gains are attributed to KD's ability to rebalance the optimization objective across time series with varying noise levels, enabling the MLP-despite its limited capacity-to prioritize cleaner, more informative inputs. However, we also note that while KD contributes to improved robustness, its impact on predictive accuracy is less pronounced compared the gains brought by the RSTIB mechanism. This likely because the overall objective still necessitates a delicate trade-off between target preservation and the compression effect induced by reparameterization. Although KD adjusts the relative weighting, it remains constrained by the inherent optimization objective.
J. Experimental Implementation Details
We provide a comprehensive description of the experimental setrings. For all experiments, the best models are selected based on the Mean Absolute Error (MAE) metric on the validation set. All comparative baselines are trained using their default settings. The models are trained on NVIDIA GeForce RTX 3090Ti GPUs, utilizing the PyTorch framework (Paszke et al., 2019). The main code bases referenced are STID (Shao et al., 2022a) and STExplainer (Tang et al., 2023), as and
implemented in regarding the attachment of spatial-temporal information, we adopt the spatial-temporal prompts technique in STGKD (Tang et al., http://ss//opppeiiiieeeeeeeee//e
respectively. Besides,
WNG=akk It combines pratial-tempora identity (Shao et al., 2022a) with dynamic graph construction (Han et al., 2021). The reference implementation for this technique can be found in and
We implement the noise injection by firstly loading the original datasets, then we conduct the data normalization. Further, we build the index information about time of day and day of week. Notably, The index information is not perturbed by the noise. It is concatenated with perturbed input afterwards. Besides, the robustness experimental setup is inspired by the approach used in RobustTSF (Cheng et al., 2024), which investigates time series forecasting under anomalous conditions. While their work focuses on time series forecasting with outliers and noise, we investigate similar setting but under spatial spatial-temporal forecasting scenario by introducing noise perturbations and artificial data missing patterns into spatial-temporal graphs. Both works share a similar methodology of injecting synthetic corruptions into the data to evaluate robustness under controlled perturbations. We implement this attachment by adopting the default settings in their works and combining them following the guideline of (Tang al., 2024) for fair comparison. The additional hyperparameter settings and additional experimental details provided in the subsequent sub-sections.
http://giiuub..oomzzziiist
J.1. Implementation Details for RSTIB-MLP
We adopt PyTorch 1.13.1 on NVIDIA RTX 3090Ti GPUs. The algorithm of RSTIB-MLP is shown in Algorithm 1. We follow STID (Shao et al., 2022a)'s 's default model configuration, using 3 Multi-Layer Perceptrons layers. The nonlinear activation J is ReLU. We follow STID's default learning rate setting, i.e., we initialize the learning rate 0.002, and apply a decay factor r = 0.5 for all three benchmarks. A summary of the default hyperparameter settings is in Table 9. Table provides the hyperparameters that produce the results in Section 5.1. For some specific hyperparameters with a searching space, we provide the results of hyperparameter investigation, mainly consisted of the Lagrange multipliers A₂ E {0.01, 0.001, 0.0001, 0.00001), >, E (0.01, 0.001, 0.0001, 0.00001), Ag e (0.01, 0.001, 0.0001, 0.000013, and the distance function De E { {M AE, S, evaluated in PEMS04 dataset in Section K.9.
J.2. Additional Details for Robustness Study
Each specified data noise ratio is termed as y. And we perform random spatial-temports noise perturbation by adding independent Gaussian noise 7 to each feature dimension of the time series, where ~ N N(O, 1).
28

---

Information Bottleneck-guided MLPs for Robust Spatial-temporal Forecasting
Table 9. Hyperparameter scope for Section 5.1
Hyperparameters
Value/Search space
Type
Batch Size B
32
Fixed*
Epoch E
200
Fixed
Learning Rate
0.002
Fixed
Decay Factor r
0.5
Fixed
Hidden Dimension d
64
Fixed
Number of MLP Layers L
3
Fixed
Non-Linear Activation
ReLU
Fixed
Input Regularization Coefficient Xx
{0.01, 0.00000,00,, {0.01, 0.001,0.000) 0.00001
Choicef
Target Regularization Coefficient Xy
Choice
Representation Regularization Coefficient X
{0.01, 0.001, 0.001,,0000,
Choice
Distance Function
(MAE, SmoothL1,MSEY
Choice
"Fixed: constant value
*Choico: choose from set of discrete values
The boldface numbers: Default setting that produces the result for Section 5.1
K. Further Empirical Results
K.1. Additional Performance Comparison on Clean PEMS Datasets
In this section, we provide additional empirical study for the comparison between the performance of RSTIB-MiL MLL and more baselines targeting temporal traffic forecasting. The results shown in Table 10. By examing the results, it's convincing that predictive performance when learning with clean data be superior, even when comparing with STGNNs.
Table 10. Performance Comparison Under Clean PEMS04, PEMS07, PEMSO8 Datasets. The boldface means the best results.
Methods
Dataset
Metries
STGCN
AGCRN
GMSDR
FOGS
DSTAGNN
STHMLP
TrendGCN
RSTIB-MLP
MAE
20.05
19.83
20.49
19.74
19.30
18.88
18.81
18.46
PEMS04
RMSE
32.07
32.26
32.13
31.66
31.46
30.31
30.68
30.14
MAPE(%)
13.09
12.97
14.15
13.05
12.70
12.74
12,25
12.22
MAE
21.98
22.37
22.27
21.28
21.42
20.71
20.43
19.84
PEMS07
RMSE
35.66
36.55
34.94
34.88
34.51
33.99
34.32
33.90
MAPE(%
9.28
9.12
9.86
8.95
9.00
8.75
8.51
8.33
MAE
16.39
15.95
16.36
15.73
15.67
15.22
15.15
14.51
PEMS08
RMSE
25.60
25.22
25.58
24.92
24.77
24.18
24.26
24.18
MAPE(%)
10.34
10.09
10.28
9.88
9.94
9.82
9.51
9.44
K.2. Further Ablation Study on Each Regularizations
Table 11. Performance RSTIB MSTIB-MLP under different noise ratios and ablated regularization PEMS04.
Noise Ratio
10%
30 %
50 %
Method
MAE
RMSE
MAPE
MAE
RMSE
MAPE
MAE
RMSE
MAPE
RSTIB-MLL w/o x+y+Z
24.97
37.67
16.55%
29.64
45.75
20.43%
29.80
46.71
19.23%
RSTIB-MLP w/o x+y
24.50
37.28
16.23%
28.84
45.12
17.91%
29.19
45.87
18.79%
RSTIB-MLP w/o RSTIB-MLP
23.81
36.60
15.56%
27.49
43.58
16.72%
27.98
44.48
17.85%
23.64
36.44
15.22%
27.15
42.85
17.19%
27.16
43.43
17.76%
this section, we aim to separately examine each regularization term within the objective function, including input, target, and
29

---

Information Bottleneck-guided MLPs for Robust Spatial-temporal Forecasting
representation regularizations. For simplicity, we denote "w/o" mcaning the word "without", "x" input regularization, "y" as target regularization and "z" as representation regularization respectively. As shown in Table 11, all the regularization terms can enhance the performance. Notably, there is fact that the input regularization contributes significantly compared with other regularization terms, while the contributions from the representation and target regularization terms are comparable. The potential reason may lies the fact that applying input regularization ensures that the signals passed to subsequent layers less noisy. Besides, input regularization can prevent the model from overly relying on specific input patterns, thereby enhancing robustness. contrast, solely regularizing the representation may not sufficiently address the complexity and noise present in the input data.
K.3. Further Empirical Study of Combating Data Missing
To demonstrate broader applicability, we conduct a performance comparison and an ablation study showeasing how each module performs when combating against noise arising from data missing. We conduct this experiment on the PEMS04 dataset, where we randomly drop the data by certain ratios. The results are shown in Table 12 and Table 13.
Table 12. Performance comparison of different methods under varying missing ratios.
Missing Ratio
10 %
30 %
Method
MAE
RMSE
MAPE
MAE
RMSE
MAPE
STG-NCDE
20.25
32.58
13.22%
26.32
40.38
15.27%
STGKD
20.57
32.77
13.40%
29.06
44.22
16.34%
STID
22.65
35.52
13.87%
30.21
44.98
16.85%
RSTIB-MLP
19.83
31.79
12.82%
25.45
39.61
14.94%
Table 13. Ablation Study of RSTIB-MLPP modules on the PEMS04 dataset with different missing ratios
Missing Ratio
10%
30 %
Method
MAE
RMSE
МАРE
MAE
RMSE
MAРE
ASTIB-MLL
19.83
31.79
12.82%
25.45
39,61
14.94%
RSTIB-MLP w/o KD
19.95
32.10
12.91%
26.31
40.46
15.22%
ASTIB-MLE w/o KD + RSTIB
20.57
32.77
13.40%
29.06
44.22
16.34%
RSTIB- MLP w/o KD + IB
21.35
33.76
13.94%
29.34
44.69
17.91%
These results indicate that RSTIB-Mi MLP can surpass all the MLP-based baselines, even be comparable with STGNNs like STG-NCDE (Choi et al., 2022). Besides, each module also contributes to the overall robustness. Observing from the results, it is obvious that RSTIB implementation can significantly enhance the robustness, especially when combating data missing with higher missing data ratio. Besides, maditional IB implementation and knowledge distillation can also contribute robustness enhancement, sharing similar conclusions from our previous results.
K.4. Model Architecture agnostic Study
To ensure consistency, implement RSTIB STID (Shao et al., 2022a) to evaluate whether each module's 's contribution enhancing robustness model- or network architecture-agnosiic. Table 14 demonstrates the results conducted on PEMS04. We keep the same
notations as in Section 5.2.
Table 14. Results of RSTIB implementation on STID for clean and noisy PEMS04 datasets.
Noise Ratio
0% (Clean)
10%
30%
50%
Metrics
MAE
RMSE
MAPE
MAE
RMSE
MAPE
MAE
RMSE
MAPE
MAE
RMSE
MAPE
STID
18.79
30.37
12.57%
27.83
41.34
17.31%
36.53
52.74
21.11%
36.22
52.15
21.45%
STID+IB
18.65
30.23
12.53%
25.70
38.17
17.17%
34.55
48.07
19.10%
34.99
50.94
19.59%
STID+RSTIB
18.57
30.16
12.51%
24.27
36.89
16.34%
28.67
44.73
18.06%
29.02
46.44
18.37%
STID+RSTIB+KD
18.50
30.02
12.32%
23.99
36.57
16.22%
28.12
44.31
17.81%
28.86
45.63
17.92%
The above results demonstrate the consistency of our method's performance:
Each evaluated module contributes to the enhancement of predictive performance under clean setting. However, we observe that the enhancement of the performance may not be significant. The potential reason could he the fact that objective function includes more regularization terms, achieving an optimal balance may be harder, leading to potential over-regularization under clean data.
30

---

Information Bottleneck-guided MLPs for Robust Spatial-temporal Forecasting
Each module can also be applied to another bascline model, which is STID (Shao et al., 2022a) in this evaluation. Besides, the enhancement from different modules under noisy scenarios can also be clearly observed. Thus, the RSTIB implementation contributes significantly to the enhancement of STID's ability in combating the noise.
K.5. The Average Improvements of RSTIB-MLP When Combating Against Noise Perfurbation
In this section, we calculate the average improvements of RSTIB-MLP when comparing with the best competing methods by averaging over all the noise ratios each noisy dataset. The results are summarized in Table 15. We can tell from the table that RSTIB-MLP can gain large improvement on several datasets when comparing with specific baselines. For example, RSTIB-MLP improves MAE, RMSE and MAPE by 8.39%. 5.51%, and 3.74% on PEMS04 dataset, and by 7.02%, 4.75%, and 8.08% on PEMSO8 dataset compared to one of the best competing methods, STExplainer (Tang et al., 2023).
Table 15. Average Improvements of ASTIB-MLP Compared with Each Basclines Under Noisy Datasets
Noise Katio
PEMISI
PENIS7
PEMS00
Metrics
MAR
RMSE
MAPE
MAK
RAISE
MAPE
MAF
RMISK
MAPE
STTD
33.51+211
15 15
91352-19988)
(3.35((77825)
GWN
36.82:+28.399)
51.7))
21.6%5--2.67%)
32.70K-10.065%
53. 53.71(-3.99)
(7.10%(()11.99)
15.15/+
62.66-3.5)%)
15.50%((10.711))
TrendGCN
26 26.36i-113395 39%)
41 83%)
(9.089(-12.39%)
31 31556(775663
S2,6((2201%)
19.539((335.205
24.15(+.7228)
20. 00.5496(3002665 39.88(2285%)
STExphaine--CiB
28.64(-8 70%)
43.85145 50%
16.969((44999)) 999%
12.73(4 (0.4%)
53 53,61(-3 60%)
14. (3.1593
25.01(+10.2798
41.08(66.123 12%%)
18 18.49((20.459)
STExplainer
28.99)-8.99%)
43.46
(7.10%
30.79|-5.190 19%)
51. 51.96-0.85%)
(4.17%(--)..735)
24.29+7.02%%
(5.12)((88008) 40.69(44.5%%
STGKD
27.37(-4.9693
42 -8.00+3300 69%)
17,595((41.34 34%)
30 30.28(+3.4445
51 96(-0)768)
(3.339((-7.655))
24. 24.05+663380 38.9))
30.92(-2.9855)
[5 (5.87%<<+1 (.766%)
Graph
28.74)-9.2166)
43.788 22%)
17 -1.09)
31.05 11.00155.9765
$2.31(+1_48%%)
(4.29%((133.500
+8.09))
410000=55.995
15 (8.89))
STC-Drpuue
31.581+17.3 17.
12:141)
19.01% (.73%)
52 $2.4K(11.658 65%)
(4.37% (3,94--113)))))
24, 21,22 +2.6666886)
40.631-6.0655 04.%)
15 15.1255 32%)
STG-NCDE
28.34-800%)
42
10.20%6--2..82 62%)
255%
53 077%
15 (9,5990
(6.38()141128)
40.82(-5.034)
16 16.77%((166919
PreTS
28.70(-9007)
43.72%)
17.22%% 90%)
31.226+6.7516)
53.01(-2.65%)
15.479(220.399)
24.54(+7.889)
40.971-554435
(1.25))
RSTID-MLP
25.98
40.91
16.72%
29.21
$1.55
12.31%
22.48
38,70
13.85%
Noise Rutio
LargeST(SD)
Weather2K-R
Electricity
Metries
MAR
RMSE
MAPK
MAE
RMSE
MAPE
RMSK
MAPE
STTDD
49 381+6. 74%
630022x-70019)
8144,17(+6.2100
75.38 (5.365(()227)
66800-5 59%)
31 (27
GWN
32.53/-9 24%)
51 677%)
19 1196661556119)
6808 57(-13 7398)
8470,200 100.2%)))
74.69% 38%)
TrendGCN
30.31(-280%)
17.84)+2.98%)
17,69((-31119)
5893.37(-0.558)
2642_31(+0.50%)
(2.61%(+0.19%)
35.61(+2.88%)
62.96(-28866)
STExplaine--GIB
49 3..9)
18.45((-9.395
6250 (3(-66025)
811.2.80 1122.8006 (10))
74.88%(4336695
STEsplainer
20 50(-2.866))
47 22.4442 58%
17.69% 15%)
(257.664-55.066
8100, 100480045500 (7.99)
74 (4.551))))
STGKD
30.43(-2.99%)
47.922+2.200 70%
(7.57%(-1.76%)
7639, 894-889-555 .29%)
72. (2.325)))332%)
36.35(+1.59%)
63.8((1105%5)
31 (160776663699)
IrTGrapph
30.341-275%%
30%%
17 000))
5891 (56-0.556))
7679. 5350)
72.4688(40
(99%
63.96(-4. 40%)
31. 31.1333(+7.209
DropDot
30 (1344(2)888)
47 47.0%442.855
(0.06(--5 64%)
6260. 31(-6.026)
R11 1...884(55 9999)
74: 77%6 545))
22.57(+2.50 50%))
64 (4-877--5.00 969))
31. (1.588 1895
STG-NCDE
30.72:-4.1244
48, 188%)
17. 17.78%56-3. 97%)
6255, 566-6.00%)
6110. 60..6 96%%)
744, (+..500))))
37. 32.35))))325)
64. 64.82(-5.8855 88%)
31 (1/.6///++ 22%)
PWTS
)..75-))795)
38.57i+4.4365
17.70%(--5558 525%)
5889, (4(-0,48%)
7637, (77.68//00.000 509))
72.44%% (40.244))
38.8 18.81(+10.069)
65 65,29(46.48 42%)
12.269K<(100990
RSTIR-MPP
29.47
46.42
16.735
5861.39
7599 68
72.76%
K.6. A Study of Average Performance Decay Comparison
Table 16. Average Performance Decay Comparison Under Noisy Datasets
Noise
PEMSS
FEMSUT
PENISIS
Metries
MAE
RMSE:
MAPK
MAK
RMSE
MAPE
MAR
RMISK
MAPE
"STII
11,33-78
744 (9.3)
(69)
34 34.01-67.559
42 100701)
16.72
51.73/68.
28%
21.65%
(0996(-72.922 72.92%)
32.701-51.4899
53.711-61.199 61.19))
17.16%5499688 96.84%)
25.166-71.519
12.661 (1.15)
15.53% (25)
TrenGGCN
26.36(-40.14 14%))
41.68/3358885 85%)
19.089 76%)
86.865550000 056)
52.611-53:299)
19.5366-1229408
(29.49%)
24.15: 41.9)
88(-86664 3999
20.5456-115500
STExplainer-CGi1
28,64) 69.6344))))))
43.85 42.511%
16.96%6 31.37%
32.73:59927155 59.2716)
52.65%)
14.39% 67.12%
25.011 68.19%
41 081 70.675)
18.45% 79.34%%
STExplainer
28.49(:53:425)
43.48K 14.5%)
(1100((-00:975)
30.7915539555 95%)
51,96--55.3499
14.17% 66
24.29i 68.4%)
15,1 (5.39))
STGKD
27.37)-40.4266
42.69)
17.53%61-42.006
30.281-19.1000 1018)
51.90-51.49931
13,33956-50
28%)
24.06659000220
15.87 -8.57%)
BITGruph
33.74:5227115
43.78/ 43.82%)
17.24961
(0.73%)
31,051 53.33%
52.31( 54.99%)
14.29% 66 10%)
25.57 65.45%)
41 11.00L 69.42%
15.22% 54. 245)
STC-Dropout
31,581488 434))
46.971-54.119)
19.01%61544 54 18%)
11199-5188995
52,41(-54.5600 54.5699
14.3754-64 23%)
24.226-64.77 66.7666
40.6366670069)
15.12956-55 (89%)
SIG:NCDE
28.34)47..546)
42.92/
19.2096
31.49:-3.350
53.22 57.279)
77..06%)
26.34670..4%%
40
16.73%6
(6.65%)
FITTS
28.00,52.909)
43.72/-435588
17.22%
51.326-572399)
58.016-57.55 5398)
15.4794-77.8222
54.554665222 (29%)
40.97(-696555)
15.23966-53 99%))
RSTIB-MLP
25.98-40,7448
40.91 7599)
16.72%81-56.622 X229
29:21(-4.2296)
31.556-52.0681
12.31%(-7.7889
22.48-54-3299)
38.70K-600009)
13.858(-466729)
Noise Railo
TargeST(SD)
Waalber2K.
Fleerricty
Metries
MAE
RMSE
MAPE
MAPE
MAE
RMSS
MAPE
STID
32.19(82.905)
49. 09.88-71.700 70%)
68:66665.10
10%)
6302.236-57
1114317/-00 88%)
75 75.3651-1553 34%%
38, 10--800888800 30%)
64 80(-62.339)
31 11.02551005555)
GWN
97.7))
5L.61C-74.4455
19.9698(-688000
01%)
6808 57%-59
8470 201 45%)
74.69((13.179)
TrenGGCN
30.31E-74.305)
47.84(-61.465))
(7.64%((51.554)
5890.37(-47.200 78%)
7642.314-22 50%)
72.615(-111199
35.616-76.229)
62.96(559663)
29.8 825)
STEsphane--GGt
30.13(-61.995)
08.082-620033 03%)
18.45% 39%)
6260:13--6.2158
8112 172.800-033999
74.88%--14555 58%)
STEsplainer
29.50; 68. (616)
43.644 06644-65.00 65.07%)
17.69% 52.90%)
6257.6 56.7%%
$109 301 30.5))
74.76% 14 14.60%)
STGKD
50.433-72.9005 90%)
47.92/-62. 88%)
17.57% 20%)
5891.45 617.5))
7639 391 23. 23.31%%
72.48% 11.3%%
36 36.351-804405
63. 13.822- 59.355%)
31 31.07% (956--9555 95.55))))
BiTGraph
50.24(-0.4228)
47.81c-404455) 449%)
17.61%((-58886
4801
28:-22.9115
72.48%i-11..555
06.00-85..0
13.066(-60000 4278)
51.1 139 11%)
Sro-Dropout
30.34(-7..88%)
47.93 63 y
17. 17.60% 68%)
4200.31:57.0655
8111.841 30.73%)
74.77%
(4.11%)
37.37 87 (00)
64.87( 62 (9.5)
31.58% 91.74%)
STG-NCDE
50.72(-94, 74%)
48.
(7.78% 79%)
6258, Stc 56. 76%)
81 $110 60 30 (.544)
74.73% 14 14.58%)
34: 88. 11%)
64 11.82--8 62.37%
51 31.6055k-9 (.22%)
Pr
30.62-74.57563
48.57) 67.43%
17.70%s 12%))
14| 47.811
7637 68: 22.81%)
72.45%5 11 (1.23
38.810 92.8%)
29.29 61.41%)
32.26161 19..99938
RSTID-MLP
29.47(-68.015)
46.62(-5..66%)
16.73/66 (9.38%)))))
5861.39 3961.399-47.854)
7599.68:222 22.75%%
72.26% 11.27%)
34.53k 74.39%)
53.7255)
28.8/1 83.46%)
In this section, we aim to investigate if RSTIB-MLP's performance is also superior regarding performance degradation caused by the noise perturbation. The detailed average performance degradation of cach baseline, including RSTIB-MLE by averaging across different noise ratios compared with clean sccnario on Table 16. Notably, the performance regarding the average performance decline of RSTIB-MI -MLP is still more superior compared with other baselines. For all the metrics, including MAE, RMSE, MAPE in 6 benchmark datasets(33 x6 6- = 18) cases), only 3 cases that RSTIB-MLP has not achieved the best or second-best results. Along with the absolute hest performance achieved by RSTIB-MIP in all cases, it is still reasonable to claim that RSTIB-MILP has better, or comparably good, robustness, while achieveing substantially improved computationally efficiency.
31

---

Information Bottleneck-guided MLPs for Robust Spatial-temporal Forecasting
K.7. Performance Comparison with Transformer-based baselines
In this section, we aim to investigate the performance comparison with large amount of parameters equipped transformer hased baselines with ASTIB-MLPs. PDFormer (Jiang et al., 2023a), STAEformer (Liu et al., 2023) are chosen as the baseline models. which are designed for spatial-temporal traffic forecasting, thus the results for the PEMS08 and PEMS04 datasets provided in Table 17 and Fig.1I.
PEMS04
PEMS08
SiALformmrr MAE PDFormer RSTIB-MUP
S1ALtormer MAL
PDFormer MAE
40
35
30
25
20
15
0
0.1
0.3
0.5
0.1
0.3
05
Noise Ratio
Noise Ratio
Figure 11. MAE metric of different baselines on PEMS04 and PEMS08 when Subjecting to Noises
As expected, it is observed that RSTIB-MLP is less effective under cleaner conditions, while it shows superior performance as the noise ratio increases. This suggests that STAEformer and PDFormer may experience faster degradation in performance under noisy conditions due their complex architectures, which also pointed out in (Yi et al., 2024). Thus, this analysis aids in understanding the trade-off between efficiency and robustness against noise when selecting models for practical deployment.
Table 17. Performance Comparison with Transformer based Baselines
Noise Ratio
0% (clea))
10%
30%
50 %
Metrics
MAE
RMSE
MAPE
MAE
RMSE
MAPE
MAE
RMSE
MAPE
MAE
RMSE
MAPE
Dataset
PEMS08
STAEformer
13,50
23.11
8.96%
16.81
26.09
14.89 %
25.38
42.26
15.62%
45.30
63.89
32.69%
PDFormer
13.64
23.54
9.09%
20.21
32.35
15.52%
28.23
43.22
17.84%
$2.20
67.12
35.26%
RSTIB-MLP
14.51
24.18
9.44%
19.90
31.86
12.92%
23.16
40.46
14.26 %
24.37
43.77
14.36 %
Dataset
PEMS04
STAElormer
18.27
30.38
12.10%
20.88
32.05
14.02 %
28.64
43.84
19.10%
56,20
74,20
38.02%
PDFormer
18.40
29.94% %
12.04%
24.72
38.25
16.31%
33.78
45.21
21.93%
58.32
76.23
39.45%
RSTIB-MLP
18.46
30.14
12.22%
23.64
36.44
15.22%
27.15
42.85
17.19%
27.16
43.43
17.76%
K.8. Model Interpretation Case Study
To gain deeper insights into the learned intermediate representations, tend to visualize the representations learnt by different models. Specifically, GWN (Wu et al., 2019), STID (Shao al., 2022a), STGKD (Tang et al., 2024), RSTIB-MLP are included the case models. The case study we conduct follows the steps below:
First, representations of the spatial-tempont signals in the test set are mapped into R2 space using t-SNE method for dimension reduction. Then, Gaussian Kernel Density Estimation is adopted estimate the distribution of the embeddings. The models all trained under noise perturbation with the noise ratio = 0.1. We tell from the results that baselines except RSTIB-MLD tend result in the fragmentation regions into several disconnected subspaces, or collapse into just individual region. In comparison to this, RSTIB-MLP can be more effective in organizing different spatial regions into larger subspaces with a better cohesion.
K.9. Full Hyperparameter Investigation Results
We have undertaken hyperparameter investigation, where selectively vary specific hyperparameters while maintaining the their default settings. Our investigation centers on kinds of key hyperparameters: the Lagrange multipliers RSTIB- MLB. denoted as A(a, Ay. Xa) and the distance function to calculate the noise impact indicator à. We are conducting this comprehensive study to understand how these hyperparameters influence the overall model performance. The experimental outcomes on the PEMS04 dataset are presented in Fig. 13. Here follows what have drawn from our observations: i) The Lagrange Multipliers We set A Ay: X to be the with each other and vary within the range of 10-2, 10-3 3, 10-4, 10 S, ii) Distance function calculate noise impact indicators. We explore different distance functions to calculate the impact indicator for knowledge distillation. Our options for
32

---

Information Bottlencck-guided MLPs for Robust Spatial-tempone Forecasting
0
0
32
32
64
64
64
96
96
96
96
128
128
128
0
32
64
96
128
0
32
64
96
128
32
64
96
128
32
64
96
128
Figure Model Interpretation Case Study: Representations Spatial-temporal signals in test set are mapped into a R2 space using t-SNE method for dimension reduction. Then. Gaussian Kernel Density Estimation is adopted to estimate the distribution of the omhedddings.(_earae by GWN, STID, STGKD, RSTIB-MIPP the to right respectively, under noisy PEMS04 dataset with noise ratio 30%)
MAPE vs Distance Function
RMSE vs Distance Function
vs Distance Function
12:35
MAPE
RMSE
18.53
MAE
30.22
18.52
12.30
30,20
18.51
MAPE
12.25
RMSE
MAE
18.50
30.18
18.49
12.20
30.16
18.48
12.15
18.47
30.14
MAE
Smooth L1
MSE
MAE
Smooth LL
MSE
MAE
Smooth L1
MSE
Distance Function
Distance Function
Distance Function
MAPE vs
RMSE vs
MAE vs
12.22
30.40
RMSE
18.75
MAE
30.35
18.70
MAPE
RMSE
MAE
18.65
12.18
30.25
12.16
30.20
MAPE
30.15
10.3
1c.4
1e.5
1c.2
1c.3
1c.4
16.2
1c.4
Figure 13. Hyperparameter Analysis of À and Distance Function
33

---

Information Bottleneck-guided MLPs for Robust Spatial-temporal Forecasting
the distance functions include Mean Absolute Error(MAE), Mcan Squared Error(MSEE) and Smooth L1 Loss(SmoothLI).
observe that, in PEMS04 dataset, concerning choice setting x 10 allows the MAE and RMSE values to achieve the best results. In contrast, setting À = 1 x 10 yields an optimal value for MAPE. As for selecting the distance function, using MAE as the distance function leads to the best outcomes for the corresponding MAE and RMSE metrics. Meanwhile, employing the MSE to compute the impact indicator results in optimized MAPE.
Table 18. Teacher Model Agnostic PEMS07 Dataset with Varied Noise Ratios
Noise Ratio(y)
0%
50%
90%
MAE RMSE MAPE(%)
MAE RMSE MAPE(%)
MAE RMSE MAPE(%)
STID
20.41 33.68
8.74
32.38
57.29
14.07
33.33
58,50
13.96
STGKD
20.30 34.30
8.87
32.16 56.89
14.08
34.59
59.24
14.24
STExplainer- CGiB
20.55 35.12
8.61
35.12
59.17
16.78
44.14
69.23
18.87
STExplainer
20.00 33.45
8.51
32.52
57.64
15.48
45.37
70.57
19.90
TrendGCN
20.43 34.32
8.51
36.78 57.89
23.22
55.99
80.22
32.76
STG-NCDE
20.53 33.84
8.80
33.48 58.83
16.78
46.20
66.33
21.32
Ours-t-MLP
19.93 34.11
8.36
30.74
56.02
12.95
31.36
57.50
13.08
Ours-I-STGCN
19.84 33.90
8.33
30.94
56.79
12.91
30.93
56.91
12.91
K.10. Teacher Model Agnostic Study
We assert that our superior performance independent indicates the adoption of MLP networks as the teacher model, and Ours-I-TGGCN indicates the adoption of STGCN networks as the teacher model. It is important note that the teacher models are pre-trained, with their parameters fixed during the training of the RSTIB-MLP A plausible explanation for these statisties is that aim obtain the normalized indicators, thus indicating relative relationship among time series. Consequently, the overall performance of its different teacher models does significantly influence the ASTIB-MLP' performance, allowing for a more flexible configuration.
choice teacher model. Table 18 presents our results, where Ours-I-MLP
K.11. Replication Study
This section provides the statistically significant robustness study of the RSTIB-MLP compared with some chosen basclines when subjected random initialization. We conducted multiple experiments five noisy conditions to ensure statistical significance. We report the average performance and standard deviation. The statistical outcomes of this investigation are detailed Table 19, Table 20, Table 21, Table 22, Table 23. The empirical findings indicate RSTIB-MLP's remarkable resilience to various initialization conditions.
PEMS 04/07/08 datascts, selecting five random seeds and
Table Replication Study Performance Comparison Under Noise Perturbation with Noise Ratio y 10% on three Datasets. The boldface means the best results.
Method
STID
STGKD
STExplainer--GiB
STExplainer
TrendGCN
STG-NCDE
RSTIB-MLP
Dataset
Metric
PEMS04
MAE
27.79:10.45
21.26:0.1
25.86:t0.41
21.51=0.26
23.76:00.15
25.02+0.21
23.70 = 0.50
RMSE
41.45-+0.07
37.13-0.24
38.37-+0.48
36.94-0.24
37.06-+0.27
37.31+0011
36.88 + 0.44
MAPE(%)
17.4140.37
16.13=0.33
16.004+0.1 17
16.01=0.29
17.66-40.05
17.48:0..7
15.33 - 0.22
PEMS07
MAE
27.87-+0.22
26.89-0.22
27.98-10.44
13.26-0.30
26.76-40.25
28.8740.41
26.64 : 0.27
RMSE
45.10:10.45
03.64.0.45
43.80:10.35
44.07=0.15
44.70:10.03
44.35.10.41
43.82 :: 0.48
M)
12.33.10.42
14.61.10.01
14.19.0.05
11.55
PEMSOS
MAE
20.26:0.44
20.78=0.11
23.53:1:.21
20.22=0441
20.65:10.21
21.2910.25
19.91 :: 0.02
35.43.10.11
32.04
MAPE/S
11.17:0.11
15.11=0.14
21.28:2.211
13.52=0.09
11.92:40.45
15.27=0.04
13.10 = 0.36
34

---

Information Bottleneck-guided MLPs for Robust Spatial-temporal Forecasting
Table 20. Replication Study for Performance Comparison Under Noise Perturbation with Noise Ratio ? y= 30% three Datasets, The
boldface means the best results.
Method
STID
STGKD
STExplainer-CCGB
STExplainer
TrendGCN
STG-NCDE
RSTIB-MLP
Datasel
Metric
PEMSO
MAE
36.46-10.50
28.64-0.09
31.78:10.39
0.49
27.16-10.44
29.10:0.11
27.31 + 0.05
RMSE
52.60+00.05
44.85-0.12
48.47+40. 18
46.98-0.07
42.96+00,1
44.36+0042
43,03 + 0.47
M)
21.23:40.43
17448=0.07
18.09:b..1 16
17.98=0445
19.43:4043
19:26:0002
17.21 = 0.44
PEMS07
MAE
31.87+40.49
31.89-0.39
34.89-40.20
31.43-0.23
31.82++0,11
32.29++0.19
M1.29 + 0.15
RMSE
55.21.10.03
53.24_10.29
$7.44.10.16 16
54.21-0.09
55.23-10.40
56.2910.04
54.14 1:
M)
13.63-10.07
14.98 0.27
20.68.10..
PEMSOS
MAE
26.61:1044
25.63=0.45
24.8810.38
25.00=0,30
24.90:10442
28.41:1:440
23.24 ::
RMSE
15.79=0.08
13.16=0.228
43.15:0.15
13.37=0.37
41.68:0040
11.99=0.10
40,47 0.02
MAPE(S)
15.61:0.35
16. 16.21=0.29
15.1440.21
16.75=0.47
23.55:40.88
16.2920.45
14.38 = 0.27
Table 21. Replication Study for Performance Comparison Under Noise Perturbation with Noise Ratio y 50% on three Datasets. The boldface the hest results.
Method
STID
STGKD
STExplainer-CC
STExplainer
TrendGCN
STG-NCDE
RSTIR-MI
Daraset
Metric
PEMS04
MAE
36.26:0040
29. 29.13=0.29
18.1080.00
15=155044
27.90:10.05
31.06:0.29
27.29 =:
52.3211.21
44.74110.05
46.50=0.46
44.1110.15
47:2910.36
43.60
MAPE(R)
21.42.10.01
18.56_10.33
16.88.10.30
17.23..0.34
20.31.10.39
20.7410.41
17.60 0.12
PEM107
MAE
32.56++0.15
32.10-0.28
35.08-+0.44
32.53-0.49
36.82-+0.11
33.34+-0.39
30.97 + 0.15
RMSE
37.40+0.00
56. 0..03-0.30
59.06+t0.1 17
67.47=0.16
57.72++022
18.99+0.24
56.92 + 0.22
())E(%)
13.91:10.2
11.02=0.33
16.75:20.26
15.11=0.47
23.11#0.11
16. 16.76:0.33
12.98 = 0.39
PEMS08
MAE
27.73+40.46
25.88-0.24
26,63++032
27. 17.16-0.02
26,86++0,25
29.40+0.001
24.47 + 0.24
RMSE
48.49.10.24
43.94.10.16
44.60.10.43
45.70.00.33
45.67.10.22
47.13.10.25
43.84 0.26
MAPF(S)
16.2740.26
16.77-0.37
15:30-0.40
23.15-+0.44
18.45+00.77
Table 22, Replication Study for Performance Comparison Under Noise Perturbation with Noise Ratio y 70% three Datasets. The boldface means the best results.
Method
STID
STGKD
STEExpainer--GGBB
STExplainer
TrenGGCN
STG-NCDE
RSTIB-MLP
Datasel
Metric
PEMSO
MAE
31.34+0.47
29.86-0.48
38.56-+0.38
33.31-0.33
33.07-40.34
33.21++0.09
27.08 + 0.26
RMSE
47.12440.2
46.06=0.30
58.65 28.65-+0.24
52. 14
51.0440.42
49.79++0.27
43.12 + 0,22
MAPE(%)
18.1940..1
17.00=0.04
21.69+0.227
19.26=0.25
21.38:4.02
22.21+0.37
17.42 =
PEMS07
MAE
32.42-10.03
32 80
41.4810.41
43.83=0.09
43.07:10.05
13.25:0032
31,02 : 0.25
58.15.10.28
57.58
63.26..0.99
MAPE(S)
13.80++0.47
14.17=0.22
17.30-40.23
7.38
29.23-40.28
18.76+0.28
13,01 0.20
PEM108
MAE
26.32+00.00
25.02=0.08
28.28++0.99
16:1094
32.49:+0.40
30.35+:0115
24.36 + 0.30
RMSE
15.72=0.22 22
15. 05.55=0.36
15.83+0002
16. 19.96-0222
47 47.82:+0.25
19.25#0.22
43.70 = 0.50
MAPE(%
16.5110.18 18
15.04=0.100
17.55:10.25
16.86=0.07
28.25:10.20
20.15:00000
14.24 =: 0.28
Table 23. Replication Study for Performance Comparison Under Noise Perturbation with Noise Ratio 7 90% three Datasets. The boldface means the best results.
Method
STID
STGKD
STEExplainer-COS
STExplainer
TrendGCN
STG-NCDE
RSTIR-MILP
Darnsct
Metric
PEMS04
MAE
43.6221006
20.000
33.11:10.99
31.37=0.44
16.22:10.12
37.0620.42
28.11 0.29
RMSE
49.02-10.41
0..89-0.37
49.52.10.09
53.38-0.26
67.66:10.03
56.11:10400
44.67 1: 0.41
MAPE(%)
20.37.10.26
17.88.1.22 10.21
24.49.10.46
26.70-..44 0.41
38.52.10.33
25.79.10.24
17.05
0.36
PEMS07
MAE
33.42-40.05
34.77=0.39
44.25-+0.1 13
45.40-0.49
56.86-+0.09
46.33+0.07
31,00 + 0.43
RMSE
58.3940.1 12
59.33=0.00
69.36+t0.2
70. 70.63:0-43
80.23:+0.1
66.36:0.40
56.96 = 0.16
MAPE(%)
14.13:1.32
14.09=0.32
19.0110.1 14
19. 19.95=0.17
32.67:10.05
21.25::::0
13.10 =
PEMSOS
MAE
26.50.10.27
25.37
32.71
28. 28.72.00.17
46.71.10.. 1..48
32.59
24.43 0.30
45.26-10.11
44.04
()
15.9740.03
15.64-0.37
20.044+0.05
18.27-0.48
38.77-+0.99
23.44-40.12
14.22 0.47
35


Reviews:
[{"review_id": "CXBjSjx2R7", "reviewer": "Reviewer_WNiV", "summary": "\"This paper theoretically motivates and implements a novel regularization technique for training MLP-based models in spatiotemporal forecasting. The MLP models are distilled from state-of-the-art architectures that leverage graph neural networks. KL divergence terms between the data (input, output, encoded) and assumed Gaussian noise levels were used in the proposed loss function. Extensive experiments demonstrate the effectiveness of the proposed approach, highlighting its potential to enhance forecasting performance", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your positive and constructive comments. We provide a point-by-point response as follows.\n\n> ### **Re: Experimental Designs Or Analyses**\n\n- **\"What are the teacher model(s) used in the experiments in Table 2?\"**\n\nTeacher model selection settings are described below Figure 4: \"Our method is teacher model agnostic (Appendix. K.10), where we set the default teacher model to STGCN.\"\n\n- **\"How will the results differ if other teacher models were used?\"**\n\nThis is discussed in Appendix K.10, where we directly compare empirical results obtained using different teacher models. Table 18 provides empirical details, and the process of preparing the teacher model is also described in Appendix K.10. We demonstrate that the superior performance of RSTIB-MLP is independent of the teacher model choice by showing that RSTIB-MLP also achieves strong robustness even when selecting MLP as the teacher model—outperforming or performing comparably to some state-of-the-art (SOTA) STGNNs. The potential reason behind this is elaborated in Appendix K.10. We will clarify this further in our final version.\n\n- **\"What is the performance of the teacher model before distillation?\"**\n\nConsistent with the previous response, we present an empirical study in Appendix K.10 to demonstrate that RSTIB-MLP’s performance is independent of the original teacher model's performance. Even when using a basic MLP as the teacher model, RSTIB-MLP can achieve better or comparably good robustness relative to some SOTA STGNNs.\n\n- **\"Was this a distillation using only the output of the network, or were intermediate features used?\"**\n\nOnly the output of the network is utilized. Specifically, **Definition 4.9** clarifies that the teacher model's output is used exclusively to compute the noise impact indicator.\n\n> ### **Re: Other Strengths And Weaknesses**\n\n- **\"What loss function is used in the MLPs?\"**\n\nBelow is our loss function:\n\n$$\n\\mathcal{L}_{RSTIB\\text{-}MLP} = \\sum _ {i=1}^{N}\\left[-\\mathcal{L} _ {\\text{reg}}(Y_i^S, \\tilde{Y}_i)\\right] + \\sum _ {i=1}^{N}(1 + \\hat{\\alpha} _ i)(\\lambda _ x \\mathcal{L} _ {x,i} + \\lambda _ y \\mathcal{L} _ {y,i} + \\lambda _ z \\mathcal{L} _ {z,i})\n$$\n\n\n\nThis is also described in Eq.(6). Specifically, $\\mathcal{L} _ {\\text{reg}}(Y _ i^S, \\tilde{Y} _ i)$ is the lower bound of $I(Z;\\tilde{Y})$. Please refer to Proposition 4.8 for implementation details and proofs. Additionally, $\\mathcal{L} _ {x,i}$, $\\mathcal{L} _ {y,i}$, and $\\mathcal{L} _ {z,i}$ represent the corresponding regularization terms applied to the input, target, and representation regions respectively. For their analytical calculations, please refer to Proposition 4.6 and Proposition 4.7. Descriptions of how we implement these regularizations are provided above Proposition 4.6 (for input and target) and Proposition 4.7 (for representation regularization). Moreover, $\\hat{\\alpha}$ is the noise impact indicator defined in Definition 4.9, serving to dynamically adjust the training of RSTIB-MLP when incorporating these regularization techniques.\n\n- **\"Were the teacher models pretrained or trained from scratch? In general, more details on the distillation procedure should be given.\"**\n\nRelevant details are included in Appendix K.10 due to space limitations. We apologize for the unclear statement in the main text. The teacher model is trained from scratch by ourselves. The knowledge distillation procedure is implemented as follows:\n\n1). Train the teacher model from scratch, following the same procedure as the student model.\n\n2). Freeze the teacher model's parameters and start training RSTIB-MLP.\n\n3). Only the output of the teacher model is leveraged to calculate the noise impact indicator as defined in Definition 4.9.\n\nAccording to Definition 4.9, the noise impact indicator is normalized using the Softmax function, reflecting the relative relationships among time series.\n\n> ### **Re: Other Comments Or Suggestions**\n\nMany thanks and we really appreciate for the suggested idea! Yes, the efficiency is one key advantage that we aim to demonstrate. Therefore, we include \"MLPs\" in our title to stress the point of our method. We will consider this constructive suggestion in our final version.\n\nWe hope we have addressed all your concerns. Thank you very much!", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Comparison with a wide range of spatio temporal algoirhms demonstrate the advantages of the proposed method across different settings.", "theoretical_claims": "No", "experimental_designs_or_analyses": "Experimental results show the robustness of the propsoed method across several noise levels and also in the case of clean signal (the clean signal normally have small amount of noise in the measurmenets and the proposed method was able to improve the performance in these cases too). \n\nHowever, I'm not clear on the details of the distillation process used and this is a critical point in this paper. What are the teacher model(s) used in the experiments in Table 2? How will the results differ if other teacher models were used? What is the perfomrance of the teacher model before distillation? Was this a distillation using only the output of the network or were intermediate features used?", "relation_to_broader_scientific_literature": "The proposed loss function can have implications that goes beyond weather forecasting and extend to applications using sensor based measruements in general.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "The paper is missing important implmentation details:\n1. What loss function is used in the MLPs?\n2. Were the teacher model pretrained or you trained them from scratch? In general more details on the distillation procedure should be given.", "other_comments_or_suggestions": "I suggest changing the paper title as I think it should have an emphasis on efficiency. How about using the answer to the question you proposed in the abstract as title: \"can simple neural networks such as..\" or something similar as a title", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "joZjj9FBhx", "reviewer": "Reviewer_ZfWm", "summary": "The authors disclose the dual noise effect behind the spatial-temporal data noise, and propose theoretically-grounded principle termed Robust Spatial-Temporal Information Bottleneck (RSTIB) principle, which preserves wide potentials for enhancing the robustness of different types of models. Comprehensive experimental results show that an excellent trade-off between the robustness and the efficiency can be achieved by RSTIB-MLP compared to state-of-the-art STGNNS and MLP models.\n\n## update after rebuttal\n\nThe author's detailed response has largely addressed my initial concerns, so I will maintain my positive assessment.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your positive and constructive comments. We provide a point-by-point response as follows.\n\n> ### **Re: W1**\n\nPlease note that the derivation of the RSTIB principle and the implementation of RSTIB-MLP do not make assumptions regarding the noise type. AWGN is employed in our experiments primarily because it is commonly adopted as a noise model in information theory and experimental measurements within time series forecasting ([Ref1]).\n\nTo better address this concern, we have provided further empirical study in Appendix K.3. This section details how RSTIB-MLP handles data missing scenarios compared with selected baselines, and also the contribution of each module under such noisy conditions.\n\n> ### **Re: W2**\n\nIn our settings, the Lagrange multipliers are set to be constants. This means that the balance between preserving the target and applying regularization is fixed across different time series, which may not be optimal. Besides, regularization further limits the capacity of the MLP, preventing the model from fully learning complex features in the data. Therefore, we aim to better balance the preservation of the target and the regularization terms across different time series. \n\nInspired by the observation that the feature variance will largely decrease as the noise ratio increases, we leverage the KD module based on the noise impact indicator information to achieve what has been described in the **Training Regime**: When noise impact is low, we relax the KD-based regularization; when there is a significant noise impact, we intensify the KD-based regularization. As a result, the feature variance can be increased by KD in the cases when the noise impact indicator shows high noise impact. Please note that noise impact indicators quantify the noise impact on each time series and this noise impact information is used to dynamically tune/adjust the optimization of our model.\n\nHope that we have addressed all your concerns. Thank you very much!\n\n[Ref1] Teck Por Lim and Sadasivan Puthusserypady. Chaotic time series prediction and additive white Gaussian noise. *Physics Letters A*, 365(4):309–314, 2007.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "The authors summarize the previous work in spatial-temporal prediction and point out the existing problems, which are also the challenges that this paper aims to solve.", "essential_references_not_discussed": "The references are comprehensive.", "other_strengths_and_weaknesses": "Strengths:\n\n1. The RSTIB principle is based on information theory and provides a solid theoretical foundation for model design.\n\n2. In the part of experimental evaluation, the baselines and datasets used are relatively comprehensive, especially some large-scale datasets.\n\n3. The proposed RSTIB-MLP achieves SOTA predictive performance while requiring fewer computing resources.\n\nWeaknesses:\n\n1. The noise in the data is assumed to be additive white Gaussian noise (AWGN). Does the derivation hold if the noise type is not AWGN?\n\n2. How does knowledge distillation enhance the aspect of feature diversity? The author should further clarify its principle.", "other_comments_or_suggestions": "See Weaknesses.", "questions_for_authors": "Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Clh2AjZecp", "reviewer": "Reviewer_qjmA", "summary": "This paper introduces a novel MLP training method based on the Information Bottleneck principle, termed RSTIB-MLP, designed to address the balance between model efficiency and robustness. By analyzing the dual noise effect in STgraph data, authors propose the Robust Spatiotemporal Information Bottleneck (RSTIB) principle. This principle relaxes the Markov assumption of IB and explicitly minimizes the impact of noisy information. In its implementation. Experimental results demonstrate that RSTIB-MLP outperforms other counterparts in robustness against noise interference across multiple spatiotemporal benchmarks, while maintaining higher computational efficiency.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> ### **Re: Supplementary Material**\n\nWe are sincerely sorry for the typos, and will correct them in our final version. \n\nP16: \"illustrated in Fig.7(a)\"; \"represented in Fig.7(b)\". \n\n> ### **Re: Essential References Not Discussed**\n\nAppendix K.7 has examined such scenarios where transformer-based models with a very large number of parameters (**PDFormer**and **STAEformer**) are compared.\n\nPlease note that achieving new **SOTA performance** is not our claim. Instead, we dedicate to achieving a good trade-off between robustness and efficiency, which is demonstrated with SOTA STGNNs ((**iii**) above **Related Work**). We will include and discuss these three papers and further clarify this point in our final version.\n\n> ### **Re: W1**\n\nIt is evaluated theoretically (Appendix H) and empirically (Section 5.3).\n\nBelow we additionally evaluate the overall training to convergence time (TTCT) in PEMS04 and the training time per epoch(TTPE) on the large Weather2K-R dataset:\n\n|Method|TTPE(s)|TTCT(s)|\n|-|-|-|\n|RSTIB-MLP|67.2|2842.3|\n|DSTAGNN|1050.8|9283.7|\n|Graph-WaveNet|556.2|7308.6|\n|STG-NCDE|1436.1|9238.7|\n|STExplainer|1747.6|24514.0|\n\nIt is clear that our full training is much faster; e.g., RSTIB-MLP reduces up to 88.42% compared to STExplainer. Besides, the superior efficiency of RSTIB-MLP is consistent on the large dataset.\n\nThese results will be included to ensure comprehensiveness.\n\n> ### **Re: W2**\n\n(i) Please refer to **Re: W1** in the Rebuttal for Reviewer **ZfWm**.\n\n(ii) We follow the setting of RGIB as follows:\n\n- The train set is added with input noise and target noise.\n- The validate and test sets are added with the same set of input noise, while their targets remain clean (unchanged) to ensure accurate evaluation.\n\nAll methods follow these settings.\n\n> ### **Re: W3 & Q3**\n\nYes! we'll leave rooms in the main text for below:\n\n- **Assumption 4.1:**\n\nThe sliding window mechanism is a technique for processing ST data, extracting fixed-length subsequences from raw data by progressively sliding a window over temporal or spatial-temporal dimensions. A critical feature is that the same data window can flexibly serve as either input or target, creating a \"*dual noise effect*\"—when a noisy sequence serves as both the input \\(X\\) in one window and the target \\(Y\\) in another, noise propagates bidirectionally. If \\(Z - X - Y\\) holds, then \\(I(Z; Y|X) = 0\\): The noisy information behind \\(I(Z; Y|X)\\) is directly ignored. Ignoring noise in \\(Y\\) is therefore problematic. The dual noise effect allows noise influences both input and target across overlapping windows, necessitating relaxation of \\(Z - X - Y\\).\n\n- **Assumption 4.2:**\n\nST graphs exhibit invariant patterns (generalizable across time) and variant patterns (node-specific, time-varying dynamics). Invariant patterns might represent structural dependencies (e.g., road connectivity in traffic prediction), whereas variant patterns could reflect transient events (e.g., traffic congestion due to accidents). Data dynamics thus also depend on the current window's characteristics, meaning the prediction for \\(Y\\) is not entirely determined by \\(X\\), but also by \\(Y\\)'s unique dynamics. Therefore, the assumption \\(Z - X - Y\\) requires relaxation.\n\n> ### **Re: Q1**\n\nPlease refer to **Re: Experimental Designs or Analyses** in the rebuttal for Reviewer **WNiV**.\n\n> ### **Re: Q2**\n\nTo achieve this, a method calculating the noise impact indicators without supervised signals is required. Below provides one possible alternative unsupervised solution to our KD approach:\n\nFirstly, perturb the noisy \\\\(X\\\\) with noise to generate \\\\(X_{\\\\text{perturb}}\\\\). Then:\n\n$$\n\\hat{\\alpha_i} = \\frac{\\exp\\left(D\\left(X_{\\text{perturb}, i}, X\\right)\\right)}{\\sum_{j=1}^{N} \\exp\\left(D\\left(X_{\\text{perturb}, j}, X_j\\right)\\right)}, \\quad \\forall i \\in \\{1, \\ldots, N\\}\n$$\nThe following training process is similar to RSTIB-MLP with (w/) KD.\n\nWe investigate perturbation ratios of 0.1, 0.3, 0.5, and noise ratios of 0.1, 0.3, 0.5 to compare RSTIB-MLP with KD. Best results for the unsupervised method (RSTIB-MLP w/ Aug) across all perturbation ratios are reported below:\n\n|Noise Ratio|RSTIB-MLP w/|MAE|RMSE|MAPE(%)|\n|-|-|-|-|-|\n|0.1|KD|23.64|36.44|15.22|\n||Aug|24.02|36.76|15.55|\n|0.3| KD |27.15|42.85 |17.19|\n||Aug|27.62|43.73|17.69|\n|0.5|KD|27.16|43.43|17.76|\n|| Aug|27.86|44.54|18.32|\n\nResults demonstrate that RSTIB-MLP w/ Aug cannot outperform RSTIB-MLP w/ KD. Potential reasons(risks):\n\n- Noise ratio is difficult to characterize precisely. Fixed perturbation ratios and augmented \\\\(X_{\\\\text{perturb}}\\\\) samples may not align with real scenarios. Even adjustable ratios still pose this risk.\n\n- Using only information from \\(X\\) neglects dynamics unique to \\(Y\\) (Assumption 4.2). Calculating the noise impact indicators solely from \\(X\\) thus yields relatively inaccurate quantification.\n\nThis underscores the importance of our KD approach.\n\nWe hope this addresses all your concerns. Thank you very much!", "claims_and_evidence": "Yes. Authors use clear algorithms and experiments to support their claims.", "methods_and_evaluation_criteria": "Yes, they use classic ST benchmarks to formula their experiments.", "theoretical_claims": "Yes, I have checked the correctness.", "experimental_designs_or_analyses": "The experiment design is soundness. However, there are uncertainties regarding the correspondence between simulated noise scenarios and real-world conditions, and substantial challenges exist in validating the model's noise resistance capabilities.", "relation_to_broader_scientific_literature": "This study addresses an intriguing research question regarding model robustness in noisy scenarios, a topic that has received limited attention in prior research.", "essential_references_not_discussed": "How do these recent spatio-temporal prediction methods perform under this paper's experimental settings? \n\n[1] PDFormer: Propagation Delay-Aware Dynamic Long-Range Transformer for Traffic Flow Prediction. AAAI 2023.\n\n[2] UrbanGPT: Spatio-Temporal Large Language Models. KDD 2024.\n\n[3] UniST: A Prompt-Empowered Universal Model for Urban Spatio-Temporal Prediction. KDD 2024.", "other_strengths_and_weaknesses": "**Strengths**\n\n1. The study employs a diverse and extensive set of experimental datasets.\n\n2. Experimental results demonstrate the proposed method's effectiveness in handling data noise under specific scenarios.\n\n**Weakness**\n\n1. The study lacks assessment of computational efficiency. How does the proposed method's computational performance compare with the baselines?\n\n2. While the proposed methodology validates noise resistance by artificially introducing noise into the original data, several critical considerations warrant discussion:\n\n(i) Real-world scenarios present significant challenges in identifying noise occurrence patterns and frequencies, making noise characterization inherently difficult.\n\n(ii) Both training and testing datasets contain noise, which seemingly complicates the validation of the model's 'noise resistance capability'.\n\nHow do the authors interpret and address these methodological challenges?\n\n3. The assumption for relaxing the Z−X−Y restriction is vital for authors' RSTIB-MLP, but the discussion part of it is only in appendix. Could you please provide more details of it in the main text?", "other_comments_or_suggestions": "Please refer to Supplementary Material", "questions_for_authors": "1. The knowledge distillation module dynamically adjusts the regularization strength through the noise influence index (Definition 4.9). This index is calculated based on the prediction error of the teacher model. However, will the choice of the teacher model (such as STGCN) lead to significant deviations in the results? \n\n2. Have unsupervised or self-supervised methods been attempted to replace the teacher model? The process by which the authors derived their algorithm seems to have nothing to do with Knowledge Distillation (KD). Can the final loss function be directly applied to unsupervised learning?  \n\n3. The assumption for relaxing the Z−X−Y restriction is vital for authors' RSTIB-MLP, but the discussion part of it is only in appendix. Could you please provide more details of it in the main text?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SDZlrV1jgO", "reviewer": "Reviewer_Mrf2", "summary": "This paper proposes the Robust Spatio-Temporal Information Bottleneck (RSTIB) principle to enhance the robustness of spatio-temporal prediction models to noise interference. The authors introduce RSTIB-MLP, a multi-layer perceptron (MLP) based implementation that achieves state-of-the-art performance in the face of noise interference. Extensive experiments on multiple datasets demonstrate the robustness and efficiency of the proposed model.\n\n## update after rebuttal\n\nI have read the overall feedback and the author has addressed my concerns, but some sections need to be rewritten and the final version needs to be carefully revised.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your positive and constructive comments. We provide a point-by-point response as follows.\n\n> ### **Re: Claims And Evidence & Experimental Designs Or Analyses & W1 & W2 & C2 & Q1**\n\nDue to space limits, please refer to **Re: W1** in the Rebuttal for Reviewer **ZfWm**.\n\n> ### **Re: Methods And Evaluation Criteria & Supplementary Material & W3 & Q2**\n\n- **KD's role in enhancing feature diversity:**\n\nDue to space limits, please refer to **Re: W2** in the Rebuttal for Reviewer **ZfWm**.\n\n- **KD's impact on robustness:**\n\nWe observed performance gains by incorporating the KD module. The reason behind could be that KD dynamically tunes the balance across different time series, enabling MLPs with limited capacity to favor information containing less noise. However, the improvement in predictive performance is less prominent compared to robustness enhancements brought by the RSTIB, because in all situations, we still need to balance the preservation of the target and the compression of all the reparameterization. Intrinsically, although KD can tune the relative ratio, the improvement regarding predictive performance is still constrained by the objective itself.\n\n- **The impact of different regularization terms:**\n\nWe provided further ablation studies and discussions on the respective roles of the regularization terms in Appendix K.2. Please refer to that section for more details.\n\n> ### **Re: Relation To Broader Scientific Literature**\n\nAppendix I.2 discusses the distinctions between RSTIB/RSTIB-MLP and existing IB methods. Please refer to that section for more details.\n\n> ### **Re: Essential References Not Discussed**\n\nWe highly appreciate the suggested papers! Below we provide a detailed discussion on the relation between our submission and the cited works:\n\n- We are inspired that we could include an additional part for introducing the robust spatial-temporal forecasting: RobustTSF ([1]) considers time series forecasting with anomalies, while our work similarly considers spatial-temporal forecasting with noise perturbation.  We share a similar way of experimental conduction: artificially introducing noise and data missing (specifically mentioned in **Essential References Not Discussed**).\n Including this paper can enhance our experimental settings and results' credibility.\n\n- Linking feature diversity with spatiotemporal heterogeneity from [2] is really intriguing. Both works consider robustness in spatial-temporal forecasting. Our work attempts to enhance the feature diversity, while [2] aims to capture the heterogeneity. Our work quantifies feature diversity using feature variance(*Var*), [2] quantifies heterogeneity using the Average Node Deviation (AND) metric:\n\nGiven the feature matrix $X \\in \\mathbb{R}^{n \\times d}$, the AND metric is defined as:\n\n$$\nD(X) = \\frac{1}{n^2}\\sum_{i=1}^n \\sum_{j=1}^n \\sum_{k=1}^d (x_{ik}-x_{jk})^2\n$$\n\nTo link it with *Var*, we further derive $D(X)$ as below:\n\n\\begin{aligned}\nD(X) \n&= \\frac{1}{n^2} \\sum_{i=1}^n \\sum_{j=1}^n \\sum_{k=1}^d (x_{ik} - x_{jk})^2 \\\\\\\\\n&= \\frac{1}{n^2} \\sum_{i,j,k} \\left( x_{ik}^2 + x_{jk}^2 - 2x_{ik}x_{jk} \\right) \\\\\\\\\n&= \\frac{1}{n^2} \\left( 2n \\sum_{k,i} x_{ik}^2 - 2 \\sum_{k}\\left( \\sum_{i} x_{ik} \\right)\\left( \\sum_{j} x_{jk} \\right) \\right) \\\\\\\\\n&= \\frac{2}{n^2} \\left( n \\sum_{k,i} x_{ik}^2 - n^2 \\sum_{k} \\bar{x}_k^2 \\right) \\\\\\\\\n&= \\frac{2(n-1)}{n} \\sum _ {k=1}^d \\left( \\frac{1}{n-1} \\left( \\sum _ {i=1}^n x _ {ik}^2 - n\\bar{x} _ k^2 \\right) \\right) \\\\\\\\\n&= \\frac{2(n-1)}{n} \\cdot \\text{tr}(\\mathbf{Cov}),\n\\end{aligned}\n\nwhere $ \\bar{x} _ k = \\frac{1}{n} \\sum _ {i=1}^n x _ {ik} $ and $ \\text{tr}(\\mathbf{Cov}) = \\sum_{k=1}^d \\frac{1}{n-1} \\left( \\sum_{i=1}^n x_{ik}^2 - n\\bar{x}_k^2 \\right) $.\n\nThus, the AND metric is proportional to the trace of the covariance matrix $\\mathbf{Cov}$:\n\n$$\nD(X) \\propto \\text{tr}(\\mathbf{Cov})\n$$\n\nThe AND metric emphasizes overall variance, whereas our *Var* metric emphasizes balanced standard deviations. \n\nWe'll extend the above in our final version.\n\n> ### **Re: C1**\n\nMany thanks for this constructive advice! We suspect that the mentioned \"extreme noise conditions\" refer to real-world noise experiments (as mentioned in **C2**), such as data-missing scenarios. We have actually provided such evaluations and discussions in Appendix K.3. Your suggestion is helpful as it guides us towards more focused and clear settings. Nevertheless, by including robustness studies under diverse noise conditions, we have also demonstrated the consistent performance of our method.\n\n\n\nWe hope this addresses all your concerns. Thank you very much!", "claims_and_evidence": "The claims are generally supported by clear evidence. The authors provide a theoretical framework for RSTIB and demonstrate its effectiveness through experiments on noisy datasets. However, the assumption of additive white Gaussian noise (AWGN) is restrictive and may not generalize to other noise types (I understand that this is for the sake of good relaxation of the theoretical derivation). In particular, real-world data may be noisy in nature with spatio-temporal entanglement, or simply missing.", "methods_and_evaluation_criteria": "The proposed RSTIB-MLP method and evaluation criteria are applicable to the problem. The authors use standard metrics (MAE, RMSE, MAPE) and benchmark datasets for spatio-temporal prediction. The inclusion of the knowledge distillation module is innovative, but would be better if its role in enhancing feature diversity was explained more clearly.", "theoretical_claims": "Although I have not examined them carefully, I believe the theoretical claims are sound. The authors correctly derive the RSTIB principle by lifting the Markov assumption and provide proofs for the key propositions.", "experimental_designs_or_analyses": "The experimental design is comprehensive but has some limitations. The authors demonstrate that RSTIB-MLP outperforms state-of-the-art methods under noisy conditions. However, the experiments mainly use artificially added noise, which may not fully represent real-world scenarios.", "relation_to_broader_scientific_literature": "This paper builds on the existing information bottleneck (IB) method and extends it to handle the dual noise effect in spatiotemporal prediction. This work is related to recent advances in robust representation learning and applies these ideas to real-world problems. However, the authors should do a better job of distinguishing their contribution from existing works, especially those cited in the paper.", "essential_references_not_discussed": "This paper cites related works, but could benefit from discussing recent advances in robust machine learning and spatiotemporal forecasting. For example, recent works on adversarial training of time series data or methods for handling missing data in spatiotemporal models could provide additional context [1]. In addition, feature variance is used to quantitatively analyze the diversity among learned features, which can also be interpreted as spatiotemporal heterogeneity in the spatiotemporal forecasting scenario, as shown by a recent paper [2].\n\n[1] Cheng, Hao, et al. \"RobustTSF: Towards theory and design of robust time series forecasting with anomalies.\" ICLR 2024.\n\n[2] Chen, Wei, and Yuxuan Liang. \"Expand and Compress: Exploring Tuning Principles for Continual Spatio-Temporal Graph Forecasting.\"  ICLR 2025.", "other_strengths_and_weaknesses": "### **Strengths:**\n\n**S1.** The paper introduces a novel theoretical framework (RSTIB) that extends the information bottleneck principle to handle dual noise effects for spatiotemporal forecasting applications.\n\n**S2.** The experiments are comprehensive, covering multiple benchmark datasets and comparison baselines.\n\n**Weaknesses:**\n\n**W1.** Overall, my biggest concern is that its narrow setting affects the broad interest of the community. Specifically, the assumptions of AWGN are restrictive and may not generalize to other noise types.\n\n**W2.** The experiments mainly use artificially added noise, which may not fully represent real-world noisy scenes.\n\n**W3.** The role of knowledge distillation in enhancing feature diversity has not been fully explored.", "other_comments_or_suggestions": "**C1.** Basically, I think this is a good paper in the field of spatio-temporal forecasting, but its limitation lies in its diverse settings and motivations, which reduces its audience. I actually suggest that the authors change it to robust spatio-temporal forecasting under extreme noise conditions (use this as motivation to revise the writing story and introduce the theoretical framework), which I believe will win wider attention.\n\n**C2.** Therefore, the authors can include more real-world noise experiments (e.g., the spatiotemporal data missing setting of the reference article [3]) to verify the robustness claims.\n\n[3] Cini, Andrea, Ivan Marisca, and Cesare Alippi. \"Filling the g_ap_s: Multivariate time series imputation by graph neural networks.\" ICLR 2022.", "questions_for_authors": "**Q1.** How does the RSTIB principle handle noise types other than AWGN? Can the authors provide insights into robustness under different noise models?\n\n**Q2.** Can the authors elaborate on the role of knowledge distillation in enhancing feature diversity and its impact on robustness?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "not applicable"}}]

Author rebuttal:

Meta-review: This is an interesting and innovative idea. The reviewers reached a consensus in their scores (cf., the specific reviews below).
Decision: Accept (poster)