Causal Spatio-Temporal Prediction: An Effective and Efficient Multi-Modal Approach
Authors: Yuting Huang
Ziquan Fang
Zhihao Zeng
Lu Chen
Yunjun Gao
Venue: NeurIPS 2025 poster ()
Keywords: Spatial-temporal prediction; multi-modal data; causal inference
Primary area: deep_learning

Abstract: Spatio-temporal prediction plays a crucial role in intelligent transportation, weather forecasting, and urban planning. While integrating multi-modal data has shown potential for enhancing prediction accuracy, key challenges persist: (i) inadequate fusion of multi-modal information, (ii) confounding factors that obscure causal relations, and (iii) high computational complexity of prediction models. To address these challenges, we propose E$^2$-CSTP, an Effective and Efficient Causal multi-modal Spatio-Temporal Prediction framework. E$^2$-CSTP leverages cross-modal attention and gating mechanisms to effectively integrate multi-modal data. Building on this, we design a dual-branch causal inference approach: the primary branch focuses on spatio-temporal prediction, while the auxiliary branch mitigates bias by modeling additional modalities and applying causal interventions to uncover true causal dependencies. To improve model efficiency, we integrate GCN with the Mamba architecture for accelerated spatio-temporal encoding. Extensive experiments on 4 real-world datasets show that E$^2$-CSTP significantly outperforms 9 state-of-the-art methods, achieving up to 9.66% improvements in accuracy as well as 17.37%-56.11% reductions in computational overhead.

Causal Spatio-Temporal Prediction: An Effective and Efficient Multi-Modal Approach
Yuting Huang, Ziquan Zhihao Zeng, Lu Chen, Yunjun Gao
Zhejiang University
(huangyuting, zqfang, zengzhihao, luchen, gaoyj] edu en
Abstract
Spatio-temporal prediction plays a crucial role in intelligent transportation, weather forecasting, and urban planning. While integrating multi-modal data has shown potential for enhaneing prediction accuracy, key challenges persist: (i) inadequate fusion of multi-modal information, (ii) confounding factors that obscure causal relations, and (iii) high computational complexity of prediction models. To address these challenges, we propose E²-CSTP, an Effective and Efficient Causal multi- modal
Prediction framework. E2-CSTP leverages cross moddl attention and gating mechanisms to effectively integrate multi-modal data. Building on this, we design mull-branch causal inference approach: the primary branch focuses spatio-temport prediction, while the auxiliary branch mitigates bias by modeling additional modalities and applying causal interventions to uncover true causal dependencies. To improve model efficiency, integrate GCN with the Mamba architecture for accelerated vratiottempora encoding. Extensive ex- periments 4 real-world datasets show that E2-CSTP significantly outperforms 9 state-of-the-art methods, achieving up to 9.66% improvements in accuracy as well 17.37%-55 1% reductions in computational overhead.
Introduction
Spatio-temporal prediction plays a critical role in numerous applications, such as intelligent transportation systems 25,5) weather forecasting (34), environmental monitoring (38), and urban planning [49] Accurate predictions in these areas help improve decision-making and optimize resource allocation. For example, accurate traffic flow forecasting can improve road safety, while reliable weather predictions facilitate effective disaster preparedness.
Meanwhile, recent advances in information technology have facilitated the proliferation of diverse data types and sources. Multi-modal data, comprising information from distinct sensing channels (e.g., satellite images, text, and sensor- based inputs), often exhibits rich mooss-mddal correlations. Effective integration of such heterogeneous data can mitigate the constraints of single-mode modal ap- proaches while providing more comprehensive spatio-temporal data representations for enhanced predictive performance 44 22 25 For instance, urban traffic forecasting benefits significantly from incorporating multi-modal inputs like surveillance image and social media text alongside traditional traffic flow data, leading to more accurate predictions (33] Despite the efforts of previous studies, we observe that several critical challenges still persist in multi-modal spatio-temporal prediction.
The first fundamental challenge lies in the insufficient integration of spatio-temporal patterns with heterogeneous on homogeneous modalities within multi-modal datasets. For instance, MoSSL (8] models taxi and bicycle inflow/outflow patterns distinct modalities for traffic forecasting. However, weal-world
data. Specifically, prior approaches 42 12 g primarily concentrate
"Ziquan Fang is the corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

spatio-temporal systems typically involve deeply interconnected multi-modal data characterized by two key relationships: semantic correlations and complementary information exchange. Fig. aT) exemplifies these relationships in traffic prediction scenarios. As observed, (i) temporal traffic flow patterns correlate with aerial image through semantic relationships and (ii) social media text (e.g.. road closure reports) provides complementary information to conventional sensor data. Although the state-of-the-ar LLM-based studies [48] 20, 30 have explored multi modti- approaches, significant limitations persist. For example, LLM-based text training [48 is modality-speciffe and lacks general- izability, whereas GPT4MTS [20] and TimeMMD [30] focus solely on temporal patterns without considering spatial dimensions. Besides, several studies [2 B employ modality to predict another. In contrast, the focus of our work lies in utilizing supplementary modalities to enhance
spatio-temporal forecasting.
The second challenge is that the causal relations in spatio-temporal prediction are typically confounded by latent variables
Semantic correlations
Taxi
Flow
Time
Bieyele
and environmental biases 170 [2 45 13
Sunmy
Rainy
Time
63.. Fig. (b) illustrates how multi-modal data can give rise to complex causal inter- actions. As observed, bicycle and taxi flows exhibit positive correlation on sunny days due to increased outdoor activ- ity, while rainy conditions invert this cor- relation, suppressing bicycle usage while amplifying taxi demand as commuters seek sheltered transportation. Fig. [T]]
(b) Confounded multi-modal relationships
Complementary modality
Residence
Commercial bub t Constraction
Flow
Social media text
High
Full Road closure
Facilit:: Off Ramp
Work Type: Bridge
Flow
(a) Modality associations
(c) Confounded sputial causal relationships
Figure 1: Multi-modal and confounded relations.
further demonstrates spatially confounded scenario. Commercial hubs like Area typically attract high traffic volumes, but ongoing construction (a latent confounder) simultaneously reduces road capacity and diverts flow adjacent Area B. By applying causal inference, true causal relations can be accurately distinguished from confounding factors, thereby improving prediction accuracy.
Morcover, after reviewing prior ppatio-temporia prediction methods, reveal that Transformer-basee models [2 351 have become the popular approach, showing promising performance. However, their quadratic complexity ( with respect to input pemporlt sequence length T imposes severe scalability challenges, as evidenced by several hours of training time for city-scale spatio-temporal prediction in our experiments. While various efficient Transformer variants have been introduced to reduce computational complexity, they often suffer from architectural limitations or degraded modeling capacity (15). Consequently, the model efficiency bottleneck significantly hinders the practical deployment on large-scale datasets, where both accuracy and computational cost matter.
Contributions. To address the above three challenges, we propose E²-CSTP, an Effective and Efficient Causal multi modti-
Prediction framework.
Tempooal Fusion. We systematically integrate various modalities (environmental images, event-related text, and spatio-temporal time-series data) through cross- modal attention and adaptive gating mechanisms. To the best of our knowledge, this is the first work to jointly model heterogeneous features in unified prediction framework, enabling comprehensive representation learning across complementary data sources.
Unified Multi-Modal
Dual-Branch based Causal Disentanglement. We introduce a dual-branch causal inference design for spatio-temporal prediction. Specifically, the main branch focuses on learning spatio- temporal patterns, while the auxiliary branch models additional modalities and leverages causal interventions to reduce confounding bias from environmental and event-related factors.
Efficient and Hybrid Model Design. We incorporate GCN and Mamba for efficient spatio- temporal encoding. Specifically, GCN captures spatial neighborhood information to reduce the computational load of global dependencies, while Mamba handles temporal dependencies to further decrease computational complexity and accelerate spatio-temporal prediction. This hybrid design achieves faster model inference while maintaining competitive model accuracy.
Extensive Experiments. Through extensive evaluations on four weal-world datasets and nine baselines, E2CCSTP achieves up to 9.66% improvement in accuracy, 17.37%-56.. 16..16..9 1% speedup in model efficiency, and demonstrates consistent robustness across varying parameter settings.

---

2 Related Work
Single-Modal Spatio-Temporal Prediction. Early approaches ||| 32 36 primarily relied on statistical models, which depended on predefined assumptions and the statistical properties of the data. With the advancement of deep learning, models based on Recurrent Neural Networks (RNN) 17 6 Convolutional Neural Networks (CNN) [4] 60], and Graph Neural Networks (GNN) 61 66 [8] [6] have shown strong capabilities in modeling temporal, spatial, and structural
dependencies, respectively. More detailed single-modal spatio-temporal prediction studies can refer to related surveys 33
leverage self-attention for global dependency modeling and excel at capturing long-range temporal correlations. However, their inherent quadratic complexity poses a major limitation for long-term time series forecast- ing [14), especially in spatio-tempordl contexts where attention must model both spatial and temporal dependencies, leading to quadratic scaling with the number of nodes and time steps. Although several Transformer variants [68 50) reduce complexity through structural simplifications, these
23
47 5. Transformer-based models 4 28 21
modifications may hinder the model's ability to capture complex patterns [5
Multi-Modal Tentoo-l Prediction. In time series analysis, multi- methods have been applied across domains such as healthcare and finance. For instance, combining heterogeneous data sources (e.g., clinical records, medical images. genomic data) enhances medical predictive accuracy 58 while integrating social and economic signals strengthens market forecasting in finance [46]. Recent work explores LLM-based multi-modal models, GPT4MTS [20 leverages both numerical and textual inputs via prompt-based learning. MM-TSFlib (3 integrates language and time series models through end-to-end training. Wang et al. 48] combine LLMs with generative models for joint reasoning over news events and time series data, improving complex event prediction. However, these approaches focus solely on temporal dynamics and neglect spatial dependencies. In spatio-temporal prediction, Deng et al. 8 adopt self-supervieed learning to capture latent patterns in mulli--oddl spatio-temporal data. Zhang et al. [2] apply AutoML techniques to model the dynamics of multi-modal meteorological data, but their method processes different traffic and weather types separately, with limited cross-modal interaction. Wang [44] utilizes multi-modal data for smart mobility prediction, yet each task relies on a single modality, falling short of unified multi-modal modeling. Zhao al. [64] incorporate POI and weather data as contextual factors to model multi- modal traffic flow. Yan et al. [53] and Zhou et al. [67] fuse diverse urban data sources, including social media and real estate information, to improve traffic speed prediction. Han et al. [66] employ pre-trained encoder and multi-modas inputs to model event impacts on traffic. Nonetheless, these models often struggle to model complex modll dependencies and mostly limited to textual inputs, lacking support for other essential data types like images.
Spatial-Temporal Causal Inference. Causal inference aims to uncover cause-effect relations among variables. Integrating causal inference into spatio-temporal forecasting enhances both interpretability and predictive accuracy in complex, dynamic environments. In time series causal discovery, Granger causality uses non- aan--armmetii estimation to identify dependencies between variables [.. but assumes full observability of relevant variables. When latent confounders hidden causal factors are present, causal representation learning becomes necessary. Methods such as iVAE 124]. LEAP [54], and GCIM 65) explore temporal causal representations by modeling latent distributions and eliminating spurious correlations. Zhao et al. [63] propose DyGNN Explainer, a dynamic variational graph autoencoder to uncover causal and dynamic relations. CaST [52] and CauSTG (70] address out-of- distribution generalization and dynamic spatial causality via implicit modeling and intervention- based learning. Recent approaches like NuwaDDnamies 45] and CaPaint [13] apply causal inference to identify causally relevant regions. However, these works are restricted to single-modal data and do not capture the causal mechanisms underlying multi-modal spatio-temporal interactions.
3 Preliminary
The commonly used notations and descriptions are summarized in Appendix A (see Table
Definition SSpatio-Temporal Data). Spatio-
data refers to a sequence of sensor observa- tions collected at discrete time intervals over a spatial graph. Specifically, the spatial graph is denoted as g = (V, A), where V is the set of N = [V] nodes, & is the edge set, and E NxN the adjacency matrix encoding pairwise spatial relationships. The spatio-temporal data is denoted as x T-' x=1 E TR. xxxx where T is the number of historical time steps, N is the number of spatial nodes, and d is the feature dimension at each node.
3

---

(1) Cross-Modal Fenture Fasion
(0) Duaf-Branch Cousal Inference
(e.Spatie- Comple-To Encoding and Decoding
The Main Branch
STED
Nommliiz
Las
Monio Black
Marii CHmmEOOIN
Spario Jomporll
Secuesse
Price Geeph
SHAP
Canoo Seeph
Manibo
Liren
Spatil Alggamm
CNN
Medal
STED
Feature
Spatial
ZizZ
SO
Tempocal Feenue
DWCC
Cunfounding- Adjustzd
LagetNor
Smrage
Remponi
Cars
Modal Ioput
Festure Extraction Finiio
Aunilar
Figure 2: The overall framework of E2CSST.
Definition 2 (Multi-Modal Spatio-Temport Data). Multi-modal spatio-temporal data extends spatio-temporal signals by incorporating heterogeneous sources information across different modalities, such as spatio-temporal sequence, event-related text, visual images. Let X X2,,....... ) represent the n modalities, where each modality consists of spatio-temporal observations defined as X; = [x;- -
z1] E R Here, T; denotes the number of time steps (i.e., the temporal resolution), N₁ the number of spatial units (i.e., the spatial resolution), and d; represents the feature dimension of the i-th modality.
Problem Statement (Multi-Modal Opatio-Tempont Prediction). Given the graph 9 and the historical T-step of multi-modal spatio-temporal data X, where X' may include spatio-temporal sequence and auxiliary modalities, we aim learn a function 0(-) that leverages X' to predict the future S-step spatio-temporal data Yς [ye 1u+ ..... ya 1+51 RXx.xxxd
4 Methodology
Framework Overview. As illustrated in Fig. 2 the E2-CSTP framework consists of three key components: (i) cross-modal feature fusion, (I) dual-branch causal inference, and (iii) spatio- temporal encoding and decoding (STED). These modules work in concert to integrate multi-modal signals and apply causal inference techniques to reduce the impact of external confounders, thereby enhancing both prediction accuracy and model interpretability. Next, we detail each module below.
4.1 Cross-Modal Feature Fusion
We propose cross-modal attention mechanism combined with fusion gating module to integrate spatio-temporal, textual, and visual signals into a unified latent representation. Unlike naive con- catenation static fusion approaches, method dynamically attends relevant modality-specific features. The fusion process proceeds through the following steps.
Step 1: Multi-modal feature extraction and alignment. Multi-modal data typically exhibits substantial differences in temporal resolution and semantic content. To address this issue, we first perform feature alignment operations across different modalities, shown in Fig. 2a).
For text data, which lacks spatial information, alignment is performed solely along the temporal dimension. Event timestamps are extracted using regular expression parsing, and each text timestamp matched incorporates both temporal and spatial dimensions. Each image is associated with timestamp 7 and spatial coordinates p. and aligned to the nearest spatio-temporal point based on combined temporal and spatial proximity. We define the unified binary alignment matrix as follows:
nearest spatio-temporal point in time. For image data, alignment
11, if j arg min (1) (k)
{
1,
if arg min Pst
M =
M
(o,
(1)
otherwise
otherwise
where MI, and M E {0, 1ITx denote the temporal and spatial alignment matrices, respectively. ((1) represents the timestamp of the i-th text image observation, and TM denotes the timestamp of
(k)

---

the k--th element in the spatio-temporal sequence. Similarly, p()) and (k) indicate the corresponding spatial coordinates of the i-th observation and the k-th spatio-temporal point, respectively.
Subsequently, the aligned features are computed by performing matrix multipiication independently across each feature channel d. The text features are then duplicated Nat times to match the target dimensionality, resulting in the following expression, as shown below:
text repeat(MM X Nu),
Xing = M XM
where Xieu E I and Xing E IBTxxNaxdme,
Finally, the spatio-temporee sequence Xst is normalized to Fst to ensure consistent scaling across time steps. To extract semantic representations from unstructured text, we employ pre-trained BERT ( to encode textual inputs into contextualized embeddings aligned with the ppati-ttemoora context, represented Fixt BERT (Xiet). Simultaneously, visual features are extracted from images using a convolutional neural network (CNN), formulated as Fing = CNN((img), which highlights geographic cues relevant to environmental conditions.
Step 2: Multi-modal feature fusion. To facilitate interactions among features from different modalities within shared latent space, we project the spatio-temporal features Fst, text features Fiex, and image features Fing into unified hidden dimension using three separate fully connected layers. This yields mopality-specific representations Fst, Fiext. and Fing E 1Txxxxx,d,
Next, we employ two Cross-Modal Attention (CMA) modules capture interactions between spatio-temporal text and image features. Each CMA module computes queries (Q), keys (K), and values (V) from the aligned features and applies scaled dot- product attention mechanism model cross-modal similarity. The attention operation is defined as:
Atxx CMA(F Frext Frea). Attrag-ting CMA(Fss Fimg. F Fing).
where CMA(Q, K, V) = softmax QK Vir
V and dy represents the dimension of each attention head.
This attention mechanism facilitates information exchange across modalities, thereby enriching the feature representations. To integrate the outputs, concatenate the modality features and apply a fusion gating mechanism to get the gating values, as shown below:
Fiused [Fst Attn Att FusionGate([Fa.. Attna-next Att
(4)
The gating values regulate the contribution of each modality, producing the final fused representation.
4.2 Dual-Branch Causal Inference
Given that multi-modal data introduces complex causal interactions that can impact prediction accuracy, we propose an innovative dual-branch causal invariance approach to differentiate true causal relations from confounding factors, which is shown Fig. (bb).
Step 1: Causal matrix construction. To uncover latent dependencies among spatial units and en- hance the interpretabiliity of the model, we estimate the underlying causal structure using DeepSHAP- based feature importance. The SHAP value, Φi.j- quantifies the influence of node i on node j. resulting in a matrix A SHAP E NNNN If prior graph A (0) is available, we construct a hybrid adjacency matrix, shown below:
A = XA (0) + + (1- A SHAP
(5)
where A E [0, 1] balances reliance on prior knowledge versus data-driven insights.
To enhance model stability and robustness, we adopt exponential moving average and update the adjacency matrix every P 5 epochs. This interval, determined empirically, balances stability and computational efficiency. Updating too frequently introduces noisy structural fluctuations, while moderate update interval allows the model to refine its internal representations and capture meaningful spatio-temporal dependencies more effectively.
Step Dual-branch causal adjustment. Spatio-temporal prediction is often biased by unobserved confounders (S) well as observed external factors, including image features (E, derived from
5

---

Attrst-iimg and text features (C,, derived from Attma-sexx). To mitigate these biases, we introduce dual-branch causal adjustment mechanism that explicitly accounts for confounding influences.
The main branch relies solely on the spatio-temporal sequence Xa- However, this branch may overlook critical information from external factors. The auxiliary branch integrates multi-modal data Finsed to predict the future sequence. We use a multi-layer perceptron (MLP) layer to combine the output features from the two branches. The final prediction and corresponding loss functions are:
Y final = MLP(ff(sss A); (Ffused A))
(6)
- A)ll mm = || ff A)22, Speed | - Yinal lla.
(7)
where f() denotes the prediction module STED (to be detailed in Section 4.3), Lat is the loss of the main branch, Lmm is the loss of the auxiliary branch, and Lpred is the loss of the final prediction.
Overall, we aim to obtain a model by training on the two branches described above, which handled by the following loss function.
Lall = Epeed + 3Lu + (1 1 3)
(8)
where 3 is an adjusting parameter to balance the influence of the two branches.
Note that although the multi-modal fusion Fiused enriches the representation, concatenating the visual feature E and the textual feature C' can amplify the influence of an unobserved confounder S, which simultaneously affects both the spatio-temporal signal Xa and the target Ya- We formalize the data generation process using the following Structural Causal Model (SCM), as shown below:
Xa (S, F, C) Y = fr S,E,C)
where S' denotes the confounding feature, approximated by a set of latent variables {s:}.
Under the backdoor criterion 37, estimate interventional outcomes, shown below:
x, = = CydS P(Ysilo(Xa, = F. =
(10)
The intervention on each r is adjusted as follows:
i = x + x @ W[ath(S) + axp(EE) + acg(C)),
(11)
where & is the intervention result on x, W is set of weights that control the magnitude of influence of the respective factors, and ((-). p()), and q(-) are functions representing S, E, and C, respectively,
To ensure that the adjustment effectively removes the confounding influence of S, aim to make statistically independent of S, conditioned on and C. This can be achieved by minimizing the gradient of & with respect to S. The training objective is to minimize the influence of the latent variable S on & thereby driving 02 25 0. Consequently, the backdoor path Xat + S + Ya is blocked, ensuring the validity of the causal inference process. (See Appendix B for detailed proof).
4.3 Spatio-Tempon Encoding and Decoding (STED)
To capture both spatial and temporal dependencies, we design a spatio-temporal encoding module that combines Graph Convolutional Networks (GCN) and the Mamba model. As shown in Fig. (c), the GCN captures spatial neighborhood relations, while Mamba learns temporal evolution features.
The spatial encoding is performed via multi-layer Graph Convolutional operations on node features X E BR where B is the batch size, 7 is the historical time step, N is the number of nodes and d' the feature dimension. The message-passing process is performed using the edge indices constructed from the adjacency matrix A E Nx which encodes the spatial relationships between the nodes. The spatial encoding is computed as:
X sparial GCN(X, A)
(12)
For temporal modeling, while several efficient Transformer variants have been proposed to reduce complexity, they often impose architectural constraints or lose modeling fidelity [15]. In contrast, we employ Mamba selective state-space model that maintains linear complexity while effectively

---

capturing long-rang range dependencies, akin to attention mechanisms. Specifically, each Mamba block consists of an imput projection, depth-wise convolution, gated selective state-space kemell, element- wise modulation, and an output projection (Fig. 20),, right).
uemporal = Mamba(X W Win
(13)
where Win and Won out are the linear projection layers.
Spatial and temporal features are then fused clement-wist operations. To enhance training stability, residual connections are incorporated during the fusion process. Each fused output is subsequently passed through a LayerNorm operation to normalize the features, alleviating internal covariate shift and facilitating the training of deeper models.
Xenodeed = LayerNorm (X + (emporal
(14)
The module consists of three stacked layers of GCN and Mamba substructures, alternating between capturing spatial and temporal dependencies. The final output Xenoddad is a set of spatio-temporal encoded features that match the shape of the input.
Finally, the spatio-temporal encoded features are decoded through an MLP layer to generate the output predictions. The decoding process expressed as follows:
Ý MLPPPX (encuded)
(15)
4.4 Model Training
Training Algorithm. The overall training algorithm of E2-CSTP is described Appendix Compared to Transformer-base methods with quadratic complexity, our hybrid GCN-Mamba architecture significantly reduces computational overhead while maintaining predictive performance.
C
Complexity Analysis. In standard Transformer architectures, the input tensor X E BRBxxxxxX typically flattened along the temporal and spatial dimensions, resulting in a sequence of length T- The self-attention mechanism incurs a time complexity of (2.....N2.... This quadratic dependency on both the number of nodes and time steps significantly increases computational cost, particularly when dealing with long sequences large spatial graphs.
In contrast, the proposed STED module decomposes spatio-temporal modeling into spatial GCN and temporal Mamba components, each operating along single axis. The time complexity of the GCN is - N2.... d). The Mamba block operates per node and time step, with linear-timi complexity in T. Each operation (projection, convolution, kernel application, modulation) is applied to sequence of length for each of the N nodes and B samples and the complexity O(B T d). Therefore, N2 the overall complexity is O(B . . d), enabling faster and more memory-efficient training on large-scale spatio-temporal data.
In addition, the proposed STED module, whose complexity scales with sequence length T' and graph size N, drives the speed-up, while the shared text and image encoders merely handle preprocessing and therefore do not affect the relative ranking.
Experiments
We below four Research Questions (RQs) guide the experiments.
RQI. How does EC-CSTP perform compared existing single-modal and multi-modal spatio- temporal prediction methods?
RQ2. How do the various modules in E2-CSTP contribute its performance?
RQ3. How efficient E2-CSTP in training compared both baseline models and Trunsformer-bused prediction module alternatives?
RQ4. How sensitive is CCSS to changes hyperparameter settings?
5.1 Experimental Settings
Datasets. We collect 4 datasets to evaluate the proposed E²-CSTP framework: Terra 4 BjTT 59
GreenFarthNet (3), and BikeNYC (60]. (i) Terra provides spatio-temporal observations along with
multi-modal information such as geo-images and explanatory texts; (ii) BjTT is multi-modal
7

---

Table I: Overall performance. Best results are bold and the second best are underlined.
Method
Terra
BjTT
GreenEarthNet
BikeNYC
MAE
RMSE
MAPE
MAE
RMSE
MAPE
MAE
RMSE
MAPE
MAE
RMSE
MAPE
D2STGNN
2.52
3.13
29.79%
4.57
7.75
0.28
10.55
69.06%
ST-SLL
2.55
3.18
30.32%
4.83
8.00
15.78%
0.24
0.31
84.45%
7.68
14.02
78.93%
HimNet
2.54
26.81%
6.10
11,02%
0.21
76.33%
7.93
NuwaDynamics
2.49
3.07
24.85%
3.72
5.98
10.79%
0.18
0.26
73.26%
3.56
6.27
61.68%
CaPaint
2.49
25.47%
6.03
0.25
68.37%
62.95%
GPTSTT
2.47
3,06
30,06%
3.69
5.66
10.02%
0.17
0.23
98.83%
3.37
5.85
63.97%
UnIST
2.47
5.49
0.20
72.79%
5.58
T3
2.53
3.11
26. 13%
3.73
6.02
11.01%
FNF
2.51
27.42%
5.51
E7-CSTP
2,43
3.01
23.62%
3,56
5.32
9.24%
0.13
0.18
57.09%
2.99
5.53
56.13%
dataset containing traffic data and event descriptions for traffic prediction; (iii) GreenEarthNet is a multi-modal satellite dataset used to estimate vegetation; (iv) BikeNYC contains only spatio-temporal sequences, evaluating model performance based on the bike flow attribute in a single-modality setting. The datasets are chronologically divided into training, validation, and test sets in 8:1:1 ratio. We use data from the past 12 time steps to predict the subsequent 12 time steps. More detailed dataset information is provided in Appendix(D.] D.1
Baselines. We compare E²-CSTP with 9 state-of-the-art spatio-temporal prediction methods, categorized into four groups: (i) Single-modal spatio-temporal prediction methods, including D2STGNN 40), ST-SSL [19], and HimNet [[0]: (ii) Causality-based spatio-temporal prediction methods, including NuwaDynamics 45 and CaPaint [13]; (iii) Foundation models for spatio- temporal prediction, including GPT-ST [27] and UniST [57]; (iv) Multi-modal spatio-temporal prediction methods, including T3
16 and From News to Forecast (FNF) 48 For more detailed
information of the baselines and implementation, please refer to Appendix D.2 and Appendix D.3
Evaluation Metrics. We evaluate model performance using Mean Absolute Error (MAE), Root Mcan Square Error (RMSE), and Mean Absolute Percentage Error (MAPE) for accuracy, and total runtime and per-epoch runtime for efficiency. Detailed calculation methods are provided in Appendix D.4
5.2 Overall Performance Comparison (RQI)
Table presents the overall performance comparison of all methods across the four datasets. We yield the following observations. Notably, T3 and FNF are excluded from comparisons on GreenEarthNet and BikeNYC due to the absence of textual modalities in these datasets.
First, E2-CSTP consistently outperforms all baselines across metrics and datasets, with MAE improve- ments ranging from 1.61 % to 9.66% over the second-best methods. This highlights the effectiveness of our E² CPP framework in capturing fine-grad -raigra dependencies often missed by other methods.
Second, multi-modal and causality-basee models generally outperform single-modal baselines, con- firming the necessity of auxiliary modalities and causal reasoning in spatio-temporal forecasting. models, despite their theoretical appeal, often lack the contextual under- standing required for comprehensive predictions. In contrast, E2-CSTP integrates causal reasoning within a modtil framework, further reducing uncertainty through cross- modal interactions.
However, causality-bar
Third, E²-CSTP consistently outperforms other multi-modas methods on environment- and event- driven datasets, with up to 3.95% performance gain. This advantage stems from its ability to suppress confounding factors and leverage rich visual and textual cues, thereby enabling robust predictions in complex real-world scenarios. In contrast, foundation models such as GPT-ST and UniST, despite their strong transferability from large-scale pre-training, lack explicit multi-modal fusion and causal modeling, which limits their fine-grained predictive capabilities.
5.3
Ablation Study (RQ2)
Next, we conduct ablation studies to assess the contribution of six components in our E²-CSTP framework. (1) w/o Text Feature: It removes text inputs to assess the impact of language- auxiliary information. (2) w/o Image Feature: It excludee visual inputs to evaluate the contribution of image context. (3) w/o DeepSHAP: It uses only the initial adjacency matrix, omitting the DeepSHAP- (4) w/o Causal Inference (CT): It disables the causal intervention to examine the importance of confounder mitigation. (5) w/o GCN: It removes the spatial encoding,
based causal region identification.
8

---

disabling the graph- based spatial dependency modeling. (6) w/o Mamba: It eliminates temporal encoding based on the Mamba architecture, testing its contribution temporal dynamics modeling.
graph-based
E4-CSTP
Imuge Feature
DeepSHAP
GCN
2,60
2.55
0.20
4.5
4..0
MAF
2.45
3.5
0.12
3.0
0.08
2.9
Terra
BjTT
GreennFarthNet
BikeNYC
Figure 3: The ablation study,
The results are shown Fig. The w/o Text Feature and woo Image Feature variants suffer performance drops, indicating that both textual and visual inputs provide essential contextual cues for accurate prediction. The w/o DeepSHAP model fails to effectively focus on influential regions, weakening spatial reasoning. Removing the Causal Inference module leads to reduced robustness, especially under confounding conditions. The w/o GCN variant struggles with spatial structure modeling, while the w/o Mamba variant cannot capture temporal dynamics effectively. These results confirm that every module is critical for modeling complex multi-model spatio-temporal patterns.
5.4
Model Efficiency Study (RQ3)
We further evaluate the efficiency of E2-CSTP by comparing it with 9 spatio-temporal baseline models. Fig. shows the total training time required to reach convergence under the same batch size.
DISTGNN
ST-S8L.
HimNet
NawaDyramis
CaPaint
GPT-ST
UnST
FNF
F--SSTP
Trining Time (min)
Triniin Tinze (min)
1000
Training Timee (mai))
Trising Time; (min)
to"
400
200
Terra
BjTT
GreenEarthNet
BikeNYC
Figure 4: Model efficiency comparison on the total training time.
The results indicate that E²-CSTP achieves training time comparable those of single-modal methods, even when applied to multi-modal datasets. Moreover, it consistently outperforms UniST and other multi-modal models, showing a particularly notable advantage over LLM-based approaches such as FNF. On the single-modal BikeNYC dataset, E²-CSTP achieves faster training compared to all baseline methods, further showing its superiority.
To assess the effectiveness of our prediction module STED on computational cost, we replace with several popular Transformer- based alternatives, including Informer [68] Autoformer (50), FED- former [69], and iTransformer (31), thus isolating the gain brought by replacing a quadratic Transformer with our linear GCN-Mamba block. These variants are denoted w/ In, w/ Auto, w/ FED, and w/ iTTras, respectively. Fig. shows the prediction accuracy and per-epoch runtime on the Terra dataset. As observed, our method improves prediction accuracy by 1.78%-5.44 5,45% while reducing computational overhead by 17.37% % -6..11% compared to these alternatives, demonstrating that the proposed module both effective and efficient in practice.
Hime lopoch
MAE
2ie
Apto W1D
Figure Efficiency under pre- diction variants Terra.
5.45%
Due to space limitations, detailed per-epoch runtime comparisons with 9 baseline models and additional results from alternative Transformer-based prediction modules on other datasets are provided in Appendix D.5 showing similar observations.
5.5 Parameter Sensitivity Study (RQ4)
To evaluate the impact hyperparameters, we vary the graph fusion factor À across {0. 0.25..2000 0.5, 0.75, 1} and the dual-branch loss balancing factor 3 across (0, 0.25, 0.5, 0.75, 1}, se- lecting the optimal combination each dataset based on validation performance. Specifically, for the Terra and GreenEarthNet datasets, À is set to 0.25, with 3 set to 0.75 and 0.5, respectively. For

---

the BjTT and BikeB BC datasets, À is set to 0.5, while 3 is set to 0.5 and 0.75, respectively. Detailed parameter analysis can be found in Appendix D.6
Conclusion
In this paper, we propose E²-CSTP for spatio-temporal prediction that addresses the challenges of multi-modal fusion. confounding bias, and computational inefficiency. By integrating cross-modal attention and gating mechanisms, E² achieves robust fusion of spatio-temporal, textual, and visual inputs. To mitigate biases introduced by auxiliary inputs, we introduce dual-branch causal inference based on causal interventions. Experiments conducted on 4 real datasets demonstrate that E2-CSTP outperforms 9 SOTA baselines in accuracy and efficiency. A more detailed discussion of the limitations and potential future directions is provided in Appendix
Acknowledgment
This work supported in part by the NSFC under Grants No. (62402422, 62025206, U23A20296, and 62472377), Yongjiang Talent Introduction Programme (2024A-162-68 162-G), Zhejiang Provincial Natural Science Foundation of China under Grant No. LZZ25002001,, and Zhejiang Province's "Lingyan" R&D Project under Grant No. 2024C01259. Ziquan Fang is the corresponding author.
References
[1] Clireerlivier Amblard, Olivier J. J. Michel, Cédric Richard, and Paul Honeine. 2012. A Gaussian process regression approach for testing Granger causality between time series data. In ICASSP. 3357-3360.
[2] Lei Bai, Lina Yao, Can Li, Xianzhi Wang, and Can Wang. 2020. Adaptive Graph Convolutional Recurrent Network for Traffic Forecasting, In NeurIPS.
[3] Vitus Benson, Claire Robin, Christian Requena- Requena-Mesa, Lázaro Alonso, Nuno Carvalhais. José Cortés, Zhihan Gao, Nora Linscheid, Mélanie Weynants, and Markus Reichstein. 2024. Mudti-Modal Learning for Geospatial Vegetation Forecasting. In CVPR. 27788-27799.
[4] Wei Chen, Xixuan Hao, Yuankai Wu, and Yuxuan Liang. 2024. Terra: A Multimodal Spatio- Temporal Dataset Spanning the Earth. In NewrlPS.
[5] Wei Chen, Yuxuan I. Liang, Yuanshao Zhu, Yanchuan Chang, Kang Luo, Haomin Wen, Lei Li, Yanwei Yu, Qingsong Wen, Chao Chen, Kai Zheng, Yunjun Gao, Xiaofang Zhou, and Yu Zheng. 2024. Deep Learning for Trajectory Data Management and Mining: A Survey and Beyond. CoRR abs/2403. 14151 (2024).
[6] Junyoung Chung, Çaglar Gülçehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling. CoRR abs/14 1123555 (2014).
[7] Jerome T. Connor, R. Douglas Martin, and Les E. Atlas. 1994, Recurrent neural networks and robust time series prediction. IEEE Trans. Neural Networks 5, 2 (1994), 240 450.
[8] Jiewen Deng, Renhe Jiang, Jiaqi Zhang, and Xuan Song. 2024. Multi-Modality Spatio- Temporal Forecast- ing via Self-Supervised Learning. In IJCAI,
[9] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre- traitngining of Deep Bidirectional Transformers for Language Understanding. In NAACL-HLT. 4171-4188
[10] Zheng Dong, Renhe Jiang. Haotian Gao. Hangchen Liu, Jinliang Deng, Qingsong Wen, and Xuan Song. 2024. Heterogeneity-Lnfonn Informed Peta-Parameter Learning for Spatiotemporal Time Series Forecasting, In KDD. 631-641.
[11] Harris Drucker, Christopher J. C. Burges, Linda Kaufman, Alexander J. Smola, and Vladimir Vapnik. 1996. Support Vector Regression Machines. In NeurIPS. 155-161.
[12] Shengdong Du, Tianrui Li, Xun Gong, and Shi-Jinn Horng 2020. A Hybrid Method for Traffic Flow Forecasting Using Multimodal Deep Learning. Int. J. Comput. Intell. Syst. 13, 1 (2020), 85-97.
[13] Yifan Duan, Jian Zhao, pengcheng, Junyuan Mao, Hao Wu, Jingyu Xu, Shilong Wang, Caoyuan Ma. Kai Wang, Kun Wang. and Xuelong Li. 2024. Causal Deciphering and Inpainting in Temporal Dynamics via Diffusion Model. In NewrIPS.
10

---

[14] Albert Gu and Tri Dao. 2023. Mamba: Linear-Time Sequence Modeling with Selective State Spaces. CoRR abs/2312.000552 (2023).
[15] Dongchen Han, Ziyi Wang, Zhuofan Xia, Yizeng Han, Yifan Pu, Chunjiang Ge, Jun Song, Shiji Song. Bo Zheng, and Gao Huang. 2024. Demystify Mamba in Vision: A Linear Attention Perspective. In NeurIPS.
[16] Xiao Han, Zhenduo Zhang, Yiling Wu, Xinfeng Zhang, and Zhe Wu. 2024. Event Traffic Forecasting with Sparse Multimodal Data. In MM. 1866.884.
[17] Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9, 8 (1997), 1735-1780.
[18] Zhen Huang, Shen, Xinmei Tian, Houqiang Li. Jianqiang Huang, and Xian-Sheng Hua. 2020. Spatio- Temporal Inception Graph Convolutional Networks for Skeleton- Basedd Action Recognition. In MM. 2122-2130.
[19] Jiahao Ji, Jingyuan Wang. Chao Huang, Junjie Wu, Boren Xu, Zhenhe Wu, Junho Zhang, and Zheng. 2023. Pemtorll Self-Supervised Learning for Traffic Flow Prediction. In AAAI.
[20] Furong Jia, Kevin Wang, Yixiang Zheng, Defu Cao, and Yan Liu. 2024. GPT4MTS: Language Model for Multimodal Time-serics Forecasting. In AAAL
Prompt- based Large
[21] Jiawei Jiang, Chengkai Han, Wayne Xin Zhao, and Jingyuan Wang. 2023. PDFormer: Propagation Dynamic Dwlay-Aware
Transformer for Traffic Flow Prediction. In AAAI, 4365-4373.
[22] Yushan Jiang, Kanghui Ning, Zijie Pan, Xuyang Shen, Jingchao Ni, Wenchao Yu, Anderson Schneider, Haifeng Chen, Yuriy Nevmyvaka, and Dongjin Song. 2025. Multi-modal Time Series Analysis: A Tutorial and Survey. CoRR abs/2503. 11.5503..37999 13709 (2025).
[23] Guangyin Jin, Yuxuan Liang, Yuchen Fang, Zezhi Shao, Jincai Huang. Junbo Zhang. and Yu Zheng. 2024. PeatioTeeppraa Graph Neural Networks for Predictive Learning in Urban Computing: Survey, IEEE Trans. Knowl. Data Eng. 36, 10 (2024), 5388-5408.
[24] Ilyes Khemakhem, Diederik P. Kingma, Ricardo Pio Monti, and Aapo Hyvärinen. 2020. Variational Autoencoders and Nonlinear ICA: Unifying Framework. In AISTATS, Vol. 108. 2207-2217
[25] Yaxuan Kong, Yiyuan Yang, Shiyu Wang, Chenghao Liu, Yuxuan Liang, Ming Jin, Stefan Zohren, Dan Pei, Yan Liu, and Qingsong Wen. 2025. Position: Empowering Time Scries Reasoning with Multimodal LLMs. CoRR abs/2502.01477 (2025).
[26] Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. 2018 Diffusion Convolutional Recurrent Neural Network: Data-Driven Traffic Forecasting. ICLR.
[27] Zhonghang Li, Lianghao Xia andi Yong Xu, and Chao Huang. 2023, GPT-ST: Generative Pre-Training of Spatio-Temporal Graph Neural Networks. In NeurlPS.
[28] Zetao Li, Zheng Hu, Peng Han, Yu Gu, and Shimin Cai. 2025. SSL-STMFormer Self Seff-Supervised Learning Spatio-Temporal Entanglement Transformer for Traffic Flow Prediction. In AAAI. 12130-12153 12138.
[29] Fudong Lin, Summer Crawford, Kaleb Guillot, Yihe Zhang. Yan Chen, Xu Yuan, Li Chen, Shelby Williams, Robert Miavielle. Xiangming Xiao, Drew Gholson, Nicolas Ashwell, Tri Setiyono, Brenda Tubana, Lu Peng, Magdy A. Bayoumi, and Nian-Feng Tzeng. 2023. MMST-ViT: Climate Change-aware Crop Yield Prediction via Multi Modal Spatial-Temppon Vision Transformer. ICCV. 5751-5761.
[30] Haoxin Liu, Shangqing Xu, Zhiyuan Zhao, Lingkai Kong, Harshavardhan Kamarthi, Aditya B. Sasanur, Megha Sharma, Jiaming Cui, Qingsong Wen, Chao Zhang, and B. Aditya Prakash. 2024. Time-MMDD: Multi-Domain Multimodal Dataset for Time Series Analysis. NewrIPS.
[31] Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma, and Mingsheng Long, 2024. iTransformer: Inverted Transformers Are Effective for Time Series Forecasting. In ICLR.
[32] Helmut Lütkepohl. 2005. New introduction to multiple time series analysis. Springer Science & Business Media.
[33] Qinzhi Lv, Lijuan Liu, Ruotong Yang, and Yan Wang. 2025. Multimodal urhan traffic flow prediction based multi-scale time series imaging. Pattern Recognition 164 (2025), 111499.
[34] Juan Nathaniel, Yongquan Qu, Tung Nguyen, Sungduk Yu, Julius Busecke, Aditya Grover, and Pierre Gentine. 2024. ChaosBench: A Multi- Channel,,,,,, Physics- Based Benchmark for Subseasonal to toaSeasanal Climate Prediction. In NewrIPS.
11

---

[35] Yuqi Nie. Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. 2023. A Time Series is Worth 64 Words: Long-term Forecasting with Transformers. In ICLR.
[36]
Cressie Noel and Wikle Christopher K. 2011, Statistics for spatio-tempora dota. John Wiley & Sons.
[37] Judea Pearl. 2012. The Do-Calculus Revisited. In UAI. 3-11.
[38] Lukás Picek, Christophe Botella, Maximilien Servajean, César Leblane, Rémi Palard, Théo Larcher, Benjamin Deneu, Diego Marcos, Pierre Bonnet, and Alexis Joly, 2024. GeoPlant: Spatial Plant Species Prediction Datasct. In NewrIPS.
[39] Zezhi Shao, Fei Wang. Yongjun Xu, Wei Wei, Chengging Yu. Zhao Zhang, Di Yao, Tao Sun, Guangyin Jin, Xin Cao, Gao Cong, Christian S. Jensen, and Xueqi Cheng. 2025. Exploring Progress in Multivariate Time Series Forecasting: Comprehensive Benchmarking and Heterogeneity Analysis. IEEE Trans. Knowl. Data Eng. 37, 1 (2025), 291-305.
[40] Zezhi Shao, Zhao Zhang, Wei Wei, Fei Wang, Yongjun Xu, Xin Cao, and Christian S. Jensen. 2022. Decoupled Dynamic Temporal Graph Neural Network for Traffic Forecasting, Proc. VLDB Endow. 15, 11 (2022). 2733-2746.
[41] Xingjian Shi, Zhourong Chen, Hao Wang, Dit- Yeung, Wai-Kin Wong, and Wang-chun Woo, 2015. Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting. In NewrIPS. 802-810.
[42] Florian Toqué, Mostepha Khouadjja, Etienne Côme, Martin Trépanicr, and Latifa Oukhellou. 2017. Short & long term forecasting of multimodal transport passenger flows with machine learning methods, In ITSC. 560-566.
[43] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In NeurIPS. 5998-6008
[44] Chenxing Wang. 2024. Towards Effective Fusion and Forecasting of Multimodal Spatio-temporal Data for Smart Mobility. In CIKM. 5483-5486.
[45] Kun Wang, Hao Wu, Yifan Duan, Guibin Zhang, Kai Wang, Xiaojiang Peng, Yu Zheng, Yuxuan Liang, and Yang Wang. 2024, NuwaDynamics: Discovering and Updating in Causal Temporl Modeling. In ICLR.
[46] Pengkun Wang. Chuancai Ge, Zhengyang Zhou, Xu Wang. Yuantao Li, and Yang Wang. 2023. Joint Gated Co-Attention Based Multi-Modal Networks for Subregion House Price Prediction. IEEE Truns. Knowl. Data Eng. 35, 2 (2023), 1667-1680.
[47] Senzhang Wang, Jiannong Cao, and Philip Yu. 2022. Deep Learning for Spatio-Temporal Data Mining: A Survey. IEEE Trans. Knowl. Data Eng. 34, 8 (2022), 3681-3700.
[48] Xinlei Wang, Maike Feng, Jing Qiu, Jinjin Gu, and Junhua Zhao. 2024, From News to Forecast: Integrating Event Analysis in
Time Series Forecasting with Reflection. In NeurlPS.
[49] Yiheng Wang, Tianyu Wang, Yu Ying Zhang, Hongji Zhang, Haoyu Zheng, Guanjie Zheng, and Linghe Kong. 2024. Layynr: Unified Data Pipeline for Urban Science. In NeurlPS.
[50] Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long. 2021. Autoformer: Decomposition Transform- ers with auto-Correlation for Long-Term Series Forecasting. NeurlPS. 22419-22430.
[51] Shaofei Wu. 2022. Spatiotemporal Dynamic Forecasting and Analysis of Regional Traffic Flow in Urban Road Networks Using Deep Learning Convolutional Neural Network. IEEE Trans. Intell. Transp. Syst. 23, (2022), 1607-1615.
[52] Yutong Xia, Yuxuan Liang, Haomin Wen, Xu Liu, Kun Wang. Zhengyang Zhou, and Roger Zimmermann. 2023. Deciphering Spatio-Temporal Graph Forecasting: A Causal Lens and Treatment. In NeurlPS.
[53] Yimo Yan, Songyi Cui, Jiahui Liu, Yaping Zhao, Bodong Zhou, and Yong-Hong Kuo, 2025. Multimodal fusion for large-scale traffic prediction with heterogeneous retentive networks. Inf. Fusion 114 (2025). 102695.
[54] Weiran Yao, Yuewen Sun, Alex Ho, Changyin Sun, and Kun Zhang. 2022. Learning Temporally Causal Latent Processes from General Temporal Data. In ICLR.
[55] Zhongchao Yi, Zhengyang Zhou, Qihe Huang, Yanjiang Chen, Liheng Yu, Xu Wang, and Yang Wang. 2024. Get Rid of Isolation: A Continuous Multi-task Temporll Learning Framework. In NeurIPS.
12

---

[56]
Bing Yu, Haoteng Yin, and Zhanxing Zhu. 2018. Spatio-Temporal Graph Convolutional Networks: A
Deep Learning Framework for Traffic Forecasting. In IJCAI. 3634-3640.
[57] Yuan Yuan, Jingtao Ding, Jie Feng, Depeng Jin, and Yong Li. 2024. UniST: A Prompt-Empowered Universal Model for Urban Spatio-Tempoon Prediction. In KDD. 4106.
[58] Chaohe Zhang, Xu Chu, Liantao Ma, Yinghao Zhu, Yasha Wang, Jiangtao Wang, and Junfeng Zhao. 2022. M3Care: Learning with Missing Modalities in Multimodal Healthcare Data. In KDD. 2418-2428.
[59] Chengyang Zhang. Yong Zhang, Qitan Shao, Jiangtao Feng, Bo Yisheng Lv, Xinglin Piao, and Baocai Yin. 2024. BjTT: A Large-Scale Multimodal Dataset for Traffic Prediction. IEEE Trans. Intell. Transp. Syst. 25, 11 (2024), 18992-19003.
[60] Junbo Zhang, Yu Zheng, and Dekang Qi. 2017. Deep Spatio-Tempp Residual Networks for Citywide Crowd Flows Prediction. AAAI. 1655-1661.
[61] Weijia Zhang, Le Zhang, Jindong Han, Hao Liu, Yanjie Fu, Jingbo Zhou, Yu Mei, and Hui Xiong. 2024. Irregular Traffic Time Series Forecasting Based on Asynchronous Spatio-Tempont Graph Convolutional Networks. In KDD. 4302-4313.
[62] Xinbang Zhang, Qizhao Jin, Tingzhao Yu, Shiming Xiang, Qiuming Kuang, Veronique Prinet, and Chunhong Pan. 2022. Multi-modal spatio-temporal meteorological forecasting with deep neural network. ISPRS Journal of Photogrammetry and Remote Sensing 188 (2022), 380-393.
[63] Kesen Zhao and Liang Zhang. 2024. Causality-Inspired Spatial-Temporal Explanations for Dynamic Graph Neural Networks, In ICLR.
[64] Yu Zhao, Pan Deng, Junting Liu, Xiaofeng Jia, and Mulan Wang. 2023. Causal Conditional Hidden Markov Model for Multimodal Traffic Prediction. In AAAI
[65] Yu Zhao, Pan Deng, Junting Liu, Xiaofeng Jia, and Jianwei Zhang. 2023. Generative Causal Interpretation Model for Temporal Representation Learning. In KDD. 3537-3548.
[66] Chuanpan Zheng, Xiaoliang Fan, Shirui Pan, Haibing Jin, Zhaopeng Peng, Zonghan Wu, Cheng Wang, and Philip S. Yu. 2024. Temporall Joint Graph Convolutional Networks for Traffic Forecasting. IEEE Trans. Knowl. Data Eng. 36, (2024), 372-385.
[67] Bodong Zhou, Jiahui Liu, Songyi Cui, and Yaping Zhao. 2024. A Large-Scale Spatio-Temporal Multimodal Fusion Framework for Traffic Prediction. Big Data Min. Anal. 7, (2024), 621-636.
[68] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. 2021. Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting, In AAAI.
11106-11115.
[69] Tian Zhou, Ziqing Ma. Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin. 2022. FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting. In ICML, Vol. 162. 27268-27286.
[70] Zhengyang Zhou, Qihe Huang, Kuo Yang. Kun Wang, Xu Wang, Yudong Zhang, Yuxuan Liang, and Yang Wang. 2023, Maintaining the Status Quo: Capturing Invariant Relations for OOD Spatiotemporal Learning. In KDD. 3603-3614.
13

---

Appendix
A
Notations and Descriptions
Table 2
presents the frequently used notations and their descriptions.
Table 2: Notations and descriptions
Notation
Description
V.N
Set of nodes and number of nodes in the spatial graph Set of edges and adjacency matrix of the spatial graph Spatial graph g = (V.E, & A)
A
G
X
Multi-modal spatio-temporal data (Xst- Xust Xime
Y,Y
Future and predicted spatio-temporal sequence Timestamps of spatio-temporal, text, and image data Spatial coordinates for image and spatio-temporal data Alignment matrices for text and image modalities
P
M
F
Features from spatio-temporal, text, and image modalities Latent confounder, image, and text variables
S, E,E,C C
Causal inference
B.1
The definition of causal inference
Causal inference seeks quantify how interventions influence outcomes of interest. A Structural Causal Model (SCM) specifies the underlying data-generating mechanism. The do-operator supplies the formal notation for interventions. Identifiability criteria, most notably the backdoor and front- door rules, convert causal queries into estimable statistical functionals. Pearl's [37] three rules of do-calculus unify these components, allowing to strip away interventions iteratively whenever the requisite d-separation conditions are met.
B.2
The three rules of do-calculus
Pearl's do-calculus provides symbolic rules that transform interventional distributions observational ones by exploiting graphical separation statements. Let Gdo(a) be the graph obtained from SCM after performing do(X = x). Gnll(()) denotes the graph where incoming edges to Z are cut but Z is not fixed to constant.
Rule I (Insertion/delettion of observations).
Ply | do(x), = ply do(x)) if y 11 x in Gdo(s)
(16)
Rule 2 (Action/observation exchange).
P(y (o(=)))o((()) = Ply do(x), if Il 2 x in Gdo(s) mull(s)
(17)
Rule 3 (Insertion/dellttion of actions).
Pyy do(x), do(=)) = P(y do(x)) if II. 2 in Gdo(s))
(18)
These rules are complete for transforming interventional distributions into observational ones and underpin backdoor adjustment.
B.3 Deriving the backdoor adjustment
Consider a treatment- backdoor path from X in G. and (ii) Z contains no descendant of X. We recover the celebrated backdoor formula by successive applications of the do-calculus rules.
pair (X, () and a set of variables Z such that (i) Z blocks every
14

---

Step from Bayes' rule).
Ply do(z))
Σ P(y do(a), P(z do(z))
(19)
Step (Apply Rule 3 to remove the action 7))
P(y | do(x), 2) P(y x2
(20)
Step (Substitute the two observations into Step 1).
Ply do(z))
Σ P(y x P(z),
(21)
which is the backdoor adjustment formula. When Z is continuous, the summation in Eq. 21
replaced by an integral.
B.4 Validity of the adjustment mechanism
Eq. 21 states that, whenever valid back-door set Z exists, causal effects be estimated by (i) stratifying on Z, (ii) computing the conditional association between X and Y within each stratum, and (iii) averaging the results over the marginal distribution of Z. This principle underlies the dual-branch causal adjustment in Section 4.2 of the main paper, where the latent confounder S plays the role of Z.
Proof. We formally justify that the adjusted representation è in Section 4.2 satisfies two essential conditions:
4 ~ P(X,, (X)), C,C) and 11 S F,C,
(22)
ensuring structural disentanglement between environmental encoding E and event encoding C without mutual interference.
(I) Cross-modality independence.
We assert that the learned representations of environmental E and event C' variables are distribution- ally independent:
PEE) IL g(C),
(23)
and their respective gradients with respect to each other vanish:
Op 0,
0q
0,
(24)
ac
DE
indicating disentangled representations without cross-modal entanglement.
(2) Conditional independence of â and S.
To eliminate the influence of confounding variable S on the adjusted representation & we define an optimization objective to minimize the conditional variance of given E and C, with respect to S:
[ F E, C
min 01,,02,0g
(25)
The adjusted representation formulated as:
= + 3 (ath(S) a2p(E) azq(C)),
(26)
where @ denotes element wise multiplication, and a1, 02 are gating coefficients modulating the influence confounder, environment, and event, respectively.
During adversarial training, the confounder-invariint component is encouraged via gradient cancella- tion:
ah
Oz (1 + Wayh(S))
② W a1 aa
(27)
as
This leads representation of the form:
à Rz fx(S,, E, E, x W [azp(E) (sg(C)),
(28)
15

---

where So represents fixed baseline value of the confounder. Consequently, å becomes insensitive to S variations and approximates sampling from an interventional distribution conditioned on and C under fixed S, approximating P(Xa do( Xx). F,, C)
In conclusion, & fulfills conditions necessary for reliable causal inference in multi-modal fusion models. It maintains independence from confounding factors and preserves the disentanglement between environmental and event encodings. Thus, provides valid approximation of P(Xst do(Xa), E, C),, crucial for interpretable and robust causal analysis in complex data environments.
C
Training
The detailed algorithmic procedure for our framework can be found in Algorithm I
In the preprocessing stage (line 1), we align text and image data with the spatio-temporal sequence to obtain XLest and Ximg- followed by feature extraction via BERT and CNN (line 2-3). We then compute cross-modke attention between Fst and the auxiliary modalities (lines 4-7), and apply gating-based fusion mechanism to obtain Frused (line 8-9). An adjacency matrix A is constructed by combining prior knowledge and SHAP-basee feature importance (line 10-11).
The prediction function STED (lines 12-17) models spatial dependencies via GCN and temporal dynamics via Mamba, and outputs the predictions via MLP. During training (lines 18-25), we generate predictions from both X. and backdoor-ade backdoor-adjusted adjuused fused features. Corresponding losses fpedd La, and Lmm are computed, and the model parameters are updated by minimizing the overall loss Call-
Algorithm 1: Training procedure of E2 -TSCSTP
Input: Opatio-tempora data Xg. text data X1ex,, image data Ximg Output: Predicted spatio-tempora sequence Y
Preprocessing: Align modalities obtain X Xing aligned with Xsi
2 > Feature Extraction and Normalization:
Fiex (ERT((X Fing mN))( Fa < Normalize (Xa);
> Feature Projection:
Fas Frest Fing + Project(Fas, Fucus Fing);
> Cross-modal Attention:
Attn Attna--ting < CMA(Fa. Fusss Fing);
b Fusion:
Fiused & Fuse Fu. Attna--ixt Attma--exx) 10 & Graph Construction:
A XA()) (0) (1 A)A SHAP,
12 Function STED(X, A)
Xspatial GCN(X: A):
14
X Xtemporal +- Mamba(XV X);
X Xencoded LayerNormmXypaaaa Xxempooaa);
16
MI.P(X (encoded);
return Y;
Dual-branch Training:
19
each training batch do
Yu + STED X A);
20
Y mm STED C-E-STEDDaaaaaaoaao (BackdoorAddjssteen((( (Fruscd), A);
Final < MLP(Ý Ymm)):
22
Epred Ly, Lmm +- ComputeLosses(( Pimats Pu, {mm):
23
24
Lall Epred + 9La + - Lmmi
25
Update model parameters by minimizing Lau:
16

---

D Additional Experiment Details
D.1 Dataset Details
Table summarizes the main characteristics of the datasets used in our study, including their spatial and temporal coverage, and available modalities.
Table 3: Dataset information
Modality
Dataset
#nodes
Spatial Coverage
Time Interval
Time Range
Time Series
Image
Text
Terra
100
United Kingdom
hours
01/1979 01/2024
BjTT
1260
Beijing
minutes
01/2022 03/2022
x
GreenFarthNet
1024
Global
day
01/2017 12/2020
x
BikeNYC
128
New York
hour
04/2014 09/2014
x
The detailed information of the dataset is follows:
Terra is large-scale, high-resolution, multi-modal dataset that provides hourly spatio- temporal data from 6,480,000 global grid points spanning 45 years, along with supplementary geo-images and text descriptions. In our analysis, utilize wind speed as the spatio-temporal sequence modality, LLM-generated text descriptions as the text modality, and topographic maps the image modality. We extract data from the Terra dataset covering the period from January 1979 to January 2024 at 1 spatial resolution, focusing on the United Kingdom with selection of 100 grid points.
Terra 4
BiTT 59]: BjTTis is a public multi-modal dataset for urban traffic prediction, containing 32,400 time-series records of traffic velocity and congestion from 1,260 roads in Beijing's 's Fifth Ring Road area over three months. The dataset also includes textual descriptions of traffic events such accidents and roadwork. For our research, modality, event descriptions as the text modality, and extract traffic data at 4-minute intervals from January March 2022.
road velocity the spatio-temporal
GreenEarthNet 3|: GreenEarthNet is a comprehensive, large-scale, multi-modal dataset de- veloped for vegetation estimation using satellite time series data. It comprises apatio-temporaa minicubes, each containing a series of 30 satellite images taken at five-day intervals, along with 150 daily meteorological observations and an elevation map. In work, adopt the Normal- ized Difference Vegetation Index (NDVT) as the modality for spatio-temporal sequences, and satellite images represent the image modality. Specifically,
crop each spatio-temporal minicube to a resolution of 32 x 32 pixels, corresponding to a 0.64 x 0.64 km area, to reduce computational overhead while preserving essential spatial patterns.
2i6) BikeNYC is large-scale urban mobility dataset that provides spatio-temporal trajectory data collected from the New York City bike-sharing system in 2014. The dataset includes detailed trip records, comprising trip duration, start and end station identifiers, and corresponding start and end timestamps. For our study, sequence modality to compare model performance a single-mode setting.
bike inflow the spatio-temporal
D.2 Baselines Description
(i) Single-modal spatio-temporal prediction methods.
D2STGNN 44)]
introduces novel framework that decouples diffusion and inherent signals in traffic data to improve spatio-temporal prediction.
ST-SSL 35 introduces apatio-tempore self-supervised learning framework for traffic flow prediction, leveraging adaptive graph-bac based data augmentation and supervised tasks to effectively capture spatial and temporal heterogeneity.
HimNet [10] proposes meta-learning approach that captures spatio-temporal heterogeneity via learned embeddings to generate adaptive forecasting parameters.
(ii) Causality- based spatio-temporal prediction methods.
17

---

introduces two-stage causal learning framework that identifies and exploits causal regions spatio-temporal data to enhance generalization and robustness.
NuwaDynamics 45
proposes a causal framework that leverages diffusion-based inpainting to address non-causal regions and improve generalization in spatio-temporal forecasting.
CaPaint 13
(iii) Foundation models for spatio-temporal prediction methods.
GPT-ST 27 introduces a spatio-temporal masked autoencoder with adaptive masking and hierarchical pattern encoding to pre-train customized representations for improved downstream prediction.
UniST (57] introduces universal spatio- prediction framework that utilizes pre-training and knowledge-guidee prompts to generalize across diverse urban tasks with minimal labeled data.
(iv) Multi-modal spatio-temporal prediction methods.
T3 [16] proposes multi-modal traffic forecasting model that fuses pre-trained textual and traffic embeddings to capture event impacts and address data sparsity.
From News to Forecast (FNF) 48 leverages LLM-based agents to integrate and reason over news and time series data, boosting forecasting accuracy through adaptive event incorporation.
D.3 Implementations
The parameters of all baseline models follow their paper's settings. All experiments are conducted on Rocky Linux 8.8 server equipped with NVIDIA A40 GPUs. We implement E²-CSTP using Python 3.8.20 and PyTorch 2.0.1. Model training is performed using the Adam optimizer with an initial learning rate of 0.001, which decays by factor of 0.5 every 5 epochs. We adopt early stopping based on the validation loss with patience 10 epochs to prevent overfitting and ensure stable convergence.
D.4 Metries
In this appendix, we provide the detailed calculations and interpretations of the evaluation metrics used in this work, including Mean Absolute Error (MAE), Root Mean Square Error (RMSE), and Mean Absolute Percentage Error (MAPE). These metrics are widely adopted spatio-temporal forecasting tasks for assessing the accuracy of predicted numerical values over time.
MAE measures the average magnitude of the errors between predicted and actual values, without considering their direction. RMSE calculates the square root of the average of squared differences between predictions and actual observations. MAPE expresses the prediction error percentage. providing scale- indepeddddnnnnt measure of accuracy.
MAE
(29)
N
1m1
N Σ 1=1
RMSE
(ys
02)2
(30)
N
MAPE 100% N
14 91
(31)
where yi is the ground truth, ds is the predicted value, and N is the total number of prediction instances.
D.5 Model Efficiency Study
Besides the total training time, we also evaluate the per-epoch runtime on datasets under the same batch size (64 for Terra and BikeNYC, 4 for BjTT and GreenEarthNet), Figure 6 further
18

---

shows that, on modil datasets, the per- per-epoch runtime of E2-CSTP is slower than the strongest baseline, which an outcome that is unsurprising given the extra cross-modal attention layers. By contrast, when trained on single-modal data BikeNYC, the model records notable 20 % speed-up per epoch compared with the second-best method, confirming that the proposed architectural refinements translate into tangible computational gains in purely single-modal settings.
DESTGNN
ST-SSL
HimNet
NewaDyyammes
CaPaint
GPT-ST
UnITT
FNF
E-CSTP
100
18P
Time/ Epochts)
Eposh (3)
Timo Epoech
Fipech
10"
107
10"
to
Terr
BiTT
GreenEanthNet
BileNYC
Figure 6: Model efficiency comparison the per-epoch runtime.
To further evaluate the performance of prediction module STED, conduct additional experi- ments by replacing with several Transformer- based Aiternaives-IInormme,, Autoformer, FEDformer, and iTrranffrmerr-on the BjTT, GreenEarthNet, and BikeB YC datasets, in addition to the results presented for Terra in the main text. Each variant is denoted as w/ In, w/ Auto, w/ FED, and w/ iTrans, respectively.
We evaluate both the prediction accuracy (measured by MAE) and computational cost (measured by per-epoch runtime) under the same batch size and training settings for fair comparison. As illustrated in Fig. 7 our proposed prediction module STED consistently achieves better accuracy, with improvements ranging from 15.64% 44. 44.20%, while also reducing per-epoch runtime by 14.20% to 89.25% across the additional datasets.
These results are consistent with our findings on the Terra dataset and further demonstrate that our prediction module STED offers a favorable trade-off between accuracy and efficiency across diverse spatio-temporal scenarios.
(03
4.5
MAT Tinccipoch
(00)
5000
0:20
MAE TrasFyush
2003
Timm//Fpooh
50
Time Epooc (4)
5000
0.16
2000
Autoo
FED
(a) BjTT
(b) GreenFarthNet
(c) BikeNYC
Figure 7: E2CCTT with Transformer- based variants across datasets.
D.6 Parameter Sensitivity Study
We evaluate effects hyperparameters of the graph fusion factor among {0. 0.,5..2,.0 0.5. 0.75, 1} and dual-branch loss balancing 3 among {0, 0.25, 0.5, 0.75, 1). shown in Fig. 8 and Fig.
MAE
MAPE
10
MAB (N)
AANPEO MATE
X6NPE MALE
MADE
(a) Terma
(b) BjTT
(c) GreenEanthNel
(d) BikeNYC
Figure 8: Parameter sensitivity study X.
The hyperparameter À controls the degree of integration between the spatial adjacency matrix and the causal matrix. moderate value of À (e.g., 0.25 or 0.5) generally yields better performance, as balances the influence of raw spatial structure and the refined causal graph.
19

---

Specifically, for Terra and GreenEarthNet (remote sensing tasks), lower À (0.25) works better, which places greater emphasis on the causal graph while reducing the weight of the raw spatial structure. These datasets contain stable spatial correlations (e.g., land use or vegetation types), making the data-driven refinement more valuable than default proximity-based connections. This suggests that in these datasets, dominated by stable and consistent spatial patterns, relying more on causal refinement improves robustness and avoids redundancy. In contrast, BjTT and BikeNYC (urban mobility tasks) benefit from more balanced integration (0.5), where the influence of the raw spatial and causal graphs is equally weighted. These datasets involve highly dynamic and noisy opatio-tempont flows, requiring blend of prior structure and adaptive refinement. This indicates that both types of structural information are important for these datasets, likely due to the presence of more dynamic, heterogeneous, noisy ppatial-tepoon patterns in urban environments.
MAE
MAPE
E MATE 150 XAV
MATE (50 MAL
6 MATE (50
MARRE
240
(8) Terra
BJTT
(c) GreenEarthNet
(d) BikeNYC
Figure 9: Parameter sensitivity study on 8.
controls the trade-off between the main and the auxiliary branches, where a larger / places more emphasis on the spatio-temporal features, and smaller 3 increases the relative weight of the auxiliary branch, which incorporates multi-modal data and causal intervention.
For Terra, the optimal /3 0.75, highlighting the importance of spatio-temporal modeling in this dataset. GathhNarthNet and BjTT achieve optimal performance with of 0.5, suggesting balanced contribution of both branches. In contrast, BikeBYC benefits from a / of 0.75. Although this dataset does not include multi-modal inputs, the auxiliary branch still encodes causally refined spatio-temporal features, which contribute to improved prediction performance. Similar reasoning applies to 3, where datasets with stronger exogenous influence (e.g., weather, events) benefit from greater auxiliary-brant -brnchh emphasis, while those with purer internal spatio-temporal structure favor the primary prediction path.
E
Limitations and Future Work
E²-CSTP has the potential to benefit critical real-world applications such as intelligent transportation, environmental monitoring, and urban infrastructure management by enabling more accurate and efficient ppatio-temporal predictions. While framework handles spatio-temporal sequence, text and image modalities, other data types such as audio, LiDAR, or social signals (e.g., user interactions) not considered. Future work could explore generalized framework capable of handling wider range of modalities.
20

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction cover the contributions and scope of the paper regarding building an effective and efficient causal multi-modal spatio-temporal prediction framework.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In the Appendix we systematically discuss the limitations of our research and outline directions for future work.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
21

---

Answer:
[Yes]
Justification:
We provide a proof in the Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide the implementation details for experiment settings in Section 5 and Appendix [D3] We also provide the source code and datasets at /https://gittie com/
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results. 5. Open access to data and code
22

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
We provide the source code and datasets https://github.ccom 2JU-DATLLYE2-CSS
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.cc.
cublic//uuddes/Coeeeubbiss
www.sssinoooiiillooo for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ wwtp:/cccppppgguuuiiaeoeaec
Stleee//CoSSSSbmsssS.S.. www.ssssiicggcooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the details for experimental settings in Section and Appendix D.3 Full details are shown in the provided code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The experimental results reported in the paper are the average values of five independent experimental runs, but error bars are not included.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
23

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In this paper, we provide detailed information about the experimental resources, including GPU configurations used Appendix D.3
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.. httpp//pp...ppulllttttttccccaaaa
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., s if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the contribution and the societal impacts of the work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the original paper that produced the code package or dataset. We have explicitly mentioned the licenses and terms of use for each asset and have ensured full compliance with these terms throughout our research.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
25

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.com//aaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
26

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "Nbx3keUmXO", "reviewer": "Reviewer_Z2uh", "summary": "The paper introduces E²-CSTP, a novel framework for multi-modal spatio-temporal prediction that addresses three key challenges: (1) insufficient fusion of multi-modal data, (2) confounding factors obscuring causal relations, and (3) high computational complexity. The proposed framework integrates cross-modal attention and gating mechanisms for effective multi-modal fusion, employs a dual-branch causal inference approach to mitigate bias, and combines GCN with the Mamba architecture for efficient spatio-temporal encoding. Extensive experiments on four real-world datasets demonstrate that E²-CSTP outperforms nine state-of-the-art baselines in accuracy (up to 9.66% improvement) and computational efficiency (17.37%-56.11% reduction in overhead).", "questions": "See above", "limitations": "See above", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. Comprehensive Multi-Modal Fusion: The framework effectively integrates heterogeneous data (spatio-temporal sequences, text, and images) using cross-modal attention and adaptive gating, enabling richer feature representations.\n2. Innovative Causal Inference: The dual-branch design disentangles true causal relations from confounders, enhancing interpretability and robustness in predictions.\n3. Computational Efficiency: By combining GCN for spatial encoding and Mamba for temporal modeling, the framework achieves linear complexity, significantly reducing computational overhead compared to Transformer-based methods.\nWeaknesses\n\n1. Limited Modality Scope: The framework focuses on text, images, and spatio-temporal data, but other modalities (e.g., audio, sensor networks) are not explored, which could limit its applicability in certain domains.\n2. Dependence on Pre-trained Models: The use of BERT and CNN for feature extraction assumes availability of high-quality pre-trained models, which may not hold for all languages or niche domains.", "was_revised": true, "final_justification": "I maintain the score as 4 and vote to accept this work. Because the authors answer and solve my concerns.", "rebuttal": "We sincerely thank the reviewer for their constructive feedback and for recognizing the contributions of our work. Our responses to the **Weaknesses** are detailed below.\n```\nW1: Limited Modality Scope: The framework focuses on text, images, and spatio-temporal data, but other modalities (e.g., audio, sensor networks) are not explored, which could limit its applicability in certain domains.\n```\n- In this paper, we focus on the **most representative and widely used modalities** in the field of spatio-temporal prediction. Specifically, we have considered **text, image, and spatio-temporal data**, as these are the most available in real-world applications such as intelligent transportation and weather forecasting [1]. While many recent studies [2–3] have investigated the fusion of **two** modalities—e.g., image and spatio-temporal data, or text and image, for forecasting and decision-making tasks, **few (if any) prior works attempt to jointly model all three modalities within a unified framework**, especially when these modalities carry complementary yet heterogeneous information. To the best of our knowledge, our work is the first to integrate all three data sources for spatio-temporal prediction, enabling a more holistic and effective representation learning approach. Our selection of modalities is also informed by practical considerations. For example, **audio data** are rarely used in spatio-temporal forecasting, and **sensor network** data in this context typically refer to measurements such as traffic flow or wind speed—**which have been already incorporated into our framework**.\n\n- On the other hand, our framework is **inherently extensible**. The proposed **cross-modal attention** and **gating mechanisms** are designed to be modality-agnostic—any data source that can be represented as a feature vector can be seamlessly incorporated into the model without requiring architectural changes. **We will clarify this aspect in the revised manuscript**.\n\n- **Finally**, we agree that the suggested additional modalities (e.g., audio) represent valuable future directions, and we plan to explore them within our framework in future work.\n\n\n\n```\nW2: Dependence on Pre-trained Models: The use of BERT and CNN for feature extraction assumes availability of high-quality pre-trained models, which may not hold for all languages or niche domains.\n```\n\n- We clarify that our framework **does not depend heavily on high-quality pre-trained models**. Instead, we deliberately adopt **simple and widely used architectures**, specifically **BERT for text and CNN for images**, because of their proven effectiveness and general applicability across domains [4–5]. These components serve as **off-the-shelf encoders**, and our design does not assume access to domain-specific or highly specialized pre-trained models. \n- Specifically, **for text**, while we use the standard BERT architecture as a text encoder, we do not rely on domain-adapted versions, and our framework remains compatible with lightweight or randomly initialized language models. For **image encoding**, we use a CNN not as a fixed pre-trained model, but as a trainable, lightweight extractor—further emphasizing that **our model remains fully trainable end-to-end** without requiring strong prior knowledge.\n- In low-resource languages or niche domains where pre-trained language models may be unavailable or underperforming, our framework can still function by simply replacing the encoder with a fine-tuned or even randomly initialized one. The architecture is modular and allows for easy substitution or adaptation of modality-specific components. **We will clarify this aspect in the revised manuscript**.\n\n\n\nWe gratefully acknowledge the reviewer's insightful feedback, which has significantly strengthened our work. **Should any aspects require further elaboration, we stand ready to provide additional clarification and supporting materials**.\n\n\n\n**Reference:**\n\n[1] Terra: A Multimodal Spatio-Temporal Dataset Spanning the Earth, NeurIPS 2024\n\n[2] Towards Effective Fusion and Forecasting of Multimodal Spatio-temporal Data for Smart Mobility, CIKM 2024\n\n[3] Multi-Modality Spatio-Temporal Forecasting via Self-Supervised Learning, IJCAI 2024\n\n[4] See How You Read? Multi-Reading Habits Fusion Reasoning for Multi-Modal Fake News Detection, AAAI 2023\n\n[5] Time-MMD: Multi-Domain Multimodal Dataset for Time Series Analysis, NeurIPS 2024"}, {"review_id": "3fXxXeNwze", "reviewer": "Reviewer_Skfh", "summary": "This ppaper presents a new framework for spatio-temporal prediction. The framework has two branches, a main branch that focuses on spatio-temporal prediction and a secondary branch that models additional modalities and introduces causal interventions. The authors also combined GCN and Mamba for efficiency. They used four datasets to evaluate the proposed framework, demonstrating its improvement in accuracy and computational overhead relative to the SOTA approach.", "questions": "1. which of the main components of the framework are customized for spatio-temporal prediction? Which parts can be generalized broadly to other tasks?\n2. the article mentions that the model employs causal reasoning to reduce the impact of confounding factors, how can it be explained more intuitively how the model distinguishes between causality and confounding factors?", "limitations": "Yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength.\n1. Rationality of design. The key innovation of this framework lies in the designed auxiliary branches and the introduction of the GCN and Mamba architectures, a combination that is intuitively sensible and exemplifies the innovation of hybrid architectures.\n2. quality of the paper. This paper is well written and easy to understand. Ideas, task definitions, and methods are described clearly.\n3. comprehensive experiments. The baseline selected for the comparison experiments is detailed and innovative. It seems to have enough workload.\n4. open project.\nWeekness \n1. Although models use causal reasoning to reduce the impact of confounding factors, their internal mechanisms remain relatively complex, making it difficult to intuitively understand the model's predictive process. How do these steps further influence the model's decisions? A visual analysis would be nice.", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewer for their thoughtful and constructive comments, and we greatly appreciate their recognition of our contributions. Below, we provide detailed responses to the specific **Weaknesses** and **Questions** raised.\n\n```\nW1: Although models use causal reasoning to reduce the impact of confounding factors, their internal mechanisms remain relatively complex, making it difficult to intuitively understand the model's predictive process. How do these steps further influence the model's decisions? A visual analysis would be nice.\n```\n\nOur **E$^{2}$‑CSTP framework** enhances accuracy and interpretability by incorporating causal reasoning into a modular architecture. Central to our approach are a **DeepSHAP-based adjacency matrix** and a **dual-branch causal adjustment module**, which jointly help disentangle true causal effects from spurious correlations introduced by confounding variables. Below, we give detailed explanation to illustrate the process for better understanding.\n\n- The **DeepSHAP-based adjacency matrix** captures interdependencies among spatial units by quantifying feature importance. This matrix serves as the foundation for a Graph Convolutional Network that effectively models spatial interactions based on data-driven relevance rather than fixed assumptions.\n- The **dual-branch causal adjustment** consists of two complementary pathways. The **main branch** leverages only spatio-temporal data, ensuring that predictions are grounded in directly observable information. The **auxiliary branch** incorporates fused multimodal data (e.g., text, image), enriching the representation space with complementary cues. The outputs of **both branches** are then reconciled through an **interventional adjustment module**, which estimates the causal effect of multimodal features while explicitly blocking backdoor paths—thereby mitigating the influence of both observed and latent confounders.\n\nOverall, the **causal adjustment mechanism** and the **dual-branch design** enhance the model’s robustness and interpretability by isolating the contributions of each modality and clarifying their impact on the final prediction.\n\nWe appreciate the reviewer’s suggestion to include visualizations. While **the rebuttal phase does not permit the inclusion of additional figures**, we will incorporate detailed visualization analyses in the revised version of the paper. These visualizations will illustrate how the dual-branch architecture and interventional adjustments influence model decisions.\n\n\n\n```\nQ1: Which of the main components of the framework are customized for spatio-temporal prediction? Which parts can be generalized broadly to other tasks?\n```\n\nThis is a thoughtful question. The **E²‑CSTP framework** is primarily tailored for spatio-temporal prediction, but it also incorporates several components with broad applicability beyond this domain.\n\n- The **Spatio-Temporal Encoding and Decoding (STED)** module is highly specialized. It combines the **Mamba** model for efficient temporal sequence modeling and **GCNs** to capture spatial dependencies among graph-structured nodes. This design explicitly leverages spatial and temporal characteristics central to tasks like traffic forecasting and environmental monitoring.\n- In contrast, the **cross-modal feature fusion mechanism**—comprising modality **alignment**, **cross-modal attention (CMA)**, and a **dynamic fusion gate**—is modality-agnostic and thus broadly applicable. This component can be adapted to diverse multi-modal learning problems, such as healthcare diagnostics, multimedia analysis, and recommendation systems, where integrating heterogeneous modalities (e.g., text, images, sensor signals) is essential.\n- Moreover, the **dual-branch causal adjustment framework**, originally designed to disentangle confounders in spatio-temporal contexts, is also broadly extensible. By explicitly contrasting the **main branch**, which uses only spatio-temporal sequences $X_{\\rm st}$, with the **auxiliary branch** incorporating fused multi-modal features $F_{\\mathrm{fused}}$, the model isolates stable causal effects from potentially spurious correlations introduced by external signals (e.g., image $E$, text $C$). The learned weighting parameter $\\beta$ enables flexible balancing, making this causal adjustment suitable for various downstream tasks such as forecasting, classification, and risk assessment.\n- Finally, the **interventional adjustment strategy**, while developed for correcting spatio-temporal biases, provides a flexible causal inference tool applicable to any structured setting involving confounding factors.\n\nTogether, these designs demonstrate a balance between domain-specific customization for spatio-temporal data and modular components readily adaptable to other multi-modal and causal inference tasks.\n\n\n\n```\nQ2: The article mentions that the model employs causal reasoning to reduce the impact of confounding factors, how can it be explained more intuitively how the model distinguishes between causality and confounding factors?\n```\nThanks for this question. Below, we clarify how E$^{2}$‑CSTP separates causality and confounding factors.\n\n- **Causal matrix refined by DeepSHAP scores.**\n  To capture latent dependencies among spatial units and enhance interpretability, we estimate an adjacency matrix $\\mathbf{A}^{\\text{SHAP}}$ using DeepSHAP-based feature attribution, where each value $\\phi_{i,j}$ reflects the influence of node $i$ on node $j$. When a prior adjacency $\\mathbf{A}^{(0)}$ is available, we construct a hybrid graph $\\mathbf{A} = \\lambda \\mathbf{A}^{(0)} + (1 - \\lambda) \\mathbf{A}^{\\text{SHAP}}$, where $\\lambda$ controls the balance between structural priors and data-driven importance. This formulation allows the model to incorporate empirically supported dependencies while reducing the influence of potentially spurious or irrelevant connections.\n\n- **Dual‑branch optimisation.**\n  Model training employs two parallel predictors.  The main branch is conditioned exclusively on the observed spatio‑temporal data $X_{\\rm st}$, whereas the auxiliary branch operates on the complete multi-modal representation $F_{\\text{fused}}$, which potentially contains confounding artefacts. A lightweight gating MLP learns per‑sample fusion weights.  When prediction improvements originate solely from signals absent in the primary branch, the gating mechanism down‑weights their contribution, preserving influence for features that demonstrate stable causal utility.\n- **Interventional adjustment with gradient regularization.**\n  Immediately before the STED module, we apply a learned intervention of the form:\n\n  $$\n  \\hat{x} = x + x \\odot W\\bigl[\\alpha_{1}h(S) + \\alpha_{2}p(E) + \\alpha_{3}q(C)\\bigr],\n  $$\n\n  where $W$ is a learnable weight matrix, and $h(\\cdot)$, $p(\\cdot)$, $q(\\cdot)$ are feature transformations of the latent confounder $S$, image features $E$, and text features $C$, respectively.\n\n  To ensure this adjustment removes confounding bias, we impose a gradient regularization term that drives $\\partial \\hat{x} / \\partial S \\to 0$. This penalty encourages the model to re-encode $x$ such that the latent confounder $S$ has no effect on the adjusted representation $\\hat{x}$, effectively blocking the backdoor path $X_{\\rm st} \\leftarrow S \\rightarrow Y_{\\rm st}$ and isolating only the component of $X_{\\rm st}$ that carries a stable causal influence on the target $Y_{\\rm st}$.\n\nCollectively, the DeepSHAP‑based causal matrix, the dual‑branch training, and the interventional adjustment ensure that E$^{2}$‑CSTP responds primarily to genuine causality while suppressing confounding factors.\n\n\n\nAgain, the authors sincerely thank the reviewer for their thoughtful engagement with our manuscript. **For any remaining questions or required clarifications, we would be pleased to submit supplemental documentation as needed**."}, {"review_id": "sUr8D9EPVx", "reviewer": "Reviewer_THYi", "summary": "This paper presents E2-CSTP, a novel framework for causal, multi-modal spatio-temporal prediction, aiming to address challenges in multi-modal data fusion, causal confounding, and model efficiency. E2-CSTP integrates cross-modal attention and gating mechanisms to fuse multi-modal inputs and employs a dual-branch design—one branch for spatio-temporal prediction and another for causal intervention to mitigate bias. The model combines Graph Convolutional Networks (GCNs) with the Mamba architecture to enhance computational efficiency. Experimental results on four real-world datasets show up to 9.66% accuracy improvement and 17.37%–56.11% reductions in computational overhead compared to nine state-of-the-art baselines.\n\nHowever, the paper falls short on two key points. First, it naively combines existing tools, offering limited novelty given the abundance of similar works on multi-modal data fusion. Second, if the primary goal is prediction accuracy, the incorporation of causal inference is unnecessary and not well-justified in this context.", "questions": "There are several methodological concerns that should be addressed. First, the use of unified binary alignment matrices presents major limitations in handling irregular time series (or longitudinal data) and irregular spatio-temporal data. Second, several key notations are introduced without explicit definitions. For example, what are \\( F_{st} \\) and \\( \\tilde{F}_{st} \\)? These should be clearly defined prior to their use. Similarly, the meaning of \\( N_{st} \\), and the relationship between \\( N \\) and \\( \\{N_i\\} \\), should be clarified. Third, it remains unclear whether the confounding variable \\( S \\) varies across space and time. This is a significant issue, as spatio-temporal systems often involve unobserved spatio-temporal confounders, and causal relationships may themselves vary over space and time.", "limitations": "yes.", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The authors claimed four major contributions in this work. First, they propose a unified multi-modal spatio-temporal prediction framework that integrates environmental images, event-related text, and time-series data using cross-modal attention and adaptive gating. Second, they introduce a dual-branch causal design, where the main branch captures spatio-temporal patterns and the auxiliary branch applies causal interventions to reduce confounding bias. Third, they develop an efficient hybrid model combining GCN for spatial encoding and Mamba for temporal modeling to accelerate inference. Fourth, extensive experiments on four real-world datasets demonstrate up to 9.66% improvement in accuracy and 17.37%–56.11% reductions in computational overhead compared to nine baselines.\n\nThere are several methodological concerns that should be addressed. First, the use of unified binary alignment matrices presents major limitations in handling irregular time series (or longitudinal data) and irregular spatio-temporal data. Second, several key notations are introduced without explicit definitions. For example, what are \\( F_{st} \\) and \\( \\tilde{F}_{st} \\)? These should be clearly defined prior to their use. Similarly, the meaning of \\( N_{st} \\), and the relationship between \\( N \\) and \\( \\{N_i\\} \\), should be clarified. Third, it remains unclear whether the confounding variable \\( S \\) varies across space and time. This is a significant issue, as spatio-temporal systems often involve unobserved spatio-temporal confounders, and causal relationships may themselves vary over space and time.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for offering the valuable feedback. We have addressed each of the **Weaknesses**/**Questions** as outlined below.\n\n```\nW1&Q1: First, the use of unified binary alignment matrices presents major limitations in handling irregular time series (or longitudinal data) and irregular spatio-temporal data.\n```\n**We believe there may be a misunderstanding regarding the intent of our binary alignment matrices.** Actually, the used **unified binary alignment matrices were explicitly designed to address irregularities** in both the temporal and spatial dimensions.\n\nAs described in Section 4.1 (lines 172-181, Page 4), our approach **does not rely on regular sampling intervals or uniform grids**. Instead, alignment is performed via a **nearest-neighbor matching scheme**:\n\n* For **temporal alignment** ($\\mathbf{M}^t_{i,j}$), each textual or visual observation is mapped to the closest timestamp in the spatio-temporal sequence, regardless of sampling rate.\n* For **spatial alignment** ($\\mathbf{M}^s_{i,j}$), image observations are matched to spatial coordinates via Euclidean proximity, again without assuming spatial regularity.\n\nThis strategy allows our model to **naturally handle missing, sparse, or unevenly distributed data**, and is particularly suitable for longitudinal settings where different modalities may be observed at varying granularities. More importantly, by using **binary alignment matrices** instead of interpolation or re-sampling, our method **avoids imposing artificial continuity or temporal regularity assumptions**, thereby preserving the structural fidelity of each modality. This design makes our alignment strategy both **flexible and robust** for real-world irregular spatio-temporal scenarios.\n\n\n\n```\nW2&Q2: Second, several key notations are introduced without explicit definitions. For example, what are (F_{st}) and (\\tilde{F}{st})? These should be clearly defined prior to their use. Similarly, the meaning of (N{st}), and the relationship between (N) and ({N_i}), should be clarified.\n```\n\nThank you for pointing this out. **Upon careful review**, we confirm that **all the pointed notations in question**, such as $F\\_{\\mathrm{st}}$, $\\tilde{F}\\_{\\mathrm{st}}$, $N\\_{\\mathrm{st}}$, $N$, and $\\{N\\_i\\}$, **are defined in the paper**.\n\n- **Lines 192–195** provide explicit definitions for $F\\_{\\mathrm{st}}$ and $\\tilde{F}\\_{\\mathrm{st}}$. To facilitate interactions among features from different modalities within a shared latent space, we project the **spatio-temporal features** $F\\_{\\mathrm{st}}$, **text features** $F\\_{\\mathrm{text}}$, and **image features** $F\\_{\\mathrm{img}}$ into a unified hidden dimension $d$ using three separate fully connected layers. This results in modality-specific representations $\\tilde{F}\\_{\\mathrm{st}}, \\tilde{F}\\_{\\mathrm{text}}, \\tilde{F}\\_{\\mathrm{img}} \\in \\mathbb{R}^{T\\_{\\mathrm{st}} \\times N\\_{\\mathrm{st}} \\times d}$.\n\n- Regarding the spatial node count:\n  - $N$ is defined in **line 145** as the total number of spatial nodes.\n  - In **line 152**, we generalize this as $N\\_i$ to denote the number of spatial nodes for each modality $i$.\n  - In **line 185**, we use $N\\_{\\mathrm{st}}$ to specifically denote the number of nodes in the spatio-temporal modality, which is the prediction target.\n  - The usage of $N$ in **line 157** was intended to refer to $N\\_{\\mathrm{st}}$, and we will revise the text to make this reference clearer in the final version.\n\n\n\n```\nW3&Q3: Third, it remains unclear whether the confounding variable (S) varies across space and time. This is a significant issue, as spatio-temporal systems often involve unobserved spatio-temporal confounders, and causal relationships may themselves vary over space and time.\n```\nThis is a thoughtful question. We would like to clarify that **our model is designed to implicitly capture variability across space and time** through multiple architectural and training mechanisms:\n\n- **Spatio-temporal dependencies are preserved via STED.**  \n  The **Spatio-Temporal Encoding-Decoding (STED)** module (Section 4.3) captures local temporal trends and spatial interactions via stacked **Mamba** and **Graph Convolutional** blocks. Since these layers operate directly over the spatio-temporal graph, the model is able to represent and propagate latent patterns—including unobserved confounding—that may vary over time and space.\n\n- **Causal adjustment is applied pointwise in space and time.**  \n  As shown in Eq. (11), the **intervention adjustment** is applied to each **spatial node at each time step independently**. The learned adjustment term is conditioned on features specific to that node and timestep, which effectively supports localized modeling of latent influences. Though $S$ is not observed, its effects are approximated via node-level adjustment vectors $s_i$, which are learned in a **data-driven, fine-grained** manner.\n\n- **Flexible latent approximation via fusion and attention.**  \n  The **dual-branch causal structure**, which includes **attention-based cross-modal fusion** and **DeepSHAP-based refinement**, allows the model to dynamically adapt to region-specific and temporally-varying relationships. This removes the need to assume stationarity or homogeneity in the causal graph.\n\nOur framework supports its **implicit, localized representation** via (i) spatio-temporal feature encoding, (ii) node-level intervention adjustment, and (iii) joint training across the full spatial and temporal extent of the data. As such, the model **naturally accommodates unobserved confounders that vary across space and time**. Also, we appreciate the reviewer for raising this subtle but important point, and will revise the final manuscript to clarify these modeling assumptions more explicitly.\n\n\n\nOverall, we appreciate the reviewer's thorough engagement with our work. While some questions appear to stem from potential misunderstandings, **we sincerely value this opportunity to improve the manuscript's clarity**. All suggested points have been carefully addressed above."}, {"review_id": "vq8iseQAmN", "reviewer": "Reviewer_Buzp", "summary": "The author proposes E^2-CSTP, a multi-modal spatial-temporal prediction framework that integrates multiple sources of data. The framework has three main components: cross-modal feature fusion, dual-branch causal inference, and spatial-temporal encoding. Experiments are conducted on different datasets, showing significant improvements over the current SOTA methods, achieving a notable improvement in accuracy and computational burden.", "questions": "As weakness", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength\n- The paper provides comprehensive empirical studies against 9 baseline methods on 4 datasets on multiple dimensions.\n- The framework's approach to fuse time-series, text, and image data together is a novel approach.\n- Linear-time STED predictor replaces quadratic self-attention, resulting in good improvement on computational efficiency.\n\n\nWeakness\n- The adjacency matrix is built from DeepSHAP importances and without comparison to standard causal-discovery methods. It is stated that the matrix was updated every P epochs, but the author did not mention how the value of P was determined. No ablation test on different values of P\n- For cross-modal feature alignment, the inputs are aligned using deterministic ST matrix thresholds. And the experiment did not test the robustness of this. This suggests the model presumes perfectly matched geo-temporal stamps. Such practice will be sensitive to issues like missing data points or irregular sampling\n- The claimed linear time performance comes after every batch has already spent time and resources on the full BERT and the CNN. But the paper did not address the time performance of this part. \n- The performance is highly sensitive to the manually tuned hyperparameters. Fig 8 and 9 show that suboptimal λ and the β value can degrade the performance on an observable scale. And the author picks up different values for different datasets, showing issue of strong coupling. The generality of this framework is questionable since it has to do a grid search on each new dataset for those values.", "was_revised": true, "final_justification": "The author has justified my questions and concerns about the manuscript in the rebuttal. The author ensures that they will add additional ablation and explicit discussion based on my reviews. Although the author mentioned that the grid search on hyperparameters is unavoidable, they justify that it will be inexpensive. So this does not contradict the claim that the proposed model is efficient and scalable. So I would like to recommend this paper as borderline accept.", "rebuttal": "We thank the reviewer for the valuable feedback. We have addressed each of the concerns as outlined below.\n```\nW1: The method builds the adjacency matrix using DeepSHAP without comparing it to standard causal discovery approaches, and it lacks justification or ablation for the choice of the update interval P.\n```\n**i) Without comparison to standard causal-discovery methods**. We respectfully clarify that **our experiments do include comparisons against two state-of-the-art methods that explicitly adopt standard causal discovery pipelines**:\n\n- **NuwaDynamics** \\[1] introduces a two-stage learning framework that first identifies causal regions via conventional causal discovery techniques, and then performs localized prediction with improved robustness.\n- **CaPaint** \\[2] constructs a causal diffusion-based inpainting mechanism to mitigate non-causal regions and enhance generalization.\n\nThese models thus serve as representatives of standard causal-discovery-based pipelines. Our method outperforms both NuwaDynamics and CaPaint across multiple benchmarks (see Table 1), highlighting the efficacy of our **DeepSHAP-based alternative**, which derives interpretability from data-driven attribution.\n\n**ii) Choice of update interval $P$**. Following Cheng et al. \\[3], we construct the adjacency matrix using **DeepSHAP-based feature importances** to approximate inter-node causal dependencies. In our implementation, we **update the adjacency matrix every $P = 5$ epochs**, a value selected through empirical observation and computational considerations:\n\n- **Stability consideration**: Updating too frequently (e.g., every 1–2 epochs) during early training stages led to minor or noisy structural changes, as the underlying feature distributions remained relatively static.\n- **Meaningful structural updates**: A moderate delay in updates (i.e., P = 5) gives the model sufficient time to evolve its internal representations, enabling more informative and stable DeepSHAP-based graph refinements.\n- **Computational efficiency**: SHAP computation is resource-intensive. Excessive updates significantly increase overhead without commensurate benefits in structure accuracy or final performance.\n\nFurthermore, we apply a **learnable parameter** $\\lambda \\in [0,1]$ to blend the prior graph $\\mathbf{A}^{(0)}$ with DeepSHAP-based estimates, mitigating noise from local fluctuations and ensuring smoother structural transitions. We will **add clarifying discussion and ablation results on different $P$ values** in the revised version.\n\n**Reference:**\n\n[1] NuwaDynamics: Discovering and Updating in Causal Spatio-Temporal Modeling, ICLR 2024\n\n[2] Causal Deciphering and Inpainting in Spatio-Temporal Dynamics via Diffusion Model, NeurIPS 2024\n\n[3] CausalTime: Realistically Generated Time-series for Benchmarking of Causal Discovery, ICLR 2024\n\n```\nW2: Cross-modal alignment assumes perfect geo-temporal matching and lacks robustness to missing or irregular data.\n```\nWe appreciate the reviewer’s attention to the alignment mechanism. However, we would like to clarify that **our alignment strategy does not rely on perfect geo-temporal matching** or fixed thresholds. Rather, it is explicitly designed to accommodate irregular sampling and missing observations:\n\n* **Nearest-neighbor alignment, not threshold-based matching.**\n   As described in Section 4.1 and Eq. (1), we align multi-modal inputs to the spatio-temporal backbone using an argmin operation over distance, not fixed thresholds. Specifically:\n\n   * Temporal alignment assigns each observation to the nearest timestamp $\\tau^{(k)}_\\text{st}$, regardless of irregular intervals.\n   * Spatial alignment finds the closest spatial point $\\rho^{(k)}_\\text{st}$ via Euclidean distance.\n\n   This approach remains robust even when data are unevenly spaced, asynchronously sampled, or partially missing. No assumption is made that modalities arrive in synchrony.\n\n* **Discrete binary matrices preserve data fidelity.**\n   We use sparse binary matrices ($\\mathbf{M}^t, \\mathbf{M}^s$) instead of interpolating or discretizing inputs, maintaining precise alignment with minimal distortion. This is particularly suitable for handling event-based or sparse modalities (e.g., social media, satellite images) where timestamps may not align exactly with the underlying time grid.\n\nOverall, **the model does not assume perfect geo-temporal correspondence**. Instead, the alignment step is based on adaptive nearest-neighbor matching, which enables robust integration of heterogeneous, irregular, and incomplete signals.\n\n```\nW3: The claimed linear time performance comes after every batch has already spent time and resources on the full BERT and the CNN. But the paper did not address the time performance of this part.\n```\n**We would like to clarify the reported time performance** in this paper. We evaluate both the overall end-to-end time cost and the time cost after replacing the prediction module.\n\n* Figure  4 reports the overall time, **including the full BERT and CNN encoders**, thus accounting for the total inference cost.\n\n* Figures  5 and 7 show the **end-to-end time performance** after replacing the full BERT+CNN encoders with our lightweight prediction module. This isolates the gain brought by replacing a quadratic Transformer with our linear GCN‑Mamba block.\n\nOur prediction core, whose complexity scales with sequence length $T$ and graph size $N$, drives the speed-up, while the shared text and image encoders merely handle preprocessing and therefore do not affect the relative ranking\n\n**We have now supplied an explicit complexity analysis of the preprocessing stage to make this clear**. The time complexity of the Cross-Modal Feature Fusion module is dominated by the cross-modal attention operations. Specifically, after aligning multi-modal inputs (text and image) to the spatio-temporal grid of size $T_{\\text{st}} \\times N_{\\text{st}}$, the module computes pairwise attention between modalities via two Cross-Modal Attention (CMA) blocks: one between spatio-temporal and text features, and another between spatio-temporal and image features. Each CMA involves computing attention scores and weighted values across all spatio-temporal slots, leading to a time complexity of $O(B \\cdot (T_{\\text{st}} N_{\\text{st}})^2 \\cdot d)$, where $B$ is the batch size and $d$ is the hidden dimension. Additional operations—including multi-modal alignment, BERT and CNN encoding, linear projections, and fusion gating—contribute lower-order terms bounded by $O(B \\cdot T_{\\text{st}}^2 \\cdot d)$ or $O(B \\cdot T_{\\text{st}} \\cdot N_{\\text{st}} \\cdot d)$, and thus do not alter the overall complexity. Therefore, the final time complexity of the fusion module is $O(B \\cdot (T_{\\text{st}} N_{\\text{st}})^2 \\cdot d)$, reflecting the quadratic cost of pairwise cross-modal interactions across the spatio-temporal grid.\n\n```\nW4: The model’s performance is highly sensitive to hyperparameters like $\\lambda$ and $\\beta$, requiring dataset-specific tuning and limiting generalizability.\n```\n\nWe thank the reviewer for this insightful observation. We acknowledge that the optimal settings of the **graph-fusion weight** $\\lambda$ and the **branch-balancing factor** $\\beta$ vary across datasets. However, we respectfully clarify that this variation **reflects inherent dataset characteristics rather than a weakness in model generality or robustness.**\n\n**i) Optimal ranges are consistently narrow and interpretable**\n\nAcross all four datasets, the best-performing configurations consistently lie within the following intervals:\n$$\n\\lambda\\in[0.25,0.50],\\quad \n   \\beta\\in[0.50,0.75],\n$$\nThis **central concentration** demonstrates that the model is **not overly sensitive** to hyperparameter fluctuations, and that reasonable defaults can be reliably used when dataset-specific tuning is unavailable.\n\n**ii) Dataset-specific behaviors are explainable and stable**\n\nWe also note that the shifts in optimal $\\lambda$ and $\\beta$ settings **align with identifiable characteristics of the datasets**, as detailed in Appendix D.6 (Page 20):\n\n- For **Terra** and **GreenEarthNet** (remote sensing tasks), we observe that:\n  - A **lower** $\\lambda = 0.25$ yields better results, suggesting that the **SHAP-derived causal graph** provides more discriminative information than the raw prior adjacency.\n  - These datasets contain **stable spatial correlations** (e.g., land use or vegetation types), making the data-driven refinement more valuable than default proximity-based connections.\n\n- For **BjTT** and **BikeNYC** (urban mobility tasks), a **balanced** $\\lambda = 0.50$ performs best:\n  - These datasets involve **highly dynamic and noisy spatio-temporal flows**, requiring a blend of prior structure and adaptive refinement.\n  - This trade-off captures both stable commuting hubs (via the prior graph) and emergent short-term behaviors (via SHAP refinement).\n\nSimilar reasoning applies to $\\beta$, where datasets with stronger exogenous influence (e.g., weather, events) benefit from greater auxiliary-branch emphasis, while those with purer internal spatio-temporal structure favor the primary prediction path.\n\n**iii) Practicality of tuning**\n\nFinally, we note that:\n\n- Only **two scalar hyperparameters** ($\\lambda$, $\\beta$) are exposed for tuning.\n- Their ranges are **bounded, low-dimensional**, and tuning them requires only a **coarse grid search** (e.g., step size 0.1–0.25), which is computationally inexpensive.\n- We will add recommendations for default values and include robustness plots in the supplementary material to facilitate future use.\n\nWe hope this clarifies that our framework, while adaptive to data-specific nuances, remains generalizable and practical in real-world deployment scenarios.\n\nOnce again, we sincerely appreciate the reviewer’s valuable comments and constructive questions. **If there are any further concerns or clarifications needed, we would be happy to provide additional details**."}]

Author rebuttal:

Meta-review: The paper introduces a new framework for multi-modal spatio-temporal prediction that addresses the challenges of fusing diverse data sources by using a dual-branch design that combines direct spatio-temporal modeling with a model for causal dependencies. The framework applies a combination of Graph Convolutional Networks (GCNs) and the Mamba architecture to achieve efficient encoding. The demonstrated empirical evaluation is careful. It includes four real-world datasets and comparisons against nine state-of-the-art baselines. The results consistently demonstrate clear advantages, with reported gains of up to 9.66% in accuracy and reductions in computational overhead ranging from 17% to over 56%. These improvements highlight the framework’s clear practical value in real-world predictive tasks where both accuracy and efficiency are crucial. The work makes a strong and timely contribution to the field.

The paper is very well written and the ideas are clearly and carefully presented. The empirical validation of the framework is extensive and conclusive. Three reviewers all gave accepting scores (4,4,5) and appreciated the authors careful and extensive rebuttal and were quite satisfied with their arguments. The reviewers all stated explicit support for acceptance. However, one reviewer did not appreciate the paper
(score 2), referring to a lack of technical novelty and exhibiting a general disbelief that any inference aspects of causality are not necessary for spatio-temporal predictions. As the AC of the submission, based on my reading of the paper and general understanding/experience of the challenges of making spatio-temporal predictions I do not agree with this view. Inclusions of the causal branch is a worthwhile idea that also, as nicely demonstrated in the ablation analysis of the paper, clearly demonstrates its importance by SOTA performance.
Decision: Accept (poster)