Causal Deciphering and Inpainting in Spatio-Temporal Dynamics via Diffusion Model
Authors: Yifan Duan
Jian Zhao
pengcheng
Junyuan Mao
Hao Wu
Jingyu Xu
shilong wang
Caoyuan Ma
Kai Wang
Kun Wang
Xuelong Li
Venue: NeurIPS 2024 poster (2024)
Keywords: spatio-temporal data mining; causal inference;
Primary area: interpretability_and_explainability

Abstract: Spatio-temporal (ST) prediction has garnered a De facto attention in earth sciences, such as meteorological prediction, human mobility perception. However, the scarcity of data coupled with the high expenses involved in sensor deployment results in notable data imbalances. Furthermore, models that are excessively customized and devoid of causal connections further undermine the generalizability and  interpretability. To this end, we establish a causal framework for ST predictions, termed CaPaint, which targets to identify causal regions in data and endow model with causal reasoning ability in a two-stage process. Going beyond this process, we utilize the back-door adjustment to specifically address the sub-regions identified as non-causal in the upstream phase. Specifically, we employ a novel image inpainting technique. By using a fine-tuned unconditional Diffusion Probabilistic Model (DDPM) as the generative prior, we in-fill the masks defined as environmental parts, offering the possibility of reliable extrapolation for potential data distributions. CaPaint overcomes the high complexity dilemma of optimal ST causal discovery models by reducing the data generation complexity from exponential to quasi-linear levels. Extensive experiments conducted on five real-world ST benchmarks demonstrate that integrating the CaPaint concept allows models to achieve improvements ranging from 4.3% to 77.3%. Moreover, compared to traditional mainstream ST augmenters, CaPaint underscores the potential of diffusion models in ST enhancement, offering a novel paradigm for this field. Our project is available at https://anonymous.4open.science/r/12345-DFCC.

Causal Deciphering and Inpainting in Spatio-Temporal Dynamics via Diffusion Model
Yifan Duan't Jian Zhao²-1 pengcheng's Junyuan Mao',, Hao Wu', Jingyu Xu², Shilong Wang', Caoyuan Ma², Kai Wang", Kun Wang"1, Xuelong Li University of Science and Technology of China, "TeleAI, China Telecom, "Wuhan University, National University of Singapore, 'Bejjing Forestry University, Nanyang Technological University {duanyifan28 sslong1255 229 mao wuhao2022 wstc..ccc.... edu. cn, (kevinxu, hrriinuuumaacayyyunaeeelllle @whhu. edu. cn, j edu, cn, wwwwwwi..eddiii edu. cn, wk520529xjh@gmaal com, (E0823044, zhao nus edu
Abstract
Spatio-temporal (ST) prediction has garnered De facto attention in earth sci- ences, such as meteorological prediction, human mobility perception. How- ever, the scarcity of data coupled with the high expenses involved in sensor deployment results in notable data imbalances. Furthermore, models that are excessively customized and devoid of causal connections further undermine the general izabilityy and interpretability. To this end, we establish a frame- work for ST predictions from causal perspective, termed CaPaint, which targets to identify causal regions in data and endow model with causal reasoning ability two-stage process. Going beyond this process, build on the front door adjustment as the theoretical foundation to specifically address the sub-regions identified as non non-causal in the upstream phase. By using fine-tuned unconditional Diffusion Probabilistic Model (DDPM) as the generative prior, we in-fill the masks defined as environmental parts, offering the possibility of reliable extrapolation for potential data distributions. CaPaint overcomes the high complexity dilemma optimal ST causal discovery models by reducing the data generation complex- ity from exponential to quasi-linear levels. Extensive experiments conducted on five real-world ST benchmarks demonstrate that integrating the CaPaint concept allows models to achieve improvements ranging from 3.7%~~77.3%% Moreover, compared to traditional mainstream ST augmenters, CaPaint underscores the po- tential of diffusion models in ST data augmentation, offering a novel paradigm for this field. Our project available at CaPaint.
Introduction
Deep learning methodologies have achieved groundbreaking success across wide array of spatio-
temporal (ST) dynamics systems [28, 84, 44], which include metcorological forecasting [3, 50, 50, 59, 92], wildfire spread modeling [71, 20), intelligent transportation [29, 27, 87], and human mobility systems [27, 90], to name just a few. Traditional ST dynamics approaches, based on first-principles [4, 53], often come with high computational costs. In contrast, ST dynamic analysis methods based on deep learning are not directly reliant on the explicit expression of physical laws but are data-driven [28, 84, 3, 27]. relying training models with large-scale observable datasets [86, 65, 92].
In a parallel vein, numerous efforts aim to incorporate physical laws into deep networks [35, 54, 31, 81], termed Physics- Cnfrsic-Inffon ormed Neural Networks (PINNs), which blend deep learning principles with physics to address challenges in scientific computing, particularly in fluid dynamics.
"qqual contribution
Corresponding authors
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

PINNS augment traditional neural network models by including term in the loss function that accounts for the physical laws governing fluid dynamics, such as the Navier-Stokes equations [11]. This ensures that the network's predictions are not only consistent with empirical data but also comply with the fundamental principles of fluid dynamies. However, the off-the-shelf PINNs often suffer from limited generalization capabilities, primarily due to their customized loss function designs and the neglect of specific network parameter contexts [70, 16].
To date, the data-driven deep models are still dominant in ST dynamical systems, where the numerical simulation methods and PINNS generally lag behind. The reason may stem from the rise of large models [1. 76, 28] and the high costs associated with collecting ST data from sensors [93, 39], which creates significant conflict between the increasing size of data-hungry models and the uneven, insufficient data collection. To this end, in the ST domain, there is looming research aimed at enhancing the causality and interpretability of models.
Unfortunately, research into causality within the field of ST dynamics is lagging. Although some work has considered causal design, due specific domain constraints and architectural design, can only enhance the tailor-made capabilities of the model for specific tasks [95, 38]. Moreover, causal discovery tools [12, 15] applied to ST systems often confront the "curse of dimensionality" issue during dimension reduction, despite their effectiveness in elucidating causal relationships from statistical data [75, 47]. Furthermore, NuwaDynamice cs [82] for the first time proposed decomposing causal and non-cause regions in ST sequences and enhancing the robustness and generalizability of downstream model training by generating more potential distribution ST sequences through mixup [100]. CauSTG [106] and CaST [95] address the issue of ST distribution shifts by implicitly modeling the time series embeddings and employing intervention techniques to observe these shifts.
Though promising, CauSTG [106] and CaST [95] focus on modeling graph-related data, they lack an understanding of high-dimensional observational data (Dimension D 256). NuwaDynamice es, on the other hand, explores all environments through backdoor adjustments [5]], generating a vast number of sequences, which lead nearly O(T x (²M())) training complexity (7 represents history time step, Ng and MM*) are the number of the environmental patches and mixup, respectively).
In light of this, we propose gen- eral causal structure plugin, termed CaPaint, designed to decipher causal regions in ST data without adding extra computational cost, while inter- vening in non-causal areas to boost the model's generalizability and inter- pretability. Specifically, our method employs a straightforward approach to causal deciphering, utilizing a vi- sion transformer architecture [33] for self-supervised ST data reconstruc- tion. During reconstruction, we leverage attention scores from the self-attention mechanism [23] to map onto important causal patches, thus endowing the model with interpretability. By ranking the entire set of importance scores, we define those with lower scores as environmental patches, which contribute minimally to the model. Building on this, we perform causal interventions in these environmental areas to aid the model in understanding more latent, complex, and imperceptible distributions, thereby enhancing the overall generalizability of the model (see Figure 1). Concretely, we mask trivial regions and perform generation using DDPM [24, 32] fine-tuned on specific ST data, which can also be interpreted as a ST data inpainting approach.
CaPaint Dynamical System
20:25
Observations
SOTA Exponential complexity
Quast- complexity
Vision
Transformer
Binarization
Pretrain
Senvar Deplayment
Reconstruction
Causal Discovery
Inpainting
Figure 1: Illustration of the CaPaint overview and advantage across SOTA ST causal model on complexity.
Insight. O CaPaint obeys the causal deciphering, and guided by the principle of frontdoor adjustment (2] from causal theory, CaPaint performs diffusion inpainting interventions on the environmental (non-causal) diffusion patches while reducing the temporal complexity manageable O(T x Ng)
(from OTT x NrM((=)) in [82]). CaPaint performs regional inpainting in a more natural manner, avoiding the predicament of repeatedly selecting and perturbing environmental patches. Through diffusion inpainting [42], it generates images that are more aligned with the global distribution. O
CaPaint can be understood as a ST augmenter, offering a more rational concept of ST enhancement without disrupting the inherent distribution characteristics of space and time [85]. Our major contributions can be summarized as follow:
2

---

In this paper, we introduce novel causal structure plugin, CaPaint, which leverages the concept of frontdoor adjustment from causal theory. CaPaint enables various backbone models to learn from broader distribution of data while providing enhanced interpretability for the models" predictions.
By integrating diffusion generative models with ST dynamics, CaPaint selectively perturbs non- causal regions while maintaining the integrity of core causal areas. This approach generates valuable and reliable data for scenarios where high-quality data are scarce.
We conduct extensive experiments across five diverse and representative datasets from different domains, utilizing seven backbone models to assess the effectiveness of the CaPaint method. The empirical results demonstrate that CaPaint consistently enhances performance on all tested datasets and across all backbone models (3.7%~~7..3%)
2 Related work & Technical Background
Various architectures have achieved significant predictive performance in ST domain, which can primarily be categorized as follows: CNN-based models utilize convolutional layers to effectively capture spatial features [45, 48, 77, 9]. RNN-based models, capable of processing temporal sequence data and are well-suited for understanding temporal changes, showing excellent performance in the prediction of action continuity [69, 80, 86, 72]. GNN-based models effectively capture spatial dependencies and temporal dynamics in data, making them suitable for complex tasks involving geographic locations and temporal changes [44, 25, 36, 17, 99, 98, 83, 14]. Transformer- boseed models employ self-attention mechanisms to process sequential data in parallel, enhancing the learning of long-teny long-term dependencies, and have been used for ST data prediction in complex scenarios [2, 19, 89, 91, 10, 88].
Spatio-temporal Predictive Learning:
causal discovery algorithms, originally devised for unstructured random vectors 166, 104], have progressively been adapted for ST data analysis [75, 47J. Within the extensive field of deep learning research, the study of causal inference aims to ensure
Causal inference:
stable and robust learning and reasoning paradigm. Recently, an array of techniques has been developed to delve into the nuances of causal features [60, 61, 43, 97], identifying and eliminating spurious correlations [21, 34, 56].
Generative models especially diffusion-batte model has gained significant popularity particularly in image and video generation [24, 64, 62]. Sampling optimization algorithms have been used to accelerate the sampling process of diffusion models, significantly reducing the number of steps while improving efficiency. [67, 41]. Additionally, generative models have also been applied to 3D scene generation and point cloud processing, demonstrated in [40, 30, 73, 74, 22, 63]
Image Inpainting is technique used to fill in missing or damaged parts of an image. This field can be broadly categorized into the following types. VAE-based methods: These methods leverage Variational Autoencoders to balance diversity and reconstruction [101, 103, 26]. GAN-based methods: Since the introduction of Generative Adversarial Networks, these methods have been widely used for image inpainting [55, 102, 49]. Diffusion model-based methods: Diffusion models have recently shown outstanding performance in image inpainting [46, 68.57]. 57].
Methodology
In this section, we systematically introduce causal structure plugin, CaPaint. Initially, we elucidate the methods employed in the upstream phase delineate causal and non-causal regions (Sec 3)... Subsequently, we showcase the theoretical underpinnings supporting the CaPaint (Sec 3.2). Building on this causal theory, we further engage causal intervention within observational data (Sec 3.3). Lastly, we demonstrate how sampling-enhanned enhanced ST observations can benefit the complexity of the model's ' on-device deployment (Sec 3.4).
Problem Formulation. In ST settings, We represent ST observations as a sequence {X.) each observation X, E H X xC. originates from these sequences. Our objective to predict the trajectory for the forthcoming steps, denoted {X++1)?= TTK with each future state X1+4 mapped within IR. XC.,, Here, and W indicate the spatial grid dimensions, while Cn and Cout define the input and output dimensionality of the observations, respectively.
t=1 where
3

---

H,,
Stage Cousal Patches Deciphering
Ureeee Injjection
Vision Transformett based Encoder
Covv Cow.-CCoTnnnaae Transpoopppe
GrepNerm
Observations
Input
Patchify
patches D]
Applied Data
Patch Position
Decoder for Long-
Reconstruction optimization
preprocessing
[clas]] token
Embedding
prediction
Spurious Correlation
Lowest-k Inpainting
Training
Augmented Observation with
Importance Soore
cr
P
denoising
denobiing
Freeze!
Stage 2. Causal Model Update
+
Ri
Binarycation (0.1)
Figure 2: The details of CaPaint. (Upper.) The initial phase of discovering causal patches. (Bottom.) The update phase designed to eliminate spurious correlation shifts. Following the upstream training of the ViT, a diffusion model is trained in parallel. Using the identified causal patches as conditions, this generative model then performs inpainting for generating multiple sequences.
3.1 Causal Deciphering
To find the causal (non-causal) patches with no labels, we employ a self-supereised reconstruction approach based the Vision Transformer (ViT) [13] identify key regions within ST observations. ViT segments the image into multiple patches and calculates the relationships between them using a self-attention mechanism. Due no label property, we intentionally omit the use of the [C1s] token in classification task and send data into VIT for encouraging "local-to-ge toogbball reconstruction.
Specifically, each ST data N,, divided into N = HW/p² patches, where each patch patch E R-xxpp" xCw), with (H, W) being the resolution of the original ST data and (p.p) the resolution of each patch. Subsequently, each patch is mapped to D-dimensional token through a learnable linear layer, incorporating position embedding to enhance the model's 's sensitivity to positional information. These tokens are then fed into successive L stacked transformer blocks, as described in Equation 1:
L
X' =x=X4 X MSA (LN (X))
Xous X' MLP (LN (X))
(I)
Multi-head Atteation
Residual Connection
where L.N denotes layer normalization, and MLP represents multi-layer perceptron. The upstream self- supervised reconstruction task enables the model to learn intrinsic property of ST data. Navigating the MSA mechanism [78, 96], each patch patch derived from the ST observation X, is transformed into queries q, keys k, and values v, and then calculates the relevance of each patch to others, forming weighted representation that focuses on the most informative parts. The attention weights A stored in the attention map A in each head are computed using the scaled dot-product:
Al,
At,
QK" VDW
set (Q, K. V3 Xethery A Softmax
(2)
A
A
where 1i, E x3D, are the parameter matrices, D2, represents the dimension of each head, Q. K, and V collectively denote the sets of queries q. keys k. and values v. In our approach, the determination of causal patches, is driven by an analysis of the attention maps A. Each row in an attention map is normalized and represents the importance of other patches relative to the current patch xi. However, to ascertain the overall importance of each patch across the entire input, we aggregate the contributions by summing the values along the columns of the A. To integrate insights across multiple heads, we sum these measures across all heads and then normalize the resultant vector to derive comprehensive importance score for each patch:

---

S E R
Softmax ΣΣ Ali
(3)
where S represents the normalized importance score vector, 42, E A denotes the attention that of
pays to orf for each head, H is the number of heads. We sort the importance scores in and select the patches corresponding to the lowest K scores as environmental patches storing in Oe. The remaining patches are considered causal patches Oz:
Og = Topk (I((S) x %%,, arg max (v' (xx)))
S,ES
where C(S) is the counting function, represents the proportion patches selected as causal, and V (Xx) denotes the set of patches in the ST observation X₁. We identify the causal patches by locating the indices with the highest values in S and define the non-causal parts as the environmental parts. Our goal is to perform causal interventions on the environmental parts.
3.2 Backdoor Adjustment v.s Frontdoor Adjustment
To address issues of ST data scarcity and poor transferabilitty, we examine the evaluation process using a Structural Causal Model (SCM) [52], as shown in Fig 3. We represent abstract data variables by nodes, with directed links symbolizing causality. The SCM illustrates the interaction among variables through graphical definition of causation, demonstrating the interconnected nature of these elements. As depicted in the left part, NuwaDynamic cs employs the backdoor adjustment to enhance the model's generalization performance:
Xe + X - c. The input consists of two disjoint parts Xe (causal part) and c (environ- mental or trivial part).
Ke + + + c.c. Here, Xe represents the sole endogenous parent that determines the ground truth y. However, practical scenarios, &, is also employed predicting V, which leads the formation of spurious associations.
In general, model Fo trained using Empirical Risk Minimization (ERM) often struggles to generalize to the test data Die Pte. Such distribution shifts are often induced by variations environmental patches. Hence, addressing the confounding effect caused by the environmental confounder is crucial. Backdoor adjustment techniques employed to perturb the environmental components, thereby en- hancing the model's 's potential to observe broader range of latent distributions by forcibly perturb- ing the environmental variables X,c (referred to as the do-calculus [51] operator). Unfortunately, 1 traversing all environmental variables is quite challenging. Although NuwaDynamics uses Gaussian sampling to mitigate the issue of complexity, controlling Gaussian sampling in temporal sequence operations is particularly difficult. It requires meticulous adjustment of mean and variance to ensure a balance between the number of environmen- tal samples and the training burden. @ Worse still, by traversing all environments, likely violates underlying properties, including distribution shift content and nonexistent scenarios [94]. To address this issue, we employ front-door adjustment, as illustrated in the right half of the Fig 3:
x
NuwaDynamics
X
CaPaint
X
X
X
Backdoor adjustment
y
Frontdoor adjustment
Figure 3: Different SCM architectures of SOTA and CaPaint.
Re + D + y. In this structure, 'D serves as confounder, creating misleading path between Xe and V. Here, Re represents the causal component within X
Le + X2 + y. X2 acts the surrogate variable of & and completes Xe align it with the data distribution. Initially, derives from and encompasses &c. Specifically, it envisions the potential complete observations that should exist when observing the sub-counterpart &e. Additionally, X adheres the data distribution and upholds the intrinsic knowledge of graph properties, thus eliminating any link between and Xz. Consequently, X2 is well-suited to act as the mediator, which in turn influences the model's 's predictions (+ })).
5

---

In our front-door adjustment framework, we utilize do-calculus on the variable Xe to eliminate the spurious correlations introduced by D + V. Specifically, we achieve this by summing over potential surrogate observations Xτ- This approach allows us to connect two identifiable partial effects: &e - X and &ē - y:
= =x2ldo (Kc = Xx)) P (3) (Re- =xc))= = Σ P (Diido (x= = (x== P (xt
X3
ΣΣ P p3)ftt N
Ncide = Xc) p (fxc- (xxc=Xx) Xc) P ( (²cc- - Kolao ( (xx=xxo) (5)
Xe Xe
-xxc = ΣΣ P ()) K' = ~x Xe: xxxxxxoo p 18c (²c==xx -xx p p Xxc xilăc xc)
xx)
Ne Xe
P. (Elldo (Ec (xo=Xc)) xc)) (Ke, we can feeding the surrogate observations X2 into our ST framework, conditional on the Xe,
( = Xc) holds as Xe is the only parent of X3. With data pair
- to estimate P y ~ Xc Xx Compared to previous work NuwaDynamics, CaPaint
utilizes causal regions to generate global surrogate variables in more rational manner, cireumventing the cumbersome need to traverse environmental variables inherent in backdoor adjustments. In fact, backdoor adjustments often likely violate underlying properties, leading to the generation of non-existent data distributions. The broader scenarios of CaPaint will be detailed in Appendix C.
3.3 Causal Intervention via Diffusion Inpainting
Building on the principles of causal analysis outlined above, we proceed to perform interventions on the environmental patches using diffusion inpainting, which enables us to manipulate the envi- ronmental areas. Initially, given the unique complexities of ST datasets, we fine-tune the diffusion parameters to adapt seamlessly to the domain-specifie challenges, which enhances the accuracy of our interventions on environmental patches. Diffusion models learn the distribution of data through a forward noise addition process and a reverse denoising process:
2(x, X = (Xx:; /1 3x
(X:- m.) NNXX (X),,)))
(6)
where X, represents the data state at time step f, undergoing a transformation from its previous state It-1; 3r controls the variance of the noise added at each step in the forward process, jág and Eg are neural network outputs that approximate the mean and covariance, respectively. The fine-tuning objective of the diffusion process is designed to approximate the data distribution more accurately. Specifically, the training objective for diffusion models, denoted as Eg, which predicts the noise, is typically defined as a simplified version of the variational bound:
Liipll = -i - ex,,X,, c)t
where e is the condition information. In this paper, we perform inpainting on the environmental patches of ST data. Inspired by [42], we generate a mask image for each ST data where the causal patches are black and the environmental patches are white. By independently sampling the causal and environmental patches and applying the diffusion inpainting process, we are able to generate augmented ST observation data. The detailed algorithmic process is shown in Appendix A.
3,
X t-1 VorXo + (1 -iie,
X
X,
co (X,, + o12
(8)
V Voz
VI a
X, = m C X + (1 - mm) O X
(9)
where Xeau and Xene denote causal patches and environmental patches, m is a binary mask matrix, O; represents the scaling factor at each diffusion step, determining the variance retained in the transition from X,-1 to X,. The cumulative product ā; I a represents the accumulated scaling effect from the T = 0 to step t. Equation 9 illustrates the merging of environmental patches and causal patches. Finally, the enhanced ST observation data are stored within our temporal sequence repository to bolster the downstream backbone.

---

3,4 ST Sequence Sampling Modeling
Previous work [82] assumed that the closer the time point is to the present, the greater its influence, and thus used Gaussian sampling to select more ST data closer to the current time point. However, we argue that uniform sampling can better enhance the model's generalization ability. To enhance computational efficiency while ensuring prediction accuracy, we employ a ST sequence modeling approach that samples at each time point with a fixed probability controlled by the hyperparameter p. This method allows us to sample from both original and generated data at each time point, thereby creating a new spatiotemporal sequence. We use two hyperparameters: p. which controls the sampling probability, and r, which determines the number of generated spatiotemporal sequences, achieving an optimal balance between computational efficiency and prediction accuracy. The specific sampling process can be represented by the following equation:
X p..)le(X..p.r)))
(10)
where X₂ represents the collection of original and generated data at time point and Sample(X; Xt, denotes the dataset obtained by sampling from X, with probability p. The hyperparameter p is directly set as the sampling probability, while r is used to specify the number of generated ST sequences.
Experiments
In this section, we will validate the effectiveness of our proposed causal structure plugin, CaPaint. We design four research questions (RQs) to comprehensively evaluate the performance of CaPaint:
RQ1: Does CaPaint effectively enhance model performance and applicability? RQ2: How does CaPaint perform in data-scarce scenarios? RQ3: How does the performance of CaPaint compare with other augmentation methods? RQ4: Is CaPaint effective for long-term time step predictions? Through these research questions, aim to validate the effectiveness and advantages of CaPaint in handling ST data from multiple perspectives.
4.1 Experimental settings
Datasets. We extensively evaluate our proposal using a diverse range of benchmark datasets spanning multiple fields, include FireSys [7], SEVIR [79], Diffusion reaction system (DRS) [6], KTH [58] and TaxiBJ+ [37]. Specifically, FireSys represents fire dynamics, SEVIR covers meteorological events, DRS involves physical control systems, KTH focuses on human motion dynamics, and TaxiBJ+ is transportation dataset. Detailed information can be found in the Appendix B.
Backbones and Metrics To validate the generalizability of CaPaint, select multiple model frameworks for our experiments, including the classic model like ConvLSTM [65], PredRNN- V2 [86], Vision Transformer (ViT) [13], MAU [5], the sercienccyffouuse Sim [18], and some of the latest models like MmyP [105] and Earthfarsser [92]. Our evaluation metrics include mean absolute error (MAE), mean squared error (MSE), and structural similarity index measure (SSIM). Detailed
information can be found in the Appendix D.
KTH
TaxiB +
Time Step
Sevir
SSIM
0.7
Groune truth
Cround trithh
DRS
TaxiBJ+
Simvp
PredR VN-V2
CCaPain
CaPaint
FireSys
MAU+CaP
MAU
Figure 4: Visualization of prediction results for TaxiBJ+ and SEVIR datasets. The left side shows the predicted results of the last 5 frames for TaxiBJ+. The middle presents the results of ferg-term predictions for SEVIR, displaying the last five frames from step 10 - step 20. The right side compares SSIM metrics with and without the incorporation of CaPaint.
7

---

Table 1: This table showcases the results (live runs) differences between using the CaPaint concept (+CaP) and not using it (Ori) across various datasets. All MAE and MSE values are multiplied by 100. Blue and Red backgrounds indicate the percentage improvement (reduction) in MAE and MSE, respectively.
Backbone
Metric
TaxiBJ+
KTH
SEVIR
DRS
FireSys
(10 10)
Ori
+CaP
Orl
+CaP
Orl
+CaP
Ori
+CaP
Ori
+CaP
MAE
16.59
14.54
32.03
29.52
18.69
17.56
13.59
7.52
17.32
15.97
VIT (13)
MSE
11.40
8.89
36.11
32.79
9.93
9.16
6.21
1.41
23.40
21.06
A
12.4%
12.
7.8% 1....
0.2%
6.1% 1
7.7% 1... 44.7%
77.3%
7.8%
10.1%
MAE
14.67
12.75
23.56
20.39
16.23
14.47
2.03
1.44
17.15
16.29
Earthfarsser [92)
MSE
9.94
7.83
16.84
14.07
6.75
6.01
4.09
2.24
23.37
21.91
A
12.5%
21.2%
12.6%
16.4%. T.
5.0%
10.9% 1.. 29.1%
37.8%
5.1%
6.1%
MAE
17.11
16.17
30.62
.37
20.67
17:21
15:05
11.02
19:37
18.16
Mmvpp [105]
MSB
14.22
12.29
27.31
8.45
7.26
2.32
26.09
24.97
A
7.1%
13.6%
10.0%
18. 1%
16.7%
14.1%
26.8% %
43.6%
6.2%
4.3%
MAE
18.22
16.21
22.77
20.03
20.51
18.41
5.43
3.89
22.22
10.08
ConVLSTM [65]
MSE
16.79
14.67
27.37
25.15
12.12
11.41
0.64
0.31
28,
26.44
A
13
4%
12 696
12.1%
8.1%
10.2%
5.9%
28 3%%
51.6%
9.6%
7.6%
MAE
14.18
13.05
26.73
23.65
17.94
16.20
8.76
7.98
18.26
16.14
PredRNN-V2 [86]
MSE
9.60
7.89
21:45
19.11
8.54
7.73
1.37
1.18
24.71
23.12
A
8.0%
16.6% 1... 11.6%
10.9%
9.3%
9.4% 1
8.9%
4.3%
11.6%
6.5% 1.
MAE
23.28
20.96
29.54
27.82
25.07
24.14
11.84
9.97
20.67
18.65
MAU [5]
MSF
20.46
16.00
30.19
27.84
15.43
14.34
5.28
4.66
30.89
28.91
A
10.0%
f 18.9% f 5.9%
7.8% 1
3.7% f
7.1%
15.8%
11.8%
9.8% 7
6.4% 1
MAE
15.91
13.45
23.21
20.56
15.48
14.63
2.12
1.57
17.01
15.79
SimVP [18]
MSE
10.96
8:21
16.46
13.91
6.82
6.21
9.54
5.03
23.34
22.11
A
15.4%
25.1%
11.4%
15.3%
5.5%
8.9%
25.9%
17.3%
8.1%
5.3%
4.2
Evaluating the Efficacy of CaPaint (RQ1 & RQ4)
In this section, we conduct extensive experiments to demonstrate the effectiveness of the CaPuint method. For Transformer architectures, we can directly transfer the model parameters trained in upstream tasks, thereby achieving efficient downstream training. For non- non-Transformer architectures, we focus on transferring the data itself to train the downstream models. The data presented in the Tab show the performance improvements achieved by generating only one single generalized copy for each ST sequence. As shown in the Table I, we can list the Observations:
Obs 1. +CaPaint consistently leads w/o Capaint settings across all datasets. As shown in Table 1 and the right side of Fig 4, we can easily observe that introducing +CaPaint significantly improves model performance on MAE, MSE and SSIM metrics across all datasets. For example, with the ViT model on TaxiBJ+, MAE drops from 16.59 -> 14.54, MSE from 11 11.40 -> 8.89; On Diffusion Reaction Systems, MAE significantly decreases from 13.59 > 7.52, MSE from 6.21 - 1.41. This shows CaPaint's effectiveness in boosting performance in various domains.
Obs 2. +CaPaint enhances model local insights ST scenarios. By analyzing the left side of Figure we clearly see that the +CaPaint effectively reduces the model's prediction loss. Moreover, it is observed that +CaPaint provides more accurate predictions in finer details, closely aligning with the actual result curves. This demonstrates CaPaint's 's capability to enhance prediction accuracy and reliability, ensuring that the forecasts closely mirror real-world outcomes.
Obs 3. CCaPaint remains effective in long-Term ST predictions. By analyzing the middle of Figure 4, observe that +CaPaint continues demonstrate its effectiveness in long-term time step predictions for ST tasks. For instance, the details in the SEVIR dataset predictions improve significantly, indicating that CaPaint is still applicable and beneficial in challenging ST tasks.
4.3 Performance in Data-Scarce Scenarios (RQ2)
Firesys
DRS
To assess the performance of CaPaint in data-scarce see- narios, we conducted experiments using varying propor- tions of training data across multiple datasets and back- bones. Specifically, we measured the SSIM improvement at different training data proportions, demonstrating the generalizability and robustness of CaPaint.
Cupprovement (%)
40
30
Obs 1. CaPaint shows consistent improvements across all training data proportions. As shown in Figures 5 and 6, CaPaint consistently improves SSIM across all
10-
20
30
90
70
80
100
8
Training Data Proportion (%)
Figure 5: SSIM improvement across dif-
ferent datasets using the Mmvp model

---

training data proportions. This indicates that CaPaint is ef- fective regardless of the amount of training data available, reinforcing its versatility and applicability in diverse scenarios.
Obs 2. Significant performance gains in low data scenarios. The results indicate that CaPaint yields substantial performance improvements, especially in low data scenarios. For instance, with only 10% of the training data, the SSIM improvement most pronounced, highlighting the method's effectiveness in data-scarce environments. For example, in the TaxiBJ+ dataset with ViT backbone, the SSIM improvement reaches up to more than 50%, showcasing CaPaint's capability enhance model performance with limited data.
Obs 3. Diminishing returns with increased training data. While CaPaint consistently enhances performance, the degree of improvement diminishes as the proportion of training data increases. This trend suggests that the primary benefits of CaPaint are most evident when data is scarce, but the method remains beneficial even as more data becomes available.
exmp
predrmer2
seurp
Improvement (%)
40
Imprrovemen
30
Obs 4. CaPaint demonstrates superior performance with equivalent data volumes. As illustrated in Fig 7, when comparing 25% original plus 25% augmented data with 50% original data, CaPaint achieves lower MAE and MSE. This demonstrates that CaPaint consistently outperforms the original model by effectively using mix of original and augmented data, which together match the data volume used by the original model alone.
10-
21
30
50
60
70
800
90
100
Training Data Proportion (%)
Figure 6: SSIM Improvement DRS
across various backbones
4.4 Performance Comparison (RQ3)
Table 2: Comparison between CaPaint and other data aug- mentation methods across various datasets.
g0
Datasets
Flip
Rotate
Crop
NuWa
CaPaint
DRS 2.10 2.1120.19 2.34 2.0220.00 1.57
KTH 23. 1541 as 23. 14.1411.07 23.1 as 22.32_0.es 94 20. 20.56gt100
TTabiing Dera Drppnttore
SEVIR 15.414144 15.45 15,95 15.14g1.r 14.63
Figure 7: Visualizations in both MAE and MSE with Simvp and CaP vari- ous training data proportions.
TaxiBI+ 16.4740.90 99 16.3921.22 15.94 As 15.11g0.87 20.87
FireSys 17.0212.7 17.0711.04 17.15ga 16.68 15.7921.88
In this section, compare the performance of different data augmentation methods. Tab 2 shows the model performance using various data augmentation methods across multiple datasets, measured by MAE. It can be seen that traditional data augmentation methods, such as flipping, rotation, and cropping, produce results that are either on par with or slightly worse than the original data. Take the FireSys dataset as example, MAE increased from 17.01 17.07 after rotation augmentation. This indicates that conventional data augmentation methods may disrupt the intrinsic properties of ST data, thereby negatively impacting model performance.
In contrast, our method CaPaint achieves the best performance across all datasets. For instance, on the TaxiBJ+ dataset, the MAE with CaPaint augmentation is 12.87, which is significantly better than the MAE of 15.11 with NuwaDynamics manual mixup augmentation and the MAE of 15.94 with other traditional augmentation methods such as cropping. These results highlight the advantage of our method in preserving the integrity of ST data properties. CaPaint not only effectively avoids the disruption caused by data augmentation processes on ST data characteristics but also significantly enhances the model's predictive capability.
Conclusion & Future Work
In this study, we advance the exploration of applying front-door adjustment and causality principles to spatio-temporal forecasting tasks through the introduction of CaPaint. Building upon the foundation of upstream self-supervised learning, identify causal regions crucial elements for generating

---

comprehensive and potential data distributions. By integrating diffusion generative models, we ensure the generated data's rationality and generalizability. thereby enhancing the downstream models' ability to generalize beyond the observed distribution and improving their interpretability. Moving forward, we plan to explore various generative models for the production of arbitrary-channll ST data to enhance the CaPaint robustness.
6 Acknowledgement
This work was supported by National Natural Science Foundation China (62476224).
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2.303.0m 08774, 2023.
[2] Cong Bai, Feng Sun, Jinglin Zhang, Yi Song, and Shengyong Chen. Rainformer: Features extraction balanced network for radar-based precipitation nowcasting. IEEE Geoscience and Remote Sensing Letters, 19:1-5, 2022.
[3] Kaifeng Bi, Lingxi Xie, Hengheng Zhang, Xin Chen, Xiaotao Gu, and Qi Tian. Pangu-weather A 3d high-resolution model for fast and accurate global weather forecast. arXiv preprint arXiv:2211.02556 2022.
[4] Marius Bürkle, Umesha Perera, Florian Gimbert, Hisao Nakamura, Masaaki Kawata, and Yoshihiro Asai. Deep-learning approach to first-principles transport simulations. Physical Review Letters, 126(17):177701,, 2021.
[5] Zheng Chang, Xinfeng Zhang, Shanshe Wang, Siwei Ma, Yan Ye, Xiang Xinguang, and Wen Gao. Mau: A motion-aware unit for video prediction and beyond, Advances in Neural Information Processing Systems, 34:26950-26662 2021.
[6] Boyuan Chen, Kuang Huang, Sunand Raghupathi, Ishaan Chandratreya, Qiang Du and Hod Lipson. Discovering state variables hidden in experimental data, arXiv preprint a2Xiv:21 10755, 2021.
[7] Boyuan Chen, Kuang Huang, Sunand Raghupathi, Ishaan Chandratreya, Qiang Du, and Hod Lipson. Automated discovery of fundamental variables hidden in experimental data. Nature Computational Science, 2(7):433 442, 2022.
[8] Yuyao Chen and Luca Dal Negro. retrieval of photonic nanostructures from near-field data. APL Photonics, 7(1), 2022.
neural networks for imaging and parameter
[9] Jinguo Cheng, Ke Li, Yuxuan Liang, Lijun Sun, Junchi Yan, and Yuankai Wu. Rethinking urban mobility prediction: A sulti-multivariate time series forecasting approach. arXiv preprint arXiv:2312.01699, 2023.
[10] Jinguo Cheng, Chunwei Yang, Wanlin Cai, Yuxuan Liang, and Yuankai Wu. Nuwats: Mending every incomplete time series. arXiv preprint a3X3: 2024.
[11] Peter Constantin and Ciprian Foias. Navier-stokes equations. University of Chicago press, 1988.
[12] Giorgia Di Capua, Jakob Runge, Reik V Donner, Bart van den Hurk, Andrew G Turner, Ramesh Vellore. Raghavan Krishnan, and Dim Coumou. Dominant patterns of interaction between the tropics and mid-latitudes in boreal summer: Causal relationships and the role of time-scales. Weather and Climate Dynamics Discussions, 1-20:1-28, 2020.
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale, arXiv preprint arXiv: arXiv: 1199,, 2020.
10

---

[14] Yifan Duan, Guibin Zhang, Shilong Wang, Xiaojiang Peng, Wang Ziqi, Junyuan Mao, Hao Wu, Xinke Jiang, and Kun Wang. Cat-gnn: Enhancing credit card fraud detection via causal temporal graph neural networks. arXiv preprint arXiv:2402. 14708, 2024.
[15] Imme Ebert-Uphoff and Yi Deng. Causal discovery from spatio-temporal data with applica- tions to climate science. In 2014 13th International Conference on Machine Learning and Applications, pages 606-613. IEEE, 2014.
[16] Stathi Fotiadis, Mario Lino Valencia, Shunlong Hu, Stef Garasto, Chris D Cantwell, and Anil Anthony Bharath. Disentangled generative models for robust prediction of system dynamics. 2023.
[17] Xiaowei Gao, Xinke Jiang, Dingyi Zhuang, Huanfa Chen, Shenhao Wang, Stephen Law, and James Haworth. Uncertainty- probabilistic graph neural networks for road-level traffic accident prediction. 2023, URL https: //api./ppppppp semanticscholar, arg/CorrpuIDD 261681823.
[18] Zhangyang Gao, Cheng Tan, Lirong Wu, and Stan Z Li. Simvp: Simpler yet better video prediction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3170-3180, 2022.
[19] Zhihan Gao, Xingjian Shi, Hao Wang, Yi Zhu, Yuyang Bernie Wang, Mu Li, and Dit- Yeung. Earthformer: Exploring space-time transformers for earth system forecasting, Advances in Neural Information Processing Systems, 35:25390-25403, 2022.
[20] Sebastian Gerard, Yu Zhao, and Josephine Sullivan. Wildfirespreadts: A dataset of multi-modal time series for wildfire spread prediction. Advances in Neural Information Processing Systems, 36, 2024.
[21] Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. arXiv preprint arXiv:2007. 0.144, 2020.
[22] Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Yiwen Tang. Xianzheng Ma. Jiaming Han, Kexin Chen, Peng Gao, Xianzhi Li, Hongsheng Li, et al. Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following. arXiv preprint
2023.
[23] Kai Han, Yunhe Wang. Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al. A survey vision transformer. IEEE transactions on pattern analysis and machine intelligence, 45(1):87-110,, 2022.
[24] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6855 2020.
[25] Xinke Jiang, Dingyi Zhuang, Xianghui Zhang, Hao Chen, Jiayuan Luo, and Xiaowei Gao. Uncertainty quantification via spatial-temporal tweedie model for zero-inflated and long- tail travel demand prediction. Proceedings of the 32nd ACM International Conference Information and Knowledge Management, 2023. URL https: / api. comanticscholas org/ Corpus 259187717.
[26] Xinke Jiang, Zidi Qin, Jiarong Xu, and Xiang Ao. Incomplete graph learning via attribute- structure decoupled variational auto-encoder. Proceedings of the 17th ACM International Conference on Web Search and Data Mining, 2024. URL //cttss://aae http:p////ppp.ssnanicccohoooooot semanticscholar org/ 20:opppuID::28394444000
[27] Guangyin Jin, Yuxuan Liang, Yuchen Fang, Zezhi Shao, Jincai Huang, Junbo Zhang, and Yu Zheng. Spatio-temporal graph neural networks for predictive learning in urban computing: A survey. IEEE Transactions on Knowledge and Data Engineering. 2023.
[28] Ming Jin, Qingsong Wen, Yuxuan Liang, Chaoli Zhang, Siqiao Xue, Xue Wang, James Zhang, Wang, Haifeng Chen, Xiaoli Li, et al. Large models for time series and spatio-temporal data: A survey and outlook. arXiv preprint arXtv:2310.10196 2023.
11

---

[29] Sepideh Kaffash, An Truong Nguyen, and Joe Zhu. Big data algorithms and applications in intelligent transportation system: A review and bibliometric analysis. International journal of production economics, 231:107868, 2021.
[30] Animesh Karnewar, Andrea Vedaldi, David Novotny, and Niloy Mitra. Holodiffusion: Training 3d diffusion model using 2d images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18423-18433, 2023.
[31] George Em Karniadakis, Ioannis G Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang. Physies-informed machine learning. Nature Reviews Physies, 3(6):422-44)) 440, 2021.
[32] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401-4410 2019.
[33] Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey. ACM computing surveys (CSUR), 54 (10s):1-41, 2022.
[34] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, al. Wilds: A benchmark of in-the-wild distribution shifts. In International conference on machine learning, pages 5637-5664. PMLR, 2021.
[35] Aditi Krishnapriyan, Amir Gholami, Shandian Zhe, Robert Kirby, and Michael W Mahoney. Characterizing possible failure modes in physics-informer neural networks. Advances in Neural Information Processing Systems, 34:26548-26335 2021.
[36] Rongfang Li, Ting Zhong, Xinke Jiang, Goce Trajcevski, Jin Wu, and Fan Zhou. Mining spatio-temporal relations via self self-paced graph contrastive learning. Proceedings of the 28th ACM SIGKDD Conference Knowledge Discovery and Data Mining, 2022. URL https: .ocilaarrrrrgggeeeeeeeeee org/ htg/CorpuusDD. 251518220.
[37] Yuxuan Liang, Kun Ouyang, Junkai Sun, Yiwei Wang, Junbo Zhang, Yu Zheng, David Rosenblum, and Roger Zimmermann. Fine-grained urban flow prediction. In Proceedings of the Web Conference 2021, pages 1833-1845 1845, 2021.
[38] Wei Liu, Yu Zheng, Sanjay Chawla, Jing Yuan, and Xie Xing. Discovering spatio-temporal causal interactions in traffic data streams. In Proceedings of the 17th ACM SIGKDD interna- tional conference on Knowledge discovery and data mining, pages 1010-1018, 2011.
[39] Xu Liu, Yutong Xia, Yuxuan Liang, Junfeng Hu, Yiwei Wang, Lei Bai, Chao Huang. Zhen- guang Liu, Bryan Hooi, and Roger Zimmermann. Largest: A benchmark dataset for large-scale traffic forecasting, Advances in Neural Information Processing Systems, 36, 2024.
[40] Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Marc Habermann, Christian Theobalt, al. Wonder3d: Single image to 3d using cross- domain diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 9970-9990 2024.
[41] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm- solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095,, 2022.
[42] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. Proceedings of the IEEF/CVF conference computer vision and patlern recognition, pages 1461-11177,
2022.
[43] Dongsheng Luo, Wei Cheng, Dongkuan Xu, Wenchao Yu, Bo Zong, Haifeng Chen, and Xiang Zhang. Parameterized explainer for graph neural network. Advances neural information processing systems, 33:19620-19631 2020.
12

---

[44] Jiayuan Luo, Wentao Zhang, Yuchen Fang. Xiaowei Gao, Dingyi Zhuang, Hao Chen, and Xinke Jiang. Timeseries suppliers allocation risk optimization via deep black litterman model. ArXiv, 0/a/2401.173500 2024. URL https:/ //apii semanticss semanticscholar org Crrp/orrpssI:: 271854629.
[45] Michael Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error. arXiv preprint arXiv: 1et:::111::04440 1.05440, 2015.
[46] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mc- Grew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:21 /2. 10741, 2021.
[47] Peer Nowack, Jakob Runge, Veronika Eyring, and Joanna Haigh. Causal networks for climate model evaluation and constrained projections. Nature communications, 11(1):1415, 2020.
[48] Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard I. Lewis, and Satinder Singh. Action- conditional video prediction using deep networks in atari games. Advances neural informa- tion processing systems, 28, 2015.
[49] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2536 2544,, 2016.
[50] Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopad- hyay, Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Azizzadenesheli, et al. Fourcastnet: A global data-driven high-resolution weather model using adaptive fourier neural operators. arXiv preprint arXiv:2202. 12214 2022.
[51] Judea Pearl. Causality. Cambridge university press, 2009.
[52] Judea Pearl and Dana Mackenzie. The book of why: the new science of cause and effect. Basic books, 2018.
[53] Roger W Pryor. Multiphysics modeling using COMSOLØ: a first principles approach. Jones & Bartlett Publishers, 2009.
[54] Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics- moysice-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686-7011 2019.
[55] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or. Encoding in style: a stylegan encoder for image-to-imane translation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2287-2296, 2021.
[56] Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv: 191::111110. 1.08311,, 2019.
[57] Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 conference proceedings, pages 1-10, 2022.
[58] Christian Schuldt, Ivan Laptev, and Barbara Caputo. Recognizing human actions: a local svm approach. In Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004., volume 3, pages 32-36. IEEE, 2004.
[59] Martin Schultz, Clara Betancourt, Bing Gong, Felix Kleinert, Michael Langguth, Lukas Hu- bert Leufen, Amirpasha Mozaffari, and Scarlet Stadtler. Can deep learning beat numerical weather prediction? Philosophical Transactions of the Royal Society A, (799(119)):2220097,, 2021.
13

---

[60] Ramprasaath R Selvaraju, Abhishek Das. Ramakrishna Vedantam, Michael Cogswell. Devi Parikh, and Dhruv Batra. Grad-cam: Why did you say that? arXiv preprint 111.4:0611.07400 2016.
[61] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient- based localization. In Proceedings of the IEEE international conference on computer vision, pages 618-626, 2017.
[62] Fei Shen, Hu Ye, Jun Zhang, Cong Wang, Xiao Han, and Yang Wei. Advancing pose-guided image synthesis with progressive conditional diffusion models. In The Twelfth International Conference on Learning Representations, 2023.
[63] Fei Shen, Xin Jiang, Xin He, Hu Ye, Cong Wang, Xiaoyu Du, Zechao Li, and Jinghui Tang. Imagdressing-vl: Customizable virtual dressing. arXiv preprint urXiv:2407.12705, 2024.
[64] Fei Shen, Hu Ye, Sibo Liu, Jun Zhang, Cong Wang, Xiao Han, and Wei Yang. Boosting consistency in story visualization with rich-contextual conditional diffusion models. arXiv preprint arXiv:2407.00 242.022222 2024.
[65] Xingjian Shi, Zhourong Chen, Hao Wang, Din-Yan Yeung, Wai-Kin Wong, and Wang-chun Woo. Convolutional Istm network: A machine learning approach for precipitation nowcasting. Advances in neural information processing systems, 28, 2015.
[66] Shohei Shimizu, Patrik O Hoyer, Aapo Hyvärinen, Antti Kerminen, and Michael Jordan. A linear non-gaussian acyclic model for causal discovery. Journal of Machine Learning Research, 7(10), 2006.
[67] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint (ixi::2000002502. 2020.
[68] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13450 2020.
[69] Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video representations using Istms. In International conference on machine learning, pages 843 852. PMLR, 2015.
[70] Makoto Takamoto, Francesco Alesiani, and Mathias Niepert. Cape: Channel-attention-baass pde parameter embeddings for sciml. 2022.
[71] Wai Cheong Tam, Eugene Yujun Fu, Jiajia Li, Xinyan Huang, Jian Chen, and Michael Xuelin Huang. A spatial temporal graph neural network model for predicting flashover in arbitrary building floorplans. Engineering Applications of Artificial Intelligence, 115:105258, 2022.
[72] Cheng Tan, Zhangyang Gao, Lirong Wu, Yongjie Xu, Jun Xia, Siyuan Li, and Stan Z Li. Temporal attention unit: Towards efficient spatiotemporal predictive learning. Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pages 18770- 18782, 2023.
[73] Yiwen Tang, Jiaming Liu, Dong Wang. Zhigang Wang, Shanghang Zhang, Bin Zhao, and Xuelong Li. Any2point: Empowering any-modality large models for efficient 3d understanding. arXiv preprint arXiv:2404.07900 2024.
[74] Yiwen Tang, Ray Zhang, Zoey Guo, Xianzheng Ma, Bin Zhao, Zhigang Wang, Dong Wang, and Xuelong Li. Point-peft: Parameter-efficient fine-tuning for 3d pre-trained models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 5171-5179, 2024.
[75] Andoer-Andoni Tibau, Christian Reimers, Andreas Gerhardus, Joachim Denzler, Veronika Eyring, and Jakob Runge. A spatiotemporal stochastic climate model for benchmarking causal discovery methods for teleconnections. Environmental Data Science, I:e12, 2022.
14

---

[76] Hugo Touvron, Thibaut Lavril, Gautier Izacard. Xavier Martinet, Marie- Anne Lachaux, Timo- thée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
[77] Sergey Tulyakov, Ming- Yu Liu, Xiaodong Yang. and Jan Kautz. Mocogan: Decomposing motion and content for video generation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1526-1535, 2018.
[78] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
[79] Mark Veillette, Siddharth Samsi, and Chris Mattioli, Sevir: storm event imagery dataset for deep learning applications in radar and satellite meteorology. Advances in Neural Information Processing Systems, 33:22009- 2020.
[80] Ruben Villegas, Dumitru Erhan, Honglak Lee, et al. Hierarchical long-term video prediction without supervision. International Conference on Machine Learning, pages 6038-6044 PMLR, 2018.
[81] Hanchen Wang, Tianfan Fu, Yuanqi Du, Wenhao Gao, Kexin Huang, Ziming Liu, Payal Chandak, Shengchao Liu, Peter Van Katwyk, Andreea Deac, et al. Scientific discovery in the age of artificial intelligence. Nature, 620(7972):47-60, 2023.
[82] Kun Wang, Hao Wu, Yifan Duan, Guibin Zhang, Kai Wang, Xiaojiang Peng, Yu Zheng, Yuxuan Liang, and Yang Wang. Nuwadynamics: Discovering and updating causal spatio-tempon modeling.
[83] Kun Wang, Guibin Zhang, Xinnan Zhang, Junfeng Fang, Xun Wu, Guohao Li, Shirui Pan, Wei Huang, and Yuxuan Liang. The heterophilic snowflake hypothesis: Training and empowering gnns for heterophilic graphs. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 3164-3175, 2024.
[84] Senzhang Wang, Jiannong Cao, and S Yu Philip. Deep learning for spatio-temporal data mining: A survey, IEEE transactions knowledge and data engineering, 34(8):3681-3700) 2020.
[85] Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni. Generalizing from a few examples: A survey on few-shot learning, ACM computing surveys (csur), 53(3):1334, 2020.
[86] Yunbo Wang, Haixu Wu, Jianjin Zhang, Zhifeng Gao, Jianmin Wang. S Yu Philip, and Mingsheng Long. Predran: A recurrent neural network for spatiotemporal predictive learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(2):2208-231 2022.
[87] Zepu Wang, Peng Sun, Yulin Hu, and Azzedine Boukerche, Sft: A high-precision traffic flow predictor for supporting intelligent transportation systems. In GLOBECOM 2022-2022 IEEE Globul Communications Conference, pages 251-256. IEEE, 2022.
[88] Zepu Wang, Yifei Sun, Zhiyu Lei, Xincheng Zhu, and Peng Sun. Sst: A simplified swin transformer-based model for taxi destination prediction based on existing trajectory. In 2023 IEEE 26th International Conference on Intelligent Transportation Systems (ITSC), pages 1404-1409. IEEE, 2023.
[89] Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma, Junchi Yan, and Liang Sun. Transformers in time series: A survey, arXiv preprint arXiv:2202.07125, 2022.
[90] Hao Wu, Kun Wang, Fan Xu, Yue Li, Xu Wang, Weiyan Wang, Haixin Wang, and Xiao Luo. Spatio-temporal twins with cache for modeling long-term system dynamics. 2023.
[91] Hao Wu, Wei Xion, Fan Xu, Xiao Luo, Chong Chen, Xian-Sheng Hua, and Haixin Wang. Pastnet: Introducing physical inductive biases for spatio-temporal video prediction. arXiv preprint arXiv:2305.11 2023.
15

---

[92] Hao Wu, Yuxuan Liang, Wei Xiong. Zhengyang Zhou, Wei Huang, Shilong Wang, and Kun Wang. Earthfarsser: Versatile spatio-temporal dynamical systems modeling in one model. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 15906-15914, 2024.
[93] Hao Wu, Haomin Wen, Guibin Zhang, Yutong Xia, Kai Wang, Yuxuan Liang, Yu Zheng, and Kun Wang. Dynst: Dynamic sparse training for resource-constrained spatio-temporal forecasting, arXiv preprint MxXiv:2403.02914, 2024.
[94] Ying-Xin Wu, Xiang Wang, An Zhang, Xia Hu, Fuli Feng, Xiangnan He, and Tat-Seng Chua. Deconfounding to explanation evaluation in graph neural networks. arXiv preprini arXiv:2201. 2000:2, 2022.
[95] Yutong Xia, Yuxuan Liang, Haomin Wen, Xu Liu, Kun Wang, Zhengyang Zhou, and Roger Zimmermann. Deciphering spatio-temporal graph forecasting: A causal lens and treatment. Advances in Neural Information Processing Systems, 36, 2024.
[96] Songhua Yang, Xinke Jiang, Hanjie Zhao, Wenxuan Zeng, Hongde Liu, and Yuxiang Jia. Faima: Feature-aware in-context learning for multi-domain aspect-based sentiment analy- sis. ArXiv, abs/2403.01063, 2024. URL https: //ppi. semanticscholar org. ID: 268230305.
[97] Zhitao Ying, Dylan Bourgeois, Jiaxuan You, Marinka Zitnik, and Jure Leskovec. Gnnexplainer: Generating explanations for graph neural networks, Advances in neural information processing systems, 32, 2019.
[98] Guibin Zhang, Kun Wang. Wei Huang, Yanwei Yue, Yang Wang, Roger Zimmermann, Aojun Zhou, Dawei Cheng, Jin Zeng, and Yuxuan Liang. Graph lottery ticket automated. In The Twelfth International Conference on Learning Representations, 2024.
[99] Guibin Zhang, Yanwei Yue, Kun Wang, Junfeng Fang, Yongduo Sui, Kai Wang, Yuxuan Liang, Dawei Cheng. Shirui Pan, and Tianlong Chen. Two heads are better than one: Boosting graph sparse training via semantic and topological awareness, arXiv preprint arX1v:2402.01222. 2024.
[100] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:/710.00 171000941. 2017.
[101] Lei Zhao, Qihang Mo, Sihuan Lin, Zhizhong Wang. Zhiwen Zuo, Haibo Chen, Wei Xing, and Dongming Lu. Uctgan: Diverse image inpainting based on unsupervised cross-space translation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5741-5750, 2020.
[102] Shengyu Zhao, Jonathan Cui, Yilun Sheng, Yue Dong, Xiao Liang, Eric I Chang, and Yan Xu. Large scale image completion via co-modulated generative adversarial networks. arXiv preprint arXiv:2 10428, 2021.
[103] Chuanxia Zheng, Tat-Jen Cham, and Jianfei Cai. Pluralistic image completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2019.
[104] Xun Zheng, Bryon Aragam, Pradeep K Ravikumar, and Eric P Xing. Dags with no tears: Continuous optimization for structure learning, Advances in neural information processing systems, 31, 2018.
[105] Yiqi Zhong, Luming Liang, Ilya Zharkov, and Ulrich Neumann. Mnvp: Motion-matte video prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4273 4283, 2023.
4283,
[106] Zhengyang Zhou, Qihe Huang, Kuo Yang, Kun Wang, Xu Wang, Yudong Zhang, Yuxuan Liang, and Yang Wang. Maintaining the status quo: Capturing invariant relations for ood spatiotemporal learning. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1603-3614, 2023.
16

---

A CaPaint Inpainting Algorithm
Algorithm 1 Causal Intervention with Diffusion Inpainting
Input: ST observation data X, masked image Xmask
2: Output: Augmentation ST observation dataset Xa
3: Initialize XT ~ (0, 7) where 7' is the total number of diffusion steps
4: 1* Iterate backwards through diffusion steps */
5: for t = =T to do
6: 1 Sample Gaussian noise */
7:
E~ 10,
8:
1" Sample causal region
9:
X = Vā,.Xo + (1 oz)c
10:
A Sample Gaussian noise N */
~~N(0, T)
12:
1t Causal Intervention on Environmental Patches */
13:
& X voe Xx 51-a, to Xx, t) o+z)
14:
1s Combine causal and environmental patches */
15:
X,-1 2 X + (1 m) @ X
16:
X,
end for
18:
return XA the augmentation dataset
The algorithm for Causal Intervention with Diffusion Inpainting aims to augment ST observation data through series of diffusion steps that iteratively refine the data by applying causal interventions and combining them with environmental patches. Here is a detailed step-by-ster description:
Input: The original ST observation data X, and masked image Xmask.
Output: An augmented ST observation dataset XA-
The process begins by initializing Xy. which represents the data the final diffusion step, to be a sample from normal distribution centered at zero with identity covariance.
The main loop of the algorithm runs backward from the last diffusion step T to the first. In each step:
1. Gaussian noise e, is sampled to simulate the diffusion process.
causal region Xem is sampled where the causal effect calculated as blend of the original data and the Gaussian noise, emphasizing areas of interest that should retain more original data characteristics.
Gaussian noise N₂ is sampled again, providing variability to the non-causal or environ- mental regions.
The environmental patches Xam are updated using the data from the previous step adjusted by damping factor and the added noise, simulating environmental changes. then combined. where the mask M de-
5. The causal and environmental patches termines the specific locations for the causal and environmental updates in the data, specifying which parts from the causal region and which are from the environmental region.
6. The data for the next step, Xt-1, is computed by normalizing the combined updates, preparing for the next iteration or output if is the first step.
Finally, the algorithm outputs the augmented data set XA, which is the result of the iterative causal intervention and environmental blending over the diffusion process.
B Details of experiments
SSIM stands for Structural Similarity Index Measure, which is method for measuring the similarity between two images. It compares the structural information of the images, including luminance,
17

---

contrast, and texture, to determine how similar they are. SSIM is commonly used in image and video processing applications, such as image compression and quality assessment.
PSNR stands for Peak Signal-to-Noiss Ratio. It a measure of video or image quality that compares the original signal to the compressed or transmitted signal. The higher the PSNR value, the better the quality of the compressed or transmitted signal. PSNR is commonly used in video and image compression applications to evaluate the effectiveness of compression algorithms.
MSE (Mean Squared Error) loss is a commonly used loss function machine learning and deep learning models. This loss function calculates the average of the squared differences between the predicted and actual values.
Datasets, Here we summarize the details (Tab. 1) of the datasets used in this paper:
TaxiBJ+: This dataset contains trajectory data obtained from the GPS of taxis in Beijing, divided into two separate channels: inflow and outflow. Additionally, the dataset has been extended from 32x32 to 128x128 28 by collecting recent trajectory data from Beijing.
KTH: This dataset includes 25 individuals performing six different actions: walking, jogging, running, boxing, waving, and clapping. The complexity of human movements arises from the unique variations each individual displays while executing these actions. By examining previous frames, the model can understand the subtleties of human dynamics and predict future extended postural changes.
SEVIR: This dataset consists of weather images that have been sampled and aligned using radar and satellite data. It is designed as a foundational resource to support algorithm development in meteorological research.
This dataset describes the diffusion process of nonlinear wave, which satisfies the diffusion equation.
DRS:
FireSys: The FireSys dataset comprises data associated with fire observations, capturing both temporal and spatial trends of fire evolution, which faithfully represent the progression status in natural setting.
C Broader Impact
The development and application of the CaPaint framework in spatio-temporal (ST) dynamics bring several positive broader impacts. Understanding these impacts is crucial for responsible Al research and deployment.
Data Imputation in Sparse Scenarios: CaPaint excels in sparse data scenarios, effectively filling in missing data. This reduces the need for extensive sensor deployments, significantly lowering the cost associated with sensor installation. By optimizing data coverage and utilization, CaPaint not only enhances resource efficiency but also achieves substantial cost savings.
Enhanced Predictive Accuracy and Interpretability: CaPaint can identify and intervene in non non-causal regions, improving the predictive accuracy and interpretability in various ST domains such as meteorology, human mobility, and disaster management. This improvement leads to better decision-making processes and resource allocation, ultimately benefiting society by providing more reliable and understandable predictive models.
3. Cost-Effective Solutions:
By reducing the complexity of optimal ST causal discovery models, CaPaint offers a cost-effective solution for handling high-dimensional ST data. This makes advanced predictive technologies more accessible across a broader range of applications, particularly in fields with limited computational resources.
Promotion of Causal Reasoning in AI: The integration of causal reasoning into ST models encourages the development of AI systems that better mimic human understanding of cause-and- effect relationships. This can lead to more robust Al models capable of generalizing across different scenarios, fostering trust and reliability in Al applications.
Innovation Data Augmentation Techniques: CaPaint introduces novel data augmentation methods using diffusion inpainting, which can inspire further research and innovation in data augmen- tation and ST prediction. This can lead the emergence of new techniques, enhancing the robustness and performance of Al models in various domains.
18

---

The CaPaint framework represents significant udvancement in the field of ST dynamics, particularly in its ability to address sparse data scenarios, which reduces the need for extensive sensor deployments and lowers associated costs. Additionally, CaPaint enhances predictive accuracy. interpretability. and efficiency, promotes causal reasoning in AI, and introduces innovative data augmentation techniques. Responsible Al research and deployment should leverage these strengths to maximize benefits while minimizing risks.
D Metrics
In our research, we investigate the performance of our models using Mean Squared Error (MSE), Mean Absolute Error (MAE), and Structural Similarity Index Measure (SSIM). The formulas for evaluating these indicators, converted into decibels (dB) where applicable, are as follows:
Mean Squared Error (MSE)
Mean Squared Error (MSE) measures the average of the squares of the errors, that is, the average squared difference between the estimated values and the actual value. The MSE is given by:
N Ox, Y.)²
MSE
(D.1)
1=1
where Y; is the actual value, Y; is the predicted value, and N is the number of observations.
Mean Absolute Error (MAE)
Mean Absolute Error (MAE) measures the average magnitude of the errors in set of predictions, without considering their direction. It is the average over the test sample of the absolute differences between prediction and actual observation where all individual differences have equal weight. The MAE is given by:
N
MAE
1
Σ
Y;
&
(D.2)
where Y; is the actual value, Y, is the predicted value, and N the number of observations.
Structural Similarity Index Measure (SSIM)
Structural Similarity Index Measure (SSIM) is used for measuring the similarity between two images. The SSIM index is decimal value between -1 and 1, where is only reachable in the case of two identical sets of data. The SSIM formula can be quite complex due its consideration of luminance, contrast, and structure comparison functions between the two images:
(2paly + C1) C2)
SSIM (x. v) (n2 + 1 + Ci) + +
(D.3)
where Paz, By the average of and respectively, of of are the variance of and respectively, azy the covariance of x and y, and Ci, C2 are variables to stabilize the division with weak denominator.
E
Limitations
While the implementation of the CaPaint method has demonstrated significant improvements in prediction accuracy and detail preservation in spatio-temporal forecasting tasks, its enhancements are most pronounced scenarios characterized by data scarcity or uneven data distribution. contexts where datasets are abundant and exhibit broad and uniform distribution, the incremental gains offered by CaPaint may not be as substantial. Nevertheless, the method remains effective, providing consistent, albeit smaller, improvements across diverse data environments.
19

---

ST Observation
non-causal (environmental) patch
Causal Deciphering
Causal Intervention
Figure F.1: Inpainting Example of proposed CaPaint.
An example of ST Inpainting on SEVIR
The figure illustrates the process of maintaining causal regions intact while performing inpainting on non-causal (environmental) regions. The approach involves identifying and deciphering the causal regions (left), intervening by applying diffusion inpainting on the environmental patches (middle), and subsequently generating altered ST data copies (right). This method ensures that the intrinsic causal relationships within the data are preserved, while variations are introduced in the environmental context to augment the dataset effectively.
G Uneven Distribution of Sensors Leading to Data Scarcity in Global Oceanic Observation
Systom usage
47 Bngicteed plafares meet
76484 Regrowed plaffors Deeris
33 Askrage Willviis Denin
21% WAGSS Sanchen Wpp.epeeooolll.o pnlky
1393 Aanrage weebly Drantes
2017
+
900
-00
Dane
Arge
CSCK
DLOS5
Dusunnti soas
Figure G.I: Temporal distributional heterogeneity within the global oceanic observation platforms, which reveals that there are pronounced disparities in the deployment numbers of various types of sensors during different time intervals.
20

---

H
Experimental Parameters
In this experiment, we employ different deep learning models and optimize them for training. All experiments are conducted on hardware equipped with 24 NVIDIA GeForce RTX 4090 GPUs. The optimizer used is Adam, and different learning rates (LR) and batch sizes are for each model. The specific parameter settings are shown in the table below:
Model
Learning Rate (LR)
Batch Size
CLSTM
0.001
MAU
0.001
8
MMVP
0.004
4
PredRNNv2
0.001
8
SimVP
0.004
4
ViT
0.004
4
Earthfarsser
0.001
8
Table H.T: Learning rates and batch sizes for different backbones
These parameter settings are chosen based on the characteristics of each model and preliminary experimental results on the validation set, aiming optimize the training efficiency and performance of the models. The Adam optimizer is used with OneCycle learning rate scheduler, where the maximum learning rate is according the specified learning rate for each model, and the number of steps per epoch and the total number of epochs are set based on the training data and experimental setup. During the experiments, we ensure that all models are trained under the same hardware conditions to guarantee the comparability and reproducibility of the results.
I Visualizations on KTH
Ground truth
Earthfarser
+CaPaint
Figure I.1:
Visualizations on KTH dataset showing the last 5 frames
The first row shows the ground truth for walking individual. The second row, processed by Earthfarser, exhibits noticeable blurring and loss of detail. The third row, enhanced with +CaPaint, demonstrates a marked improvement capturing fine details such as the shadow the person and the accuracy of the foot motion, as highlighted in the red boxes.
J Visualizations on Diffusion Reaction System
The introduction of CaPaint has led to reductions in Mean Squared Error MSE and MAE, while the SSIM has shown improvements. These changes indicate that the CaPaint method effectively enhances model prediction accuracy and image quality. However, due to the high quality of the model predictions, the improvements might not be readily observable to the naked eye. Despite this, the
21

---

Ground truth
6
Farthfarsser
6
cCaPaint
Figure J.1: Visualizations on DRS dataset showing 10 frames
positive effects of CaPaint are clearly evident through quantitative metrics, demonstrating its potential and practicality in enhancing the accuracy of complex dynamic systems predictions.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: this paper, we introduce the spatio-temporal causal concept in the data mining realm, aimed at enhancing the reliability and accuracy of financial spatio-temporal prediction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: this work, we systematically discuss the limitations of our research and outline directions for future work.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
23

---

Answer:
[NA]
Justification:
This paper does not include experimental results related to theoretical aspects.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide the code necessary for replicating the studies described in this paper via an anonymous link, and we detail the experimental setup for the replication in the article itself.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
24

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: For the datasets disclosed in the article, we have provided information regarding their sources and origins.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/ wwtp:///ooeeeeeaeeiiiiiiiiaaaaee
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / /nips. cctpublic/eeeeccccccccccccccecct habbic/ggiids///// /Coo//ooeeemeeeee cetsSuumiiiioni. swiiisssisiogoooooooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: we have specified all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: In this paper, we have reported error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
25

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: In this paper, we provide detailed information about the experimental resources, including GPU configurations used in our studies.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https /:t::///neerppe httpp///cccccccccccccceeeeeeeee.
Answer: [Yes]
Justification:
The study presented in this paper conforms to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have provided the societal impacts of the work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
26

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
This paper does not address issues related to this aspect.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
All creators and original owners of the assets used in our paper, such as code, data, and models, have been properly credited. We have explicitly mentioned the licenses and terms of use for each asset and have ensured full compliance with these terms throughout research.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
27

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The research presented in this paper is not concerned with new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve experiments or research related to human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not address potential risks incurred by study participants.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
28

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "Rbm8eLcUX8", "reviewer": "Reviewer_botx", "summary": "This paper introduces a groundbreaking framework named CaPaint, which is designed to tackle the critical issues of data scarcity and the absence of causal connections in spatiotemporal (ST) prediction models. The authors have established a robust causal framework that not only identifies regions within data that exhibit causal relationships but also endows the model with the capability to reason about causality during a two-stage processing procedure. In the initial stage, they leverage self-supervised Vision Transformer (ViT) reconstruction to identify the crucial causal patches within ST observations. This is followed by an intervention phase where they employ diffusion inpainting techniques to manipulate non-causal areas while preserving the integrity of core causal areas. The innovative method reduces the complexity of generating data from exponential levels to quasi-linear levels, thereby significantly enhancing efficiency. Moreover, it has shown remarkable improvements across various ST benchmarks by integrating diffusion models as a novel data augmentation technique, marking a paradigm shift for this field.", "questions": "Mentioned in the weakness section.", "limitations": "I think the author addressed the limitation mentioned in this work.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Addresses data scarcity and lack of causal connections in ST prediction models effectively.\n- Novel Method: Innovative use of self-supervised Vision Transformer reconstruction for causal patch identification. And employs diffusion inpainting techniques to manipulate non-causal areas, preserving core causal integrity.\n- Demonstrates significant improvements across various ST benchmarks, integrating diffusion models as a novel data augmentation technique. Besides, it reduces data generation complexity from exponential to a quasi-linear level.", "weaknesses": "- Details on computational efficiency or scalability of the proposed method are not provided, leaving it as a potential limitation for practical applications.\n- More visualization of the prediction results should be included even in supplementary material.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "- Details on computational efficiency or scalability of the proposed method are not provided, leaving it as a potential limitation for practical applications.\n\nThank you for your valuable feedback. We appreciate your concern regarding the computational efficiency and scalability of our proposed method. \n\n**1:** We leverage the attention mechanism for causal deciphering, which does not introduce additional parameters. This ensures that our model remains efficient without incurring extra computational overhead. **2:** In the Introduction, we briefly analyze the computational efficiency of our method. Specifically, traditional backdoor adjustment requires traversing and repeatedly sampling all environmental patches, leading to exponential complexity: ${\\cal O}( {T \\times {\\cal N}_E^{{\\cal M}\\left( {\\rm{*}} \\right)}} )$. In contrast, our CaPaint method employs front-door adjustment, which eliminates the need for repeated sampling of all environmental patches. This improvement reduces the complexity to linear levels: $\\cal{O}(T \\times {\\cal N}_E)$. \n\n**Reduction in Complexity**: Our approach substantially reduces the complexity of the optimal spatio-temporal causal discovery process from exponential to quasi-linear levels. By performing targeted interventions only on identified environmental patches rather than the entire dataset, we achieve significant computational savings.\n\n**Scalability**: This reduction in complexity ensures efficient causal interventions, making our method more practical and scalable for real-world applications. Our method has been tested across multiple benchmark datasets (FireSys, SEVIR, Diffusion Reaction System (DRS), KTH, and TaxiBJ+), demonstrating its robustness and effectiveness in enhancing model performance.\n\n**Practical Applications**: By implementing these enhancements, CaPaint provides a robust framework for spatio-temporal dynamics with improved computational efficiency and scalability. This makes it suitable for deployment in various real-world scenarios where data scarcity and computational efficiency are critical concerns.\n\n- More visualization of the prediction results should be included even in supplementary material.\n\nThank you for your suggestion. In the main body of the paper, we have presented the visualization results for the SEVIR and TaxiBJ+ datasets. Additionally, we have included the visualization results for the KTH and DRS datasets in Appendix I and J, respectively. Although these visualizations are not in the supplementary material section, they are provided in the appendix. Furthermore, Appendix F also showcases the visualization of the inpainting effects of our CaPaint method on the SEVIR dataset.While we believe these visualizations adequately demonstrate the performance advantages of incorporating CaPaint, we are continuously updating our results to include more comprehensive visualizations. Specifically, we plan to add:\n\n1. **Visualizations for More Datasets**: We will supplement the visualizations with results from the FireSys dataset.\n2. **Visualizations for Different Backbones**: We will include visualization results using different backbone models.\n3. **Comparisons with Different Augmentation Methods**: We will provide comparisons of visualizations using various augmentation methods.\n\nUpon acceptance, we will include these additional results in the appendix for ease of reference. These updates will ensure that our paper provides a comprehensive overview of the performance improvements achieved by our method and its applicability to a wide range of datasets and model architectures.\n\nI believe my response has addressed your concerns. If you have any further questions, please feel free to let me know. Thank you!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nCQBM5YZDy", "reviewer": "Reviewer_s4LS", "summary": "The paper presents CaPaint to improve spatio-temporal predictions by identifying causal regions and employing diffusion inpainting techniques. The approach addresses the challenges of high computational costs in ST causal discovery.", "questions": "see weakness", "limitations": "The technical contribution of this work is somewhat incremental, providing only limited improvements compared to NuwaDynamics.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.  CaPaint seamlessly integrates with a variety of existing spatio-temporal prediction models. The paper thoroughly evaluates the method using diverse backbone models, showcasing the robustness and versatility of CaPaint across different scenarios.\n2. The experimental results across five real-world ST benchmarks demonstrate substantial improvements\n3. The combination of causal inference and diffusion models is sound.", "weaknesses": "1. The novelty of the proposed method is somewhat limited. The concept of causal patch discovery was already introduced in NuwaDynamics. This work primarily builds on that by utilizing diffusion models for data generation and proposing a different SCM, which is not necessarily better.\n2. The evaluations in Table 1 use the same datasets as NuwaDynamics. To ensure a fair comparison and better highlight the improvements, it is recommended that the authors use the same settings as NuwaDynamics and directly compare their results with it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Answer(Q1):** Thank you for your feedback. We **respectfully** disagree with the assessment that our method lacks novelty. Our work presents significant improvements and innovations over NuwaDynamics.\n\n1. **Conceptual Difference**:\n   - Our proposed method employs a different approach to causal adjustments, utilizing front-door adjustment rather than back-door adjustment as in NuwaDynamics. Back-door adjustment requires traversing and repeatedly sampling all environmental patches, leading to exponential complexity: ${\\cal O}( {T \\times {\\cal N}_E^{{\\cal M}\\left( {\\rm{*}} \\right)}} )$ \n   - In contrast, our method with front-door adjustment avoids the need for repeated sampling of all environmental patches, reducing the complexity to linear levels: $\\cal{O}(T \\times {\\cal N}_E)$\n   - This significant reduction in complexity provides practical advantages, especially in large-scale data processing and real-time applications, by substantially lowering computational resource requirements.\n2. **Different Methodology**:\n   - Our approach involves the use of generative models, specifically diffusion inpainting methods, for causal interventions. After fine-tuning the model, it achieves a global awareness of the data, enabling effective inpainting of environmental regions. In contrast, NuwaDynamics can only perceive local information around the surrounding areas.\n   - Traditionally, Diffusion Models (DMs) were utilized to generate high-quality synthetic images in the field of Computer Vision (CV). Inspired by the powerful generation capability of DMs given an input picture, we incorporate generative models like DMs into spatio-temporal video data analysis for data augmentation, without harming its efficiency greatly. Our key insight lies in addressing the challenge of data sparsity, and the experimental results have demonstrated that our method achieves significant improvements. This is a substantial innovation in this field.\n3. **Broad Applicability**:\n   - Our CaPaint method effectively addresses the issue of exponential complexity found in NuwaDynamics, resulting in more efficient processing. Furthermore, the spatio-temporal data sequences generated by our method are of higher quality, enhancing the robustness and generalizability of the model. This allows CaPaint to be applied in various scenarios, making it advantageous for practical applications and deployment in the industry.\n\nBy addressing these aspects, we hope to clarify the distinct advantages and innovations of our method compared to existing approaches.\n\n\n\n**Answer(Q2):** Thank you for your suggestion. Our CaPaint and Nuwa approaches inherently differ in their focus on front-door and back-door adjustments, respectively. Back-door adjustment, as used in Nuwa, requires traversing and repeatedly sampling as many environmental patches as possible, which leads to fundamentally different experimental setups. In contrast, front-door adjustment, which we employ, only needs to perceive the overall environment and fill in the environmental parts. This eliminates the need for repeatedly sampling numerous environmental patches, which is one of the key advantages of our method.\n\nAdditionally, in the experimental section of our paper, we have provided a comparison between our method and Nuwa under our settings. The results show that our method outperforms Nuwa across different datasets. This demonstrates the robustness and effectiveness of our approach.\n\n| Datasets    | Flip  | Rotate | Crop  | NuWa  | CaPaint |\n| ----------- | ----- | ------ | ----- | ----- | ------- |\n| **DRS**     | 2.10  | 2.11   | 2.34  | 2.02  | 1.57    |\n| **KTH**     | 23.15 | 23.14  | 23.11 | 22.32 | 20.56   |\n| **SEVIR**   | 15.41 | 15.45  | 15.95 | 15.14 | 14.63   |\n| **TaxiBJ+** | 16.47 | 16.39  | 15.94 | 15.11 | 12.87   |\n| **FireSys** | 17.02 | 17.07  | 17.15 | 16.68 | 15.79   |\n\n\n\nTo better explore the results of our method and Nuwa under the same settings, we followed your suggestion to align the experimental setup with that of Nuwa. Additionally, our experiments introduced backbones that have been updated in recent years, meaning the backbones we used are newer than those in Nuwa. Due to limited time and resources, we selected overlapping backbones such as  SimVP and PredRNN-V2, and overlapping datasets,TaxiBJ+ and KTH for comparison, setting the environmental ratio to 15% for testing.\n\n| Datasets    | SimVP   | SimVP | PredRNN-v2 | PredRNN-v2 |\n| ----------- | ------- | ----- | ---------- | ---------- |\n|             | CaPaint | Nuwa  | CaPaint    | Nuwa       |\n| **TaxiBJ+** | 2.21    | 2.56  | 2.86       | 3.25       |\n| **KTH**     | 31.26   | 33.98 | 38.45      | 40.37      |\n\nWe selected MAE as the reference metric, and it is evident that our proposed CaPaint outperforms Nuwa across the board. It is worth mentioning that Nuwa requires generating 4 and 5 augmented sequences per spatio-temporal sequence for TaxiBJ+ and KTH, respectively, to achieve its results. In contrast, we only needed to generate 2 sequences to surpass Nuwa's performance. This is because Nuwa can only mix local information around the environment, whereas our method can consider and fill in global details, generating higher-quality data. This aligns with the background of our paper, which addresses data scarcity and uneven sensor collection.\n\nIn summary, our more advanced concept is fundamentally different from Nuwa and is more advantageous for practical applications. I believe my response has addressed your concerns. If you have any further questions, please feel free to let me know. Thank you!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QggaY4cuiO", "reviewer": "Reviewer_nGNj", "summary": "The paper focuses on generalizability and interpretability for spatio-temporal predicting. The authors propose a causal structure plugin, named CaPaint, which identifies causal regions in data to generate data for scenarios where data are scarce. Experiments on five datasets demonstrate the effectiveness of the proposed method in spatio-temporal forecasting.", "questions": "Please see the Weaknesses.", "limitations": "The authors adequately addressed the limitations.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The paper focuses on the issue of modeling uneven and insufficient spatio-temporal data, which is a fascinating and significant area of research.\n\n2.To incorporate physical laws into deep networks, the authors propose a method that obeys the causal deciphering and performs interventions on the non-causal diffusion pathces, which is an extremely challenging problem.\n\n3.The authors have conducted experiments on five datasets, validating the effectiveness of the model, and have appropriately discussed the limitations of the model.", "weaknesses": "1.In line 39-44, the authors lack discussion of why the causality and interpretability of models can improve generalization capabilities when dealing with the uneven, insufficient data collection. \n\n2.In the left side of Fig.4, the visualizations of finer details are small and not clear enough, so it would be more informative to zoom in on local details of the image.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1:** In line 39-44, the authors lack discussion of why the causality and interpretability of models can improve generalization capabilities when dealing with the uneven, insufficient data collection.\n\n**Answer:** \n\n1. When data is sparse, models may learn **shortcut solutions from biased data**, as noted in several studies [1-3]. These shortcuts can significantly interfere with causal discovery, which relies on counterfactual reasoning to understand the underlying mechanisms that generate the observed data. In spatio-temporal domains, this issue is particularly critical as the temporal dependencies and spatial correlations are more complex and often underexplored. Traditional approaches have primarily focused on **spatio-temporal graphs**, but our work extends this to spatio-temporal continuity in **image data (video data)**, a relatively underexplored area. By addressing these gaps, we provide a more robust foundation for understanding and predicting spatio-temporal dynamics, ultimately enhancing the model's ability to generalize from incomplete or biased datasets.\n2. **Interpretability and causality** are strongly correlated. Discovering patterns in model and data predictions is crucial for identifying causality and enhancing interpretability. This process allows us to understand how different variables interact and contribute to the observed outcomes. In our work, we **leverage self-supervision** and employ global perception by **introducing inpainting to dynamically fill in potentially biased areas.** This approach ensures that the model can make informed predictions based on a more comprehensive understanding of the underlying data distribution, thus achieving interpretability. This interpretability, grounded in solid causal theory, allows us to make sense of the model's decisions, facilitating better generalization and robustness, especially in scenarios with uneven and insufficient data collection.\n3. To help readers better understand the relationship between data scarcity, causality, and interpretability, we have **cited more relevant literature** in the paper. These citations not only provide a historical context for the technical developments in this area but also highlight the critical correlation between data scarcity, causality, and interpretability. By grounding our discussion in established research, we illustrate how our approach aligns with and advances current understanding, providing a clearer picture of how addressing **causality and interpretability** can significantly improve generalization capabilities in spatio-temporal data analysis.\n\n**Q2:** In the left side of Fig.4, the visualizations of finer details are small and not clear enough, so it would be more informative to zoom in on local details of the image.\n\n**Answer:** Thank you for pointing this out. We appreciate your feedback and recognize the importance of clear and detailed visualizations. To address this issue, we will provide zoomed-in versions of the local details for different datasets to offer a clearer and more informative visualization. To improve the readability of the paper, we have re-arranged the layout of Figure 4. We have enlarged the relevant images and fonts to make it easier for readers to view the finer details. Additionally, we have systematically refined the layout of the entire paper, which includes adjustments to image sizes, fonts, and tables to enhance overall readability**.Upon acceptance, we will include these additional results and enhanced visualizations in the appendix for ease of reference.** This will ensure that all visual details are accessible and that the readers can fully appreciate the improvements and performance of our proposed method.\n\nI believe my response has addressed your concerns. If you have any further questions, please feel free to let me know. Thank you!\n\n**Reference**:\n\n[1] Causal Attention for Interpretable and Generalizable Graph Classification.\n\n[2] Enhancing Out-of-distribution Generalization on Graphs via Causal Attention Learning.\n\n[3] Reinforced causal explainer for graph neural networks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ibNGsVIZCb", "reviewer": "Reviewer_mbnW", "summary": "The paper introduces CaPaint, a causal structure plugin for spatio-temporal (ST) forecasting, aiming to identify causal regions in data and enable the model to perform causal reasoning. Utilizing a two-stage process and employing a novel image inpainting technique using a fine-tuned unconditional Diffusion Probabilistic Model (DDPM), the paper proposes a method to fill in the gaps identified as environmental parts, enhancing model generalizability and interpretability significantly.", "questions": "1. How effective is the inpainting technique implemented by CaPaint specifically on spatio-temporal datasets, and what are the key factors that influence its performance in these contexts? \n\n\n2. Why do traditional data augmentation methods, which can disrupt spatio-temporal characteristics, result in performances that are consistent with or only slightly worse than the original, instead of showing a significant decline? \n\n\n3. How does the performance compare when augmented data is combined with original data to form the training set for enhancing model generalizability, particularly when controlling for an equal amount of training data?", "limitations": "The authors highlight that the effectiveness of the method is limited under conditions of abundant data, as demonstrated through experiments that show more significant performance improvements under data-scarce conditions compared to when data is plentiful.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The paper is well-written, with clear, concise explanations and the use of figures effectively illustrates the model's mechanisms and results. \n\n- The paper introduces an interesting concept by incorporating causal inference into spatio-temporal data analysis, particularly through the integration of generative models \n\n- The experiments are thoroughly conducted across multiple datasets and backbones, results are overall promising.", "weaknesses": "- The abstract contains a typo where front-door adjustment is incorrectly referred to as back-door adjustment. \n\n- The paper appears to lack detailed descriptions on how the generated spatio-temporal data are synthesized into coherent ST sequences, missing crucial details on this aspect of the methodology. \n\n- The paper does not clearly demonstrate how the quality and efficiency of generation are improved. It is recommended to supplement with additional experiments to substantiate these aspects.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1:** Thank you for your careful review. We appreciate your attention to detail and will correct this typo to accurately reflect the use of front-door adjustment in the abstract.\n\n**W2:** Thank you for your valuable feedback. For each original spatio-temporal sequence, we enhance the data by first identifying causal regions using a Vision Transformer. We then apply diffusion inpainting to fine-tune the data and fill the identified environmental parts. After inpainting, we perform sampling with a probability ppp to mix the original sequences with the augmented sequences. This process ensures that the generated data maintains spatial and temporal coherence, resulting in high-quality, coherent spatio-temporal sequences.\n\n**W3:** Thank you for your suggestion. We have supplemented our paper with additional experiments comparing our method with NuwaDynamics. The results demonstrate that our method achieves similar or better performance with only 2 sequences, whereas Nuwa requires 4-5 sequences. This clearly shows the superior quality and efficiency of our generation process.\n\n| Datasets    | SimVP   | SimVP | PredRNN-v2 | PredRNN-v2 |\n| ----------- | ------- | ----- | ---------- | ---------- |\n|             | CaPaint | Nuwa  | CaPaint    | Nuwa       |\n| **TaxiBJ+** | 2.21    | 2.56  | 2.86       | 3.25       |\n| **KTH**     | 31.26   | 33.98 | 38.45      | 40.37      |\n\nWe selected MAE as the reference metric, and it is evident that our proposed CaPaint outperforms Nuwa across the board. It is worth mentioning that Nuwa requires generating 4 and 5 augmented sequences per spatio-temporal sequence for TaxiBJ+ and KTH, respectively, to achieve its results. In contrast, we only needed to generate 2 sequences to surpass Nuwa's performance. This is because Nuwa can only mix local information around the environment, whereas our method can consider and fill in global details, generating higher-quality data. This aligns with the background of our paper, which addresses data scarcity and uneven sensor collection.\n\n\n\n**Q1:** Thank you for your insightful question. We have provided an example of the inpainting technique in Appendix F. CaPaint effectively fills the environmental regions of the images based on their data distribution. We fine-tuned the Stable Diffusion model on our datasets, enabling it to learn the spatio-temporal data distributions accurately.\n\nThe key factors influencing the performance of the inpainting technique in these contexts include:\n\n1. **Data Distribution**: The model's ability to understand and replicate the inherent distribution of the spatio-temporal data.\n2. **Model Fine-tuning**: The extent to which the Stable Diffusion model has been fine-tuned on the specific datasets to capture the nuances of the data.\n3. **Environmental Region Identification**: The accuracy in identifying and masking the environmental patches that need inpainting.\n4. **Causal and Non-causal Patch Distinction**: The precision in distinguishing between causal and non-causal patches to ensure that the inpainting enhances the model’s generalizability and interpretability\n\n\n\n**Q2:** Thank you for your question. We briefly analyzed this in Section 4.4 of our experiments. Traditional data augmentation methods indeed disrupt spatio-temporal characteristics, leading to two possible outcomes:\n\n1. **Data Only Augmentation**: If only the data is augmented without corresponding changes to the labels, there is a significant decline in performance due to the loss of spatio-temporal coherence.\n2. **Data and Label Augmentation**: If both the data and labels are augmented together, the performance only slightly declines. This is because augmenting both preserves some of the spatio-temporal characteristics, maintaining a level of coherence between the input data and the expected output.\n\nBy retaining some of the intrinsic spatio-temporal properties through coordinated augmentation of both data and labels, the performance remains relatively stable.\n\n\n\n**Q3:** Thank you for your question. As shown in Figure 7 of our experiments section, even when maintaining an equal amount of training data, CaPaint demonstrates superior performance compared to the original backbone model. For instance, when comparing 50% original data with 25% original data combined with 25% augmented data, CaPaint's MAE and MSE are lower than those of the original backbone. This indicates that CaPaint effectively enhances model generalizability by leveraging the augmented data.\n\n| Metric        | 10%    | 25%    | 50%    | 75%    | 100%   |\n| ------------- | ------ | ------ | ------ | ------ | ------ |\n| **SimVP**     |        |        |        |        |        |\n| MAE           | 0.4925 | 0.4477 | 0.3215 | 0.2438 | 0.2320 |\n| MSE           | 0.5103 | 0.4434 | 0.2821 | 0.1843 | 0.1645 |\n| **SimVP+CaP** |        |        |        |        |        |\n| MAE           | 0.3925 | 0.2875 | 0.2633 | 0.2210 | 0.2057 |\n| MSE           | 0.3787 | 0.2541 | 0.2157 | 0.1586 | 0.1390 |\n\nI believe my response has addressed your concerns. If you have any further questions, please feel free to let me know. Thank you!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

We would like to extend our sincere gratitude to all reviewers for their thorough and insightful feedback on our manuscript. We appreciate the time and effort you have invested in evaluating our work. Below, we provide an overall summary addressing the common strengths, weaknesses, and suggestions highlighted in your reviews.

#### Common Strengths:

1. **Addressing Data Scarcity and Causal Connections**:  (Reviewer `botx`, `nGNj`,`mbnW`)
   - Multiple reviewers noted that our work effectively addresses the critical issues of data scarcity and the lack of causal connections in spatio-temporal (ST) prediction models. We are pleased that our efforts to establish a robust causal framework were recognized.
2. **Innovative Methodology**: (Reviewer `botx`, `s4LS`,`mbnW`)
   - The innovative use of self-supervised Vision Transformer (ViT) reconstruction for causal patch identification and the employment of diffusion inpainting techniques were highlighted as significant strengths. We are encouraged that our novel approach was well-received.
3. **Performance Improvements**: (Reviewer `botx`, `nGNj`,`mbnW`, `s4LS`)
   - Reviewers acknowledged the substantial improvements demonstrated across various ST benchmarks, validating the robustness and versatility of CaPaint. The integration of diffusion models as a novel data augmentation technique was noted as a key contribution.
4. **Soundness and Contribution**:(Reviewer `botx`, `nGNj`,`mbnW`, `s4LS`,)
   - The overall soundness and contribution of our work were rated positively. We are grateful for the recognition of the technical rigor and the potential impact of our research in advancing the field of spatio-temporal data analysis.

#### Common Weaknesses and Our Responses:

1. **Computational Efficiency and Scalability**: (Reviewer `botx`, `mbnW`)
   - Several reviewers expressed concerns regarding the computational efficiency and scalability of our method. We have addressed this by elaborating on how our use of the attention mechanism and front-door adjustment reduces complexity from exponential to quasi-linear levels, ensuring efficient causal interventions suitable for real-world applications.
2. **Visualization of Prediction Results**: (Reviewer `botx`, `nGNj`)
   - Reviewers suggested including more visualizations of prediction results, even in supplementary material. We have updated our paper to include zoomed-in versions of local details for different datasets, re-arranged the layout for better readability, and added more comprehensive visualizations for additional datasets, backbones, and augmentation methods. These enhancements will be included in the appendix upon acceptance.
3. **Clarification on Causality and Interpretability**: (Reviewer `mbnW`, `s4LS`,)
   - The relationship between causality, interpretability, and their impact on generalization capabilities when dealing with uneven and insufficient data collection was pointed out as lacking. We have enriched our discussion by citing more relevant literature and providing a detailed explanation of how our approach leverages self-supervision and inpainting to dynamically address biased areas, thereby achieving robust causal discovery and interpretability.
4. **Comparison with NuwaDynamics**: (Reviewer `s4LS`,)
   - Reviewers noted the need for a more direct comparison with NuwaDynamics using the same settings. We have aligned our experimental setup with NuwaDynamics and provided additional experiments to highlight our method's superior performance with fewer sequences, demonstrating the efficiency and quality of CaPaint.

#### Additional Improvements:

- **Typographical Corrections**: We have corrected the typo in the abstract where front-door adjustment was incorrectly referred to as back-door adjustment.
- **Detailed Methodological Clarifications**: We have provided more detailed descriptions on the synthesis of spatio-temporal data into coherent sequences and included additional experiments to substantiate the improvements in quality and efficiency of generation. 

In conclusion, we are grateful for the positive feedback and constructive criticisms provided by the reviewers. Your comments have been invaluable in helping us improve our manuscript. We believe that the revisions and additional experiments we have incorporated address the concerns raised and enhance the overall quality and clarity of our work.

Thank you once again for your detailed and thoughtful reviews. We look forward to your continued feedback and hope our revised submission meets your expectations.

Warm regards,

Authors

Meta-review: We thank the authors for their submission.  Reviewers agreed that the work is a novel and thoroughly examined way to incorporate causal assumptions into spatiotemporal models and imputation.  Reviewers also appreciated the additional experimental results directly comparing the proposed CaPaint and Nuwa methods.
Decision: Accept (poster)