Causal Deciphering and Inpainting in Spatio-Temporal Dynamics via Diffusion Model

Authors: yifan  duan
jian  zhao
junyuan  mao
hao  wu
jingyu  xu
shilong  wang
caoyuan  ma
kai  wang
kun  wang
xuelong  li
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 09:16:51

Abstract: Spatio-temporal (ST) prediction has garnered a De facto attention in earth sciences, such as meteorological prediction, human mobility perception. However, the scarcity of data coupled with the high expenses involved in sensor deployment results in notable data imbalances. Furthermore, models that are excessively customized and devoid of causal connections further undermine the generalizability and  interpretability. To this end, we establish a causal framework for ST predictions, termed CaPaint, which targets to identify causal regions in data and endow model with causal reasoning ability in a two-stage process. Going beyond this process, we utilize the back-door adjustment to specifically address the sub-regions identified as non-causal in the upstream phase. Specifically, we employ a novel image inpainting technique. By using a fine-tuned unconditional Diffusion Probabilistic Model (DDPM) as the generative prior, we in-fill the masks defined as environmental parts, offering the possibility of reliable extrapolation for potential data distributions. CaPaint overcomes the high complexity dilemma of optimal ST causal discovery models by reducing the data generation complexity from exponential to quasi-linear levels. Extensive experiments conducted on five real-world ST benchmarks demonstrate that integrating the CaPaint concept allows models to achieve improvements ranging from 4.3% to 77.3%. Moreover, compared to traditional mainstream ST augmenters, CaPaint underscores the potential of diffusion models in ST enhancement, offering a novel paradigm for this field. Our project is available at https://anonymous.4open.science/r/12345-DFCC.

Main paper: 
 introduction: 
 Deep learning methodologies have achieved groundbreaking success across a wide array of spatiotemporal (ST) dynamics systems [28, 84, 44] , which include meteorological forecasting [3, 50, 59, 92] , wildfire spread modeling [71, 20] , intelligent transportation [29, 27, 87] , and human mobility systems [27, 90] , to name just a few. Traditional ST dynamics approaches, based on first-principles [4, 53] , often come with high computational costs. In contrast, ST dynamic analysis methods based on deep learning are not directly reliant on the explicit expression of physical laws but are data-driven [28, 84, 3, 27] , relying on training models with large-scale observable datasets [86, 65, 92] .In a parallel vein, numerous efforts aim to incorporate physical laws into deep networks [35, 8, 54, 31, 81] , termed Physics-Informed Neural Networks (PINNs), which blend deep learning principles with physics to address challenges in scientific computing, particularly in fluid dynamics.PINNs augment traditional neural network models by including a term in the loss function that accounts for the physical laws governing fluid dynamics, such as the Navier-Stokes equations [11] . This ensures that the network's predictions are not only consistent with empirical data but also comply with the fundamental principles of fluid dynamics. However, the off-the-shelf PINNs often suffer from limited generalization capabilities, primarily due to their customized loss function designs and the neglect of specific network parameter contexts [70, 16] .To date, the data-driven deep models are still dominant in ST dynamical systems, where the numerical simulation methods and PINNs generally lag behind. The reason may stem from the rise of large models [1, 76, 28] and the high costs associated with collecting ST data from sensors [93, 39] , which creates a significant conflict between the increasing size of data-hungry models and the uneven, insufficient data collection. To this end, in the ST domain, there is looming research aimed at enhancing the causality and interpretability of models.Unfortunately, research into causality within the field of ST dynamics is lagging. Although some work has considered causal design, due to specific domain constraints and architectural design, it can only enhance the tailor-made capabilities of the model for specific tasks [95, 38] . Moreover, causal discovery tools [12, 15] applied to ST systems often confront the "curse of dimensionality" issue during dimension reduction, despite their effectiveness in elucidating causal relationships from statistical data [75, 47] . Furthermore, NuwaDynamics [82] for the first time proposed decomposing causal and non-causal regions in ST sequences and enhancing the robustness and generalizability of downstream model training by generating more potential distribution ST sequences through mixup [100] . CauSTG [106] and CaST [95] address the issue of ST distribution shifts by implicitly modeling the time series embeddings and employing intervention techniques to observe these shifts.Though promising, CauSTG [106] and CaST [95] focus on modeling graph-related data, they lack an understanding of high-dimensional observational data (Dimension D < 256). NuwaDynamics, on the other hand, explores all environments through backdoor adjustments [51] , generating a vast number of sequences, which lead to nearly O(T × N In light of this, we propose a general causal structure plugin, termed CaPaint, designed to decipher causal regions in ST data without adding extra computational cost, while intervening in non-causal areas to boost the model's generalizability and interpretability. Specifically, our method employs a straightforward approach to causal deciphering, utilizing a vision transformer architecture [33] for self-supervised ST data reconstruction. During reconstruction, we leverage attention scores from the self-attention mechanism [23] to map onto important causal patches, thus endowing the model with interpretability. By ranking the entire set of importance scores, we define those with lower scores as environmental patches, which contribute minimally to the model. Building on this, we perform causal interventions in these environmental areas to aid the model in understanding more latent, complex, and imperceptible distributions, thereby enhancing the overall generalizability of the model (see Figure 1 ). Concretely, we mask trivial regions and perform generation using DDPM [24, 32] fine-tuned on specific ST data, which can also be interpreted as a ST data inpainting approach.Insight. ❶ CaPaint obeys the causal deciphering, and guided by the principle of frontdoor adjustment [51, 52] from causal theory, CaPaint performs diffusion inpainting interventions on the environmental (non-causal) diffusion patches while reducing the temporal complexity to a manageable O(T × N E ) (from O(T × N• In this paper, we introduce a novel causal structure plugin, CaPaint, which leverages the concept of frontdoor adjustment from causal theory. CaPaint enables various backbone models to learn from a broader distribution of data while providing enhanced interpretability for the models' predictions.• By integrating diffusion generative models with ST dynamics, CaPaint selectively perturbs noncausal regions while maintaining the integrity of core causal areas. This approach generates valuable and reliable data for scenarios where high-quality data are scarce.• We conduct extensive experiments across five diverse and representative datasets from different domains, utilizing seven backbone models to assess the effectiveness of the CaPaint method. The empirical results demonstrate that CaPaint consistently enhances performance on all tested datasets and across all backbone models (3.7%∼77.3%).Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: In this paper, we introduce the spatio-temporal causal concept in the data mining realm, aimed at enhancing the reliability and accuracy of financial spatio-temporal prediction.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work & technical background: 
 Spatio-temporal Predictive Learning: Various architectures have achieved significant predictive performance in ST domain, which can primarily be categorized as follows: CNN-based models utilize convolutional layers to effectively capture spatial features [45, 48, 77, 9] . RNN-based models, are capable of processing temporal sequence data and are well-suited for understanding temporal changes, showing excellent performance in the prediction of action continuity [69, 80, 86, 72] . GNN-based models effectively capture spatial dependencies and temporal dynamics in data, making them suitable for complex tasks involving geographic locations and temporal changes [44, 25, 36, 17, 99, 98, 83, 14] .Transformer-based models employ self-attention mechanisms to process sequential data in parallel, enhancing the learning of long-term dependencies, and have been used for ST data prediction in complex scenarios [2, 19, 89, 91, 10, 88] .Causal inference: causal discovery algorithms, originally devised for unstructured random vectors [66, 104] , have progressively been adapted for ST data analysis [75, 47] . Within the extensive field of deep learning research, the study of causal inference aims to ensure a more stable and robust learning and reasoning paradigm. Recently, an array of techniques has been developed to delve into the nuances of causal features [60, 61, 43, 97] , identifying and eliminating spurious correlations [21, 34, 56] .Generative models especially diffusion-based model has gained significant popularity particularly in image and video generation [24, 64, 62] . Sampling optimization algorithms have been used to accelerate the sampling process of diffusion models, significantly reducing the number of steps while improving efficiency. [67, 41] . Additionally, generative models have also been applied to 3D scene generation and point cloud processing, as demonstrated in [40, 30, 73, 74, 22, 63] Image Inpainting is a technique used to fill in missing or damaged parts of an image. This field can be broadly categorized into the following types. VAE-based methods: These methods leverage Variational Autoencoders to balance diversity and reconstruction [101, 103, 26] . GAN-based methods: Since the introduction of Generative Adversarial Networks, these methods have been widely used for image inpainting [55, 102, 49] . Diffusion model-based methods: Diffusion models have recently shown outstanding performance in image inpainting [46, 68, 57] .CaPaint can identify and intervene in non-causal regions, improving the predictive accuracy and interpretability in various ST domains such as meteorology, human mobility, and disaster management. This improvement leads to better decision-making processes and resource allocation, ultimately benefiting society by providing more reliable and understandable predictive models.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: In this work, we systematically discuss the limitations of our research and outline directions for future work. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
methodology: 
 In this section, we systematically introduce causal structure plugin, CaPaint. Initially, we elucidate the methods employed in the upstream phase to delineate causal and non-causal regions (Sec 3.1). Subsequently, we showcase the theoretical underpinnings supporting the CaPaint (Sec 3.2). Building on this causal theory, we further engage in causal intervention within observational data (Sec 3.3). Lastly, we demonstrate how sampling-enhanced ST observations can benefit the complexity of the model's on-device deployment (Sec 3.4).Problem Formulation. In ST settings, We represent ST observations as a sequence {X t } T t=1 , where each observation X t ∈ R H×W ×Cin originates from these sequences. Our objective is to predict the trajectory for the forthcoming K steps, denoted as {X t+1 } T +K t=T , with each future state X t+k mapped within R H×W ×Cout . Here, H and W indicate the spatial grid dimensions, while C in and C out define the input and output dimensionality of the observations, respectively. To find the causal (non-causal) patches with no labels, we employ a self-supervised reconstruction approach based on the Vision Transformer (ViT) [13] to identify key regions within ST observations. ViT segments the image into multiple patches and calculates the relationships between them using a self-attention mechanism. Due to no label property, we intentionally omit the use of the [Cls] token in classification task and send data into ViT for encouraging "local-to-global" reconstruction.Specifically, each ST data X t , is divided into N = HW/p 2 patches, where each patch x patch t ∈ R N ×(p 2 ×Cin) , with (H, W ) being the resolution of the original ST data and (p, p) the resolution of each patch. Subsequently, each patch is mapped to a D-dimensional token through a learnable linear layer, incorporating position embedding to enhance the model's sensitivity to positional information. These tokens are then fed into successive L stacked transformer blocks, as described in Equation 1:EQUATIONwhere LN denotes layer normalization, and MLP represents multi-layer perceptron. The upstream selfsupervised reconstruction task enables the model to learn intrinsic property of ST data. Navigating the MSA mechanism [78, 96] , each patch x patch t derived from the ST observation X t is transformed into queries q, keys k, and values v, and then calculates the relevance of each patch to others, forming a weighted representation that focuses on the most informative parts. The attention weights A h i,j stored in the attention map A h in each head are computed using the scaled dot-product:EQUATIONwhere ψ tr ∈ R N ×3D h are the parameter matrices, D h represents the dimension of each head, Q, K, and V collectively denote the sets of queries q, keys k, and values v. In our approach, the determination of causal patches, is driven by an analysis of the attention maps A. Each row in an attention map is normalized and represents the importance of other patches relative to the current patch x i t . However, to ascertain the overall importance of each patch across the entire input, we aggregate the contributions by summing the values along the columns of the A. To integrate insights across multiple heads, we sum these measures across all heads and then normalize the resultant vector to derive a comprehensive importance score for each patch:EQUATIONwhere S represents the normalized importance score vector, A h i,j ∈ A denotes the attention that x i t pays to x j t for each head, H is the number of heads. We sort the importance scores in S and select the patches corresponding to the lowest K scores as environmental patches storing in O e . The remaining patches are considered causal patches O c :EQUATIONwhere C(S) is the counting function, ϵ represents the proportion of patches selected as causal, and Ψ(X t ) denotes the set of patches in the ST observation X t . We identify the causal patches by locating the indices with the highest values in S and define the non-causal parts as the environmental parts.Our goal is to perform causal interventions on the environmental parts.To address issues of ST data scarcity and poor transferability, we examine the evaluation process using a Structural Causal Model (SCM) [52] , as shown in Fig 3 . We represent abstract data variables by nodes, with directed links symbolizing causality. The SCM illustrates the interaction among variables through a graphical definition of causation, demonstrating the interconnected nature of these elements. As depicted in the left part, NuwaDynamics employs the backdoor adjustment to enhance the model's generalization performance: • XC ← D → Y. In this structure, D serves as a confounder, creating a misleading path between XC and Y. Here, XC represents the causal component within X .➠ X C ← X → X \C .• XC → X * C → Y. X * C acts as the surrogate variable of XC and completes XC to align it with the data distribution. Initially, it derives from and encompasses XC . Specifically, it envisions the potential complete observations that should exist when observing the sub-counterpart XC . Additionally, X * C adheres to the data distribution and upholds the intrinsic knowledge of graph properties, thus eliminating any link between D and X * C . Consequently, X * C is well-suited to act as the mediator, which in turn influences the model's predictions (→ Y).In our front-door adjustment framework, we utilize do-calculus on the variable XC to eliminate the spurious correlations introduced by D → Y. Specifically, we achieve this by summing over potential surrogate observations X * C . This approach allows us to connect two identifiable partial effects: XC → X * C and X * C → Y: EQUATIONP X * C |do XC = XC = P X * C | XC = XCX * C = ∼ X * C ; ∼ X C = ∼ X ′ C .Compared to previous work NuwaDynamics, CaPaint utilizes causal regions to generate global surrogate variables in a more rational manner, circumventing the cumbersome need to traverse environmental variables inherent in backdoor adjustments. In fact, backdoor adjustments often likely violate underlying properties, leading to the generation of non-existent data distributions. The broader scenarios of CaPaint will be detailed in Appendix C.Building on the principles of causal analysis outlined above, we proceed to perform interventions on the environmental patches using diffusion inpainting, which enables us to manipulate the environmental areas. Initially, given the unique complexities of ST datasets, we fine-tune the diffusion parameters to adapt seamlessly to the domain-specific challenges, which enhances the accuracy of our interventions on environmental patches. Diffusion models learn the distribution of data through a forward noise addition process and a reverse denoising process:q(Xt | Xt-1) = N (Xt; 1 -βtXt-1, βtI), p θ (Xt-1 | xt) = N (Xt-1; µ θ (Xt, t), Σ θ (Xt, t)) (6)where X t represents the data state at time step t, undergoing a transformation from its previous state x t-1 , β t controls the variance of the noise added at each step in the forward process, µ θ and Σ θ are neural network outputs that approximate the mean and covariance, respectively. The fine-tuning objective of the diffusion process is designed to approximate the data distribution more accurately. Specifically, the training objective for diffusion models, denoted as ϵ θ , which predicts the noise, is typically defined as a simplified version of the variational bound:EQUATION)where c is the condition information. In this paper, we perform inpainting on the environmental patches of ST data. Inspired by [42] , we generate a mask image for each ST data where the causal patches are black and the environmental patches are white. By independently sampling the causal and environmental patches and applying the diffusion inpainting process, we are able to generate augmented ST observation data. The detailed algorithmic process is shown in Appendix A.EQUATIONEQUATIONwhere X cau and X env denote causal patches and environmental patches, m is a binary mask matrix, α t represents the scaling factor at each diffusion step, determining the variance retained in the transition from X t-1 to X t . The cumulative product ᾱt = t i=1 α i represents the accumulated scaling effect from the T = 0 to step t. Equation 9 illustrates the merging of environmental patches and causal patches. Finally, the enhanced ST observation data are stored within our temporal sequence repository to bolster the downstream backbone.Previous work [82] assumed that the closer the time point is to the present, the greater its influence, and thus used Gaussian sampling to select more ST data closer to the current time point. However, we argue that uniform sampling can better enhance the model's generalization ability. To enhance computational efficiency while ensuring prediction accuracy, we employ a ST sequence modeling approach that samples at each time point with a fixed probability controlled by the hyperparameter p. This method allows us to sample from both original and generated data at each time point, thereby creating a new spatiotemporal sequence. We use two hyperparameters: p, which controls the sampling probability, and r, which determines the number of generated spatiotemporal sequences, achieving an optimal balance between computational efficiency and prediction accuracy. The specific sampling process can be represented by the following equation:EQUATIONwhere X t represents the collection of original and generated data at time point t, and Sample(X t , p) denotes the dataset obtained by sampling from X t with probability p. The hyperparameter p is directly set as the sampling probability, while r is used to specify the number of generated ST sequences.By reducing the complexity of optimal ST causal discovery models, CaPaint offers a cost-effective solution for handling high-dimensional ST data. This makes advanced predictive technologies more accessible across a broader range of applications, particularly in fields with limited computational resources.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: For the datasets disclosed in the article, we have provided information regarding their sources and origins.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiments: 
 In this section, we will validate the effectiveness of our proposed causal structure plugin, CaPaint.We design four research questions (RQs) to comprehensively evaluate the performance of CaPaint: RQ1: Does CaPaint effectively enhance model performance and applicability? RQ2: How does CaPaint perform in data-scarce scenarios? RQ3: How does the performance of CaPaint compare with other augmentation methods? RQ4: Is CaPaint effective for long-term time step predictions? Through these research questions, we aim to validate the effectiveness and advantages of CaPaint in handling ST data from multiple perspectives.Datasets. We extensively evaluate our proposal using a diverse range of benchmark datasets spanning multiple fields, include FireSys [7] , SEVIR [79] , Diffusion reaction system (DRS) [6] , KTH [58] and TaxiBJ+ [37] . Specifically, FireSys represents fire dynamics, SEVIR covers meteorological events, DRS involves physical control systems, KTH focuses on human motion dynamics, and TaxiBJ+ is a transportation dataset. Detailed information can be found in the Appendix B.Backbones and Metrics To validate the generalizability of CaPaint, we select multiple model frameworks for our experiments, including the classic model like ConvLSTM [65] , PredRNN-V2 [86] , Vision Transformer (ViT) [13] , MAU [5] , the efficiency-focused SimVP [18] , and some of the latest models like MmvP [105] and Earthfarsser [92] . Our evaluation metrics include mean absolute error (MAE), mean squared error (MSE), and structural similarity index measure (SSIM). Detailed information can be found in the Appendix D. 4 , we clearly see that the +CaPaint effectively reduces the model's prediction loss. Moreover, it is observed that +CaPaint provides more accurate predictions in finer details, closely aligning with the actual result curves. This demonstrates CaPaint's capability to enhance prediction accuracy and reliability, ensuring that the forecasts closely mirror real-world outcomes.Obs 3. +CaPaint remains effective in long-Term ST predictions. By analyzing the middle of Figure 4 , we observe that +CaPaint continues to demonstrate its effectiveness in long-term time step predictions for ST tasks. For instance, the details in the SEVIR dataset predictions improve significantly, indicating that CaPaint is still applicable and beneficial in challenging ST tasks. In this section, we compare the performance of different data augmentation methods. Tab 2 shows the model performance using various data augmentation methods across multiple datasets, measured by MAE. It can be seen that traditional data augmentation methods, such as flipping, rotation, and cropping, produce results that are either on par with or slightly worse than the original data. Take the FireSys dataset as an example, MAE increased from 17.01 → 17.07 after rotation augmentation. This indicates that conventional data augmentation methods may disrupt the intrinsic properties of ST data, thereby negatively impacting model performance.In contrast, our method CaPaint achieves the best performance across all datasets. For instance, on the TaxiBJ+ dataset, the MAE with CaPaint augmentation is 12.87, which is significantly better than the MAE of 15.11 with NuwaDynamics manual mixup augmentation and the MAE of 15.94 with other traditional augmentation methods such as cropping. These results highlight the advantage of our method in preserving the integrity of ST data properties. CaPaint not only effectively avoids the disruption caused by data augmentation processes on ST data characteristics but also significantly enhances the model's predictive capability.The integration of causal reasoning into ST models encourages the development of AI systems that better mimic human understanding of cause-andeffect relationships. This can lead to more robust AI models capable of generalizing across different scenarios, fostering trust and reliability in AI applications. 
conclusion & future work: 
 In this study, we advance the exploration of applying front-door adjustment and causality principles to spatio-temporal forecasting tasks through the introduction of CaPaint. Building upon the foundation of upstream self-supervised learning, we identify causal regions as crucial elements for generating comprehensive and potential data distributions. By integrating diffusion generative models, we ensure the generated data's rationality and generalizability, thereby enhancing the downstream models' ability to generalize beyond the observed distribution and improving their interpretability. Moving forward, we plan to explore various generative models for the production of arbitrary-channel ST data to enhance the CaPaint robustness.Techniques: CaPaint introduces novel data augmentation methods using diffusion inpainting, which can inspire further research and innovation in data augmentation and ST prediction. This can lead to the emergence of new techniques, enhancing the robustness and performance of AI models in various domains.The CaPaint framework represents a significant advancement in the field of ST dynamics, particularly in its ability to address sparse data scenarios, which reduces the need for extensive sensor deployments and lowers associated costs. Additionally, CaPaint enhances predictive accuracy, interpretability, and efficiency, promotes causal reasoning in AI, and introduces innovative data augmentation techniques.Responsible AI research and deployment should leverage these strengths to maximize benefits while minimizing risks. 
acknowledgement: 
 This work was supported by National Natural Science Foundation of China (62476224).Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: we have specified all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: In this paper, we have reported error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: In this paper, we provide detailed information about the experimental resources, including GPU configurations used in our studies.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: We have provided the societal impacts of the work.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper does not address issues related to this aspect.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: All creators and original owners of the assets used in our paper, such as code, data, and models, have been properly credited. We have explicitly mentioned the licenses and terms of use for each asset and have ensured full compliance with these terms throughout our research.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: The research presented in this paper is not concerned with new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: This paper does not involve experiments or research related to human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: This paper does not address potential risks incurred by study participants.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Algorithm 1 Causal Intervention with Diffusion Inpainting 1: Input: ST observation data X, masked image X mask 2: Output: Augmentation ST observation dataset X A 3: Initialize X T ∼ N (0, I) where T is the total number of diffusion steps 4: /* Iterate backwards through diffusion steps */ 5: for t = T to 1 do ϵ ∼ N (0, I)/* Sample causal region */ 9:X cau t-1 = √ ᾱt X 0 + (1 -ᾱt )ϵ 10:/* Sample Gaussian noise N */ 11:z ∼ N (0, I)/* Causal Intervention on Environmental Patches */ 13:X env t-1 = 1 √ αt X t -βt √ 1-ᾱt ϵ θ (X t , t) + σ t z 14:/* Combine causal and environmental patches */ 15:X t-1 = m ⊙ X cau t-1 + (1 -m) ⊙ X env t-116:X t ∼ N ( 1 -β t-1 X t-1 , β t-1 I) 17:end for 18: return X A as the augmentation datasetThe algorithm for Causal Intervention with Diffusion Inpainting aims to augment ST observation data through a series of diffusion steps that iteratively refine the data by applying causal interventions and combining them with environmental patches. Here is a detailed step-by-step description:• Input: The original ST observation data X, and a masked image X mask .• Output: An augmented ST observation dataset X A .• The process begins by initializing X T , which represents the data at the final diffusion step, to be a sample from a normal distribution centered at zero with identity covariance.• The main loop of the algorithm runs backward from the last diffusion step T to the first. In each step:1. Gaussian noise ϵ t is sampled to simulate the diffusion process. 2. A causal region X cau is sampled where the causal effect is calculated as a blend of the original data and the Gaussian noise, emphasizing areas of interest that should retain more original data characteristics. 3. Gaussian noise N t is sampled again, providing variability to the non-causal or environmental regions. 4. The environmental patches X env are updated using the data from the previous step adjusted by a damping factor and the added noise, simulating environmental changes. 5. The causal and environmental patches are then combined, where the mask M determines the specific locations for the causal and environmental updates in the data, specifying which parts are from the causal region and which are from the environmental region. 6. The data for the next step, X t-1 , is computed by normalizing the combined updates, preparing it for the next iteration or output if it is the first step.• Finally, the algorithm outputs the augmented data set X A , which is the result of the iterative causal intervention and environmental blending over the diffusion process.SSIM stands for Structural Similarity Index Measure, which is a method for measuring the similarity between two images. It compares the structural information of the images, including luminance, contrast, and texture, to determine how similar they are. SSIM is commonly used in image and video processing applications, such as image compression and quality assessment.PSNR stands for Peak Signal-to-Noise Ratio. It is a measure of video or image quality that compares the original signal to the compressed or transmitted signal. The higher the PSNR value, the better the quality of the compressed or transmitted signal. PSNR is commonly used in video and image compression applications to evaluate the effectiveness of compression algorithms.) loss is a commonly used loss function in machine learning and deep learning models. This loss function calculates the average of the squared differences between the predicted and actual values.Datasets. Here we summarize the details (Tab. 1) of the datasets used in this paper:• TaxiBJ+: This dataset contains trajectory data obtained from the GPS of taxis in Beijing, divided into two separate channels: inflow and outflow. Additionally, the dataset has been extended from 32×32 to 128×128 by collecting recent trajectory data from Beijing. The development and application of the CaPaint framework in spatio-temporal (ST) dynamics bring several positive broader impacts. Understanding these impacts is crucial for responsible AI research and deployment.1. Data Imputation in Sparse Scenarios: CaPaint excels in sparse data scenarios, effectively filling in missing data. This reduces the need for extensive sensor deployments, significantly lowering the cost associated with sensor installation. By optimizing data coverage and utilization, CaPaint not only enhances resource efficiency but also achieves substantial cost savings.In our research, we investigate the performance of our models using Mean Squared Error (MSE), Mean Absolute Error (MAE), and Structural Similarity Index Measure (SSIM). The formulas for evaluating these indicators, converted into decibels (dB) where applicable, are as follows:Mean Squared Error (MSE)Mean Squared Error (MSE) measures the average of the squares of the errors, that is, the average squared difference between the estimated values and the actual value. The MSE is given by:MSE = 1 N N i=1 (Y i -Ŷi ) 2 (D.1)where Y i is the actual value, Ŷi is the predicted value, and N is the number of observations.Mean Absolute Error (MAE) measures the average magnitude of the errors in a set of predictions, without considering their direction. It is the average over the test sample of the absolute differences between prediction and actual observation where all individual differences have equal weight. The MAE is given by:MAE = 1 N N i=1 Y i -Ŷi (D.2)where Y i is the actual value, Ŷi is the predicted value, and N is the number of observations.Structural Similarity Index Measure (SSIM) is used for measuring the similarity between two images. The SSIM index is a decimal value between -1 and 1, where 1 is only reachable in the case of two identical sets of data. The SSIM formula can be quite complex due to its consideration of luminance, contrast, and structure comparison functions between the two images:SSIM(x, y) = (2µ x µ y + C 1 )(2σ xy + C 2 ) (µ 2 x + µ 2 y + C 1 )(σ 2 x + σ 2 y + C 2 ) (D.3)where µ x , µ y are the average of x and y respectively, σ 2 x , σ 2 y are the variance of x and y respectively, σ xy is the covariance of x and y, and C 1 , C 2 are variables to stabilize the division with weak denominator.While the implementation of the CaPaint method has demonstrated significant improvements in prediction accuracy and detail preservation in spatio-temporal forecasting tasks, its enhancements are most pronounced in scenarios characterized by data scarcity or uneven data distribution. In contexts where datasets are abundant and exhibit a broad and uniform distribution, the incremental gains offered by CaPaint may not be as substantial. Nevertheless, the method remains effective, providing consistent, albeit smaller, improvements across diverse data environments. The figure illustrates the process of maintaining causal regions intact while performing inpainting on non-causal (environmental) regions. The approach involves identifying and deciphering the causal regions (left), intervening by applying diffusion inpainting on the environmental patches (middle), and subsequently generating altered ST data copies (right). This method ensures that the intrinsic causal relationships within the data are preserved, while variations are introduced in the environmental context to augment the dataset effectively.Oceanic Observation In The first row shows the ground truth for a walking individual. The second row, processed by Earthfarser, exhibits noticeable blurring and loss of detail. The third row, enhanced with +CaPaint, demonstrates a marked improvement in capturing fine details such as the shadow of the person and the accuracy of the foot motion, as highlighted in the red boxes.The introduction of CaPaint has led to reductions in Mean Squared Error MSE and MAE, while the SSIM has shown improvements. These changes indicate that the CaPaint method effectively enhances model prediction accuracy and image quality. However, due to the high quality of the model predictions, the improvements might not be readily observable to the naked eye. Despite this, the NeurIPS Paper ChecklistJustification: The study presented in this paper conforms to the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).M( * ) E 


Decision: We thank the authors for their submission.  Reviewers agreed that the work is a novel and thoroughly examined way to incorporate causal assumptions into spatiotemporal models and imputation.  Reviewers also appreciated the additional experimental results directly comparing the proposed CaPaint and Nuwa methods.

Reviews: review: paper_summary: paper_summary: This paper introduces a groundbreaking framework named CaPaint, which is designed to tackle the critical issues of data scarcity and the absence of causal connections in spatiotemporal (ST) prediction models. The authors have established a robust causal framework that not only identifies regions within data that exhibit causal relationships but also endows the model with the capability to reason about causality during a two-stage processing procedure. In the initial stage, they leverage self-supervised Vision Transformer (ViT) reconstruction to identify the crucial causal patches within ST observations. This is followed by an intervention phase where they employ diffusion inpainting techniques to manipulate non-causal areas while preserving the integrity of core causal areas. The innovative method reduces the complexity of generating data from exponential levels to quasi-linear levels, thereby significantly enhancing efficiency. Moreover, it has shown remarkable improvements across various ST benchmarks by integrating diffusion models as a novel data augmentation technique, marking a paradigm shift for this field.
strength_weakness: strength_weakness: - Details on computational efficiency or scalability of the proposed method are not provided, leaving it as a potential limitation for practical applications.
- More visualization of the prediction results should be included even in supplementary material.
questions: questions: Mentioned in the weakness section.
limitations: limitations: I think the author addressed the limitation mentioned in this work.
paper_summary: paper_summary: The paper presents CaPaint to improve spatio-temporal predictions by identifying causal regions and employing diffusion inpainting techniques. The approach addresses the challenges of high computational costs in ST causal discovery.
strength_weakness: strength_weakness: 1. The novelty of the proposed method is somewhat limited. The concept of causal patch discovery was already introduced in NuwaDynamics. This work primarily builds on that by utilizing diffusion models for data generation and proposing a different SCM, which is not necessarily better.
2. The evaluations in Table 1 use the same datasets as NuwaDynamics. To ensure a fair comparison and better highlight the improvements, it is recommended that the authors use the same settings as NuwaDynamics and directly compare their results with it.
questions: questions: see weakness
limitations: limitations: The technical contribution of this work is somewhat incremental, providing only limited improvements compared to NuwaDynamics.
paper_summary: paper_summary: The paper focuses on generalizability and interpretability for spatio-temporal predicting. The authors propose a causal structure plugin, named CaPaint, which identifies causal regions in data to generate data for scenarios where data are scarce. Experiments on five datasets demonstrate the effectiveness of the proposed method in spatio-temporal forecasting.
strength_weakness: strength_weakness: 1.In line 39-44, the authors lack discussion of why the causality and interpretability of models can improve generalization capabilities when dealing with the uneven, insufficient data collection. 

2.In the left side of Fig.4, the visualizations of finer details are small and not clear enough, so it would be more informative to zoom in on local details of the image.
questions: questions: Please see the Weaknesses.
limitations: limitations: The authors adequately addressed the limitations.
paper_summary: paper_summary: The paper introduces CaPaint, a causal structure plugin for spatio-temporal (ST) forecasting, aiming to identify causal regions in data and enable the model to perform causal reasoning. Utilizing a two-stage process and employing a novel image inpainting technique using a fine-tuned unconditional Diffusion Probabilistic Model (DDPM), the paper proposes a method to fill in the gaps identified as environmental parts, enhancing model generalizability and interpretability significantly.
strength_weakness: strength_weakness: - The abstract contains a typo where front-door adjustment is incorrectly referred to as back-door adjustment. 

- The paper appears to lack detailed descriptions on how the generated spatio-temporal data are synthesized into coherent ST sequences, missing crucial details on this aspect of the methodology. 

- The paper does not clearly demonstrate how the quality and efficiency of generation are improved. It is recommended to supplement with additional experiments to substantiate these aspects.
questions: questions: 1. How effective is the inpainting technique implemented by CaPaint specifically on spatio-temporal datasets, and what are the key factors that influence its performance in these contexts? 


2. Why do traditional data augmentation methods, which can disrupt spatio-temporal characteristics, result in performances that are consistent with or only slightly worse than the original, instead of showing a significant decline? 


3. How does the performance compare when augmented data is combined with original data to form the training set for enhancing model generalizability, particularly when controlling for an equal amount of training data?
limitations: limitations: The authors highlight that the effectiveness of the method is limited under conditions of abundant data, as demonstrated through experiments that show more significant performance improvements under data-scarce conditions compared to when data is plentiful.

Comments: comment: Dear Reviewer,

Thank you very much for your considerate feedback and for acknowledging our efforts in addressing the issues raised. We appreciate your kind words regarding the improvements and the detailed explanations provided. Your insights have been invaluable in guiding our revisions, and we are committed to making the final adjustments to further enhance the quality of our paper.

Thank you again for your support and thoughtful evaluation.

Best regards,
Thank you for the detailed explanation and the improved visualization. Their discussion effectively highlights the advantages of their approach, offering a thorough understanding of how causality and interpretability contribute to improved generalization. In addition, the changes on visualitzations will enhance the readability and comprehension of the paper. While the authors have addressed the issues raised in a satisfactory manner, we have decided to withhold our score at this time. We appreciate the effort put into the rebuttal and look forward to seeing the final revisions.
Dear Reviewer,

Thank you for your thoughtful feedback. We sincerely appreciate your constructive suggestions and are grateful for your support throughout this process.

Best regards,
Authors
Thanks for the authors' rebuttal and additional experimental results. I recommend the authors to add the comparison results with Nuwa to the main paper. I have raised my rating accordingly.
Dear Reviewer,

Thank you very much for your thoughtful and positive feedback. We greatly appreciate your recognition of our efforts to address the concerns raised in the initial review. We are delighted that our detailed explanations, additional experimental results, and the innovative application of diffusion inpainting have enhanced your understanding and appreciation of our work.

Your acknowledgment of the improvements we've made means a great deal to us, and we are grateful for your willingness to raise the score based on these enhancements. We will continue to refine and improve our research to contribute to the field.

Thank you again for your careful consideration and support.

Best regards
Dear Authors,

Thank you for your detailed and clear response to the issues we raised. After careful review, we find that your replies are very specific and have adequately clarified the concerns mentioned in our review. In particular, your detailed experimental results and further explanations of the methodology have provided us with a more comprehensive understanding of the paper. Additionally, your selection of error metrics and discussion of different data augmentation methods have deepened our appreciation of the model you proposed. We believe that your response has effectively addressed all the questions we previously raised, and the additional experimental results further enhance the validity and robustness of your method. Moreover, we especially appreciate the innovative application of diffusion inpainting to spatio-temporal video data, which significantly improves the model's performance. Based on these improvements and supplements, we will raise our score for your paper.

Thank you again for your careful attention and detailed responses.

Best regards,
Reviewer