Title: Generating Full-field Evolution of Physical Dynamics from Irregular Sparse Observations
Venue: NeurIPS 2025
Accepted: yes
Keywords: tensor decomposition

generative models

physical field reconstruction
Abstract: Modeling and reconstructing multidimensional physical dynamics from sparse and off-grid observations presents a fundamental challenge in scientific research. Recently, diffusion-based generative modeling shows promising potential for physical simulation. However, current approaches typically operate on on-grid data with preset spatiotemporal resolution, but struggle with the sparsely observed and continuous nature of real-world physical dynamics. To fill the gaps, we present SDIFT, Sequential DIffusion in Functional Tucker space, a novel framework that generates full-field evolution of physical dynamics from irregular sparse observations. SDIFT leverages the functional Tucker model as the latent space representer with proven universal approximation property, and represents sparse observations as latent functions and Tucker core sequences. We then construct a sequential diffusion model with temporally augmented UNet in the functional Tucker space, denoising noise drawn from a Gaussian process to generate the sequence of core tensors.
  At the posterior sampling stage, we propose a  Message-Passing Posterior Sampling mechanism, enabling conditional generation of the entire sequence guided by observations at limited time steps. We validate SDIFT on three physical systems spanning astronomical (supernova explosions, light-year scale), environmental (ocean sound speed fields, kilometer scale), and molecular (organic liquid, millimeter scale) domains, demonstrating significant improvements in both reconstruction accuracy and computational efficiency compared to state-of-the-art approaches.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a latent diffusion model for spatiotemporal data, where the latent representation of each temporal snapshot follows a Tucker decomposition, and diffusion learns the distribution of Tucker factors. The paper proposes message-passing to denoise temporally consistent frames. The method works very well on interpolating frames from very sparse data.\n\nSTRENGTHS AND WEAKNESSES:\nOriginality good. The paper combines existing ideas, but the use of Tucker as diffusion latent representation is innovative and a good match for the domain problem. \n\nSignificance mixed. The scope of the paper is physics-based, low-resolution spatiotemporal dynamics, where some frames are missing and remaining ones have 1-3% of the pixels observed. The task is to impute the field. The task is interesting and is a great stress-test challenge for the methods. However, the task is also very specific, and the paper does not demonstrate why it’s a significant or realistic application. I'm not sure if the paper demonstrates \"real-world\" usecases. The paper does not discuss if the contributions have more general ML relevance.  \n\nQuality mixed. The method has good and consistent performance. The method is well-motivated and sound. The main issues of quality are in weak clarity. \n\nClarity weak. The method is relatively simple (denoising latent factors), but the notation is overly complex. The paper defines a datapoint as (i_n_m, t_m, y_i_n_m), and most parts of the paper inherit this complexity which makes the paper hard to read and digest. The K-mode notation seems unnecessary since all experiments are in 3D or 2D. The motivation, open problem and setting are vague [is this a PDE paper? What is “off-grid”? Do we interpolate or extrapolate or forecast?], and the paper does not clearly position itself to the literature. The message-passing sections are borderline unreadable to the the complicated notation. The method is a black-box, and little insight is given how the model denoises, or why (visualisations would help). There are no ablations showing how the different terms improve (Tucker vs other latent representations; message-passing vs other temporal consistency guidance methods, etc). The paper needs to dramatically improve it’s notation, and borrow good practises from exemplar diffusion papers.  \n\nOverall the paper is an ok technical report showing a succesful method, but as a scientific paper it needs to demonstrate the ideas and significance in wider ML context, and improve presentation.\n\nQUESTIONS:\n- The role of modes could be clarified. The paper starts with talking about K modes, which I think are just the spatial coordinates (ie. x,y,z in 3D data). Do we ever have K>3?\n- Is time one of the modes? Why or why not?\n- What does “solving” eq 6 mean in line 149? One needs here both the W and f’s: are both optimised? How do you optimise? Having an algorithm box would be helpful.\n- The remark about VAE is rather strange: I don’t see anything variational, stochastic or Bayesian around eq 6.\n- Line 163 claims interpretable cores. I don’t see any demonstration of this: how are the cores interpretable?\n- What is “alternating-direction strategy”?\n- It seems that one only learns one scalar function per dimension (line 170). This feels quite limited: I wonder what the f-core is then learning or representing? What do the f-cores look like in practise?\n- Is learning of W and f identifiable? One would expect that some rotational invariances would exist. Is their learning sensitive to initialisation? In line 176 the paper assumes that W_i equals W(t_i), but I don’t think you can state this unless the encoding of W_i is deterministic, which requires identifiability and invariance to initial values. In practise the encoding seems stochastic, and one would then have some stochasticity in the W_i’s you get. I also wonder what “drawn from a continuous function” means. We draw samples from a distribution, but evaluate a function. Is there some stochastic process interpretation here? Can you discuss?\n- In line 178 the W(t) is remarked as distribution. Why is it a distribution? \n- Can you clarify the roles of s and t in eq 9. If t is an observation time, then why not also integrate over observation locations?\n- I’m a bit confused what \\Eps or \\W look like: what are their dimensions? Line 184 says that \\Eps is only a function of time, but it is added to \\W which only contains spatial stuff. Is the idea that Eps-set is kept fixed? Or is there some kind of Eps-minibatching?\n- Line 200 says that we can sample using eq 9 model for any (i,t). Does eq 9 contain i?\n- In line 204 it is stated that observations are a subset of target times. Why do we want to predict target times if we already observe them? Can you clarify what is test and train?\n- I can’t follow eqs 10-12\n- How does the train/test split work in experiments? Do you do interpolation or forecasting? How does this relate to the Tar/obs splitting? It seems that half of target is obs, but 95% of frames are train and 5% are test. I’m not sure I understand what does target/test/obs/train mean.\n- How do you use the model at training and at testing stage?\n\nLIMITATIONS:\nNo issues", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles an important and practically relevant problem—reconstructing full-field spatiotemporal physical dynamics from irregular and sparse observations. The proposed SDIFT framework is methodologically innovative and theoretically sound. It builds on the Functional Tucker Model as a continuous latent representation with proven universal approximation capability, and further integrates a Gaussian process–based diffusion mechanism and a novel message-passing posterior sampling strategy. The technical contributions are solid, well-motivated, and clearly described, with thorough empirical validation across diverse physical systems. Overall, the paper offers a compelling and elegant solution to a challenging generative modeling task.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. Challenging and important problem setting: The paper tackles the highly non-trivial task of reconstructing full-field physical dynamics from sparse, irregular, and off-grid observations. Unlike many existing approaches that require access to dense spatiotemporal trajectories for pretraining, SDIFT is capable of learning directly from sparse observations, making it more practical and broadly applicable in real-world scenarios.\n2. Theoretical soundness: The proposed framework is supported by a solid theoretical foundation. In particular, the authors rigorously prove the universal approximation property of the Functional Tucker Model (FTM).\n3. Comprehensive experiments: The paper presents extensive experimental validation across three distinct physical systems (supernova explosions, ocean sound speed, and active matter), spanning different spatial and temporal scales. \n\nWeaknesses:\n\nI acknowledge that I am not deeply familiar with the literature on tensor-based representations for physical field modeling, so I may have missed potential limitations in that aspect of the method. From my perspective, the overall design appears reasonable and well-motivated. I have some questions about the framework which are left to the \"Questions\" section.\n\nQUESTIONS:\n1. Since the method does not require pretraining on densely sampled dynamics, I am curious about its behavior under extremely sparse observation conditions. Specifically:\n\n- As the number of observations decreases, is there a threshold below which the model fails to generate meaningful reconstructions due to a lack of information?\n- How does the model perform under non-uniform observation patterns, such as those common in weather-related applications, where sensors are densely located in developed regions but sparse over oceans or remote areas? This setting might challenge the assumption of homogeneity in sampling patterns.\n\n2. In some domain-specific applications, methods have been proposed to perform field reconstruction without pretraining. For example, [1] uses a learnable AI model to represent the evolution process and directly minimizes a loss function that enforces consistency between the predicted and observed values. This formulation seems more amenable to incorporating physical constraints or domain priors. Could you clarify the main advantages of the method compared to such approaches, particularly in terms of flexibility, generalizability, or computational efficiency?\n\n[1] Fablet, Ronan, Bertrand Chapron, Lucas Drumetz, Etienne Mémin, Olivier Pannekoucke, and François Rousseau. \"Learning variational data assimilation models and solvers.\" *Journal of Advances in Modeling Earth Systems* 13, no. 10 (2021): e2021MS002572.\n\nLIMITATIONS:\nYes, the authors have discussed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents SDIFT (Sequential Diffusion in Functional Tucker space), a new generative framework for reconstructing multidimensional physical dynamics from irregular and sparse spatiotemporal observations. The key innovation is the use of a functional Tucker model for latent space representation, combined with a Gaussian Process-based sequential diffusion to generate core sequences, and a Message-Passing Posterior Sampling mechanism (MPDPS) for effective conditional sequence generation. Theoretical foundation is added via a universal approximation guarantee for the functional Tucker model. The method is evaluated on three diverse real-world physical datasets (astronomical, oceanographic, and molecular), showing significant improvements in accuracy and computational efficiency over state-of-the-art baseline approaches.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n- The experimental evaluation is thorough, spanning three distinctly different domains (supernova, ocean sound speed, active matter) and comparing against a diverse set of relevant baselines. The method demonstrates consistent quantitative improvements, as detailed in Table 1 (Section 5): SDIFT with MPDPS achieves lowest VRMSE scores across datasets and observation scenarios, often with large gaps to other methods.\n\n- The figures provide clear qualitative evidence: Figure 3 and Figure 4 demonstrate that SDIFT with MPDPS yields visually superior reconstructions at both dense and missing observation times, compared to competing approaches. In particular, Figure 4 highlights the advantage of the message-passing mechanism in interpolating missing temporal slices.\n\n**Weaknesses:**\n\n- The comparison with baselines is not comprehensive. Although the authors compare parameter size and runtime in Table 2, this comparison is limited to CoNFiLD only and does not include other baselines. I am particularly curious about how the proposed method performs against the other reported results.\n\n- The reviewer is confused about the content in Section 3.1. In this section, the authors propose the Functional Tucker Model (FTM) as a universal latent representer. For Equation (6), let us consider a two-dimensional scenario and even ignore the temporal index $t$, assuming $y(i_x, i_y, t) = e^{i_x \\cdot i_y}$. The reviewer would like the authors to provide finite-dimensional solutions for $f^1_{\\theta_1}(i_x)$ and $f^2_{\\theta_2}(i_y)$ in this example.\n\n- Building upon the previous concern, for the case $y(i_x, i_y, t) = e^{i_x \\cdot i_y}$, the reviewer argues that only an approximate solution can be obtained under finite-dimensional settings. This raises a new issue: in the proof of universal approximation, the authors must account for the case where $R^k \\to \\infty$. Therefore, the reviewer is confused by the last step of Equation (24) on page 15.\n\n- There is a typographical error in the formula on page 14, Equation (18). In the $B_2$ component, $f_{r_1}^{A_1}$ should be $f_{r_1}^{k}$.\n\n- Minor issue: references [28] and [38] refer to the same publication.\n\nQUESTIONS:\nSee weekness.\n\nLIMITATIONS:\nSee weekness.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present Sequential Diffusion in Functional Tucker Space (SDIFT). They deal with the problem of posterior sampling of a continuous-time real-vector valued sequence (like solutions of a PDE). Motivated by latent space diffusion, they do posterior sampling given partial observations of the whole continuous time signal by doing diffusion modeling and posterior sampling in the latent space of a functional tucker tensor decomposition. In the work they deal with two main challenges (1) define the latent space according to the tucker decomposition in order to handle sparse/off-time-grid observations and (2) perform inference using likewise incomplete observations to generate a full continuous time field.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- precise mathematical formalization of the problem at hand\n- moving between typical VAE latent space to something more structured that is used in the applied math / numerical community, but mixing with strengths of ML + diffusion. The time-varying core tensors $W_{t_k}$ of the tucker decomposition are taken as the latent representation so that for data spatial index $i_k$ (very loosely) we have$y_{i,t} = w_t f^k_\\theta(i_k)$ for learnable $f^k_\\theta$ and learnable $w_t$ for mode $k$ . The cores $w_t$ share info across the $K$ modes' latent functions  \n- the latent space $W(t)$ is still at time varying function, so they use functional diffusion models to model it (functional diffusion model has GP noise)\n- the diffusion pre-training and posterior inference algorithms are separated into two clean pieces (where the first learning piece itself has two parts: learn the decomposition, learn the diffusion)\n- novel message passing algorithm to perform the inference\n- convincing experiments on real scientific tasks (1) supernova explosion temperature evolution (2) pacific ocean sound speed (3) active particles in simulated fluid with comprehensive description/links for more details to all data details.\n\nWeaknesses\n\nSee questions.\n\nQUESTIONS:\n- in section 3 124, for the diffusion prior pre-training phase, what's the definition of homogenous in \"B batches of homogeneous dynamics\" mean?\n\n- in section 3.1, line 166, in equation 8 , do you need a coefficient in front of the TV term in practice? If so, how do you choose it? \n\n- section 3.2, line 192, equation 9, there is a denoiser-matching objective to train the diffusion model for fixed W.  Have you tried training (8) and (9) simultaneously rather than in pieces. Any +/-'s to report?\n\n- does temporally augmented Unet just mean that the Unet gets the noised $(W_t)^{s}$ (W for data time t at diffusion time s) as well the times $t=[t_1,...t_M]$ themselves? If not, what does it mean?\n\n- *Main question*: What is proved that the specific message passing algorithm introduced, which uses $\\nabla_{W^s_{\\setminus\nt_\\ell}} \\log p(O_{t_\\ell} | W^s_{\\setminus\nt_\\ell})$ ? I get intuitively that we are passing some info from the observed to the rest, but abstracting a bit, how can I see more easily that updating the whole $W_t$ according to these gradients produces a sample of $W_t^{\\text{all}} | W_t^{\\text{obs}}$ ?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-04"}