Title: Lost in Latent Space: An Empirical Study of Latent Diffusion Models for Physics Emulation
Venue: NeurIPS 2025
Accepted: yes
Keywords: latent

diffusion models

autoencoders

physics

emulation

compression

fluid dynamics

PDE
Abstract: The steep computational cost of diffusion models at inference hinders their use as fast physics emulators. In the context of image and video generation, this computational drawback has been addressed by generating in the latent space of an autoencoder instead of the pixel space. In this work, we investigate whether a similar strategy can be effectively applied to the emulation of dynamical systems and at what cost. We find that the accuracy of latent-space emulation is surprisingly robust to a wide range of compression rates (up to 1000x). We also show that diffusion-based emulators are consistently more accurate than non-generative counterparts and compensate for uncertainty in their predictions with greater diversity. Finally, we cover practical design choices, spanning from architectures to optimizers, that we found critical to train latent-space emulators.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the potential of *latent diffusion models* for 2-dimensional fluids emulations. For the aforementioned problem setting, the authors demonstrate 1) latent space models match or exceed the performance of pixel (raw data) space models and 2) diffusion significantly improves emulation performance compared to deterministic point-regression neural solvers and 3) latent modelling brings considerable speedup at inference time. They also investigate the reconstruction ability of autoencoders for various compression rates.\n\nSTRENGTHS AND WEAKNESSES:\n**Note that I am familiar with diffusion but not physics emulation, so I will only assign low confidence to this review. I also welcome the authors to correct any mistakes/misunderstandings on my part that appear in this review.**\n\n## Strengths\n- The writing is clear, straightfoward and easy to follow/understand (for the most part). \n- The takeaways seem to have clear impact in the field the paper is targeting. Various design choices are explored, leading ultimately to the advantages of the proposed latent diffusion approach being clearly presented. (Although I am personally not familiar with this application).\n## Weaknesses\n1. The paper lacks a clear intuition as to why diffusion is superior for this application (although perhaps it lies in the literature). Could the authors provide such an intuition?\n2. This application seems very similar to video generation, especially considering the data is spatially 2D dimensional and the diffusion model is not trained with any \"physics-informed\" inductive biases. The authors would do better to link the performance of LDMs for video generation to this work. In fact, perhaps answers to why diffusion is effective for physics emulation can be found in the literature for image/video generation ([see this blog post](https://sander.ai/2024/09/02/spectral-autoregression.html]) and its accompanying references).\n3. Although the authors present the result that the compression level of the autoencoder doesn't seem to affect diffusion model performance much, they don't suggest any potential reasons. Could this be due to e.g. *emulation* VRMSE being dominated by lower frequency information (larger blocks of \"colour\") which arises from the diffusion model and not the autoencoder? I would encourage the authors to provide at least an informed suggestion to provide some additional insight to the reader.\n4. **Minor**:\n    1. Compression rate would be better visualised using a colour map that changes in intensity according to the compression value. Figures 5/7 are difficult to interpret currently.\n    2. The use of \"well calibrated\" in conjunction with \"uncertainty\" L249 is a bit confusing, since it refers typically to quite a specific concept in the uncertainty literature. I would suggest using different wording.\n    3. Various equations would benefit from some underbraces for readability e.g. indicating which latents are noised and which are clean in Eq. 7.\n    4. The use of eq 5. seems somewhat arbitrary (although the use of KL regularisation is similarly arbitrary in my experience). Was this based on eyeballing a histrogram of the element values of $z$? Did the authors ablate this choice? If I recall correctly DCAE doesn't use KL regularisation or this tanh-style squashing.\n    4. Figures 1 and 2 are a little unclear. It would be better to clearly separate the emulation timesteps to better illustrate how the masking is applied, as well as which frames serve as conditioning. For example, in Figure 1 it is unclear whether $z^{5:8}$ is conditioned on  $z^{1:4}$ or just $z^{4}$. My understanding is that it is the latter, although I wouldn't be surprised if the former produced better results.\n\nQUESTIONS:\nIn addition to the weaknesses already mentioned (which affect my rating), I have a few questions that will not affect my rating, but I hope that the authors can clarify to me.\n\n1. To what extent are such neural emulators actually useful for current applications? Can you provide an example of a concrete application that would benefit from the trade-off in accuracy for speed?\n2. In the case of low-resolution conditioned generation, would it be possible to use a simulator to predict a spatially-coarse-grained but accurate sequence of states and then upscale it using a diffusion model?\n3. How do neural emulators perform at \"OOD\" generalisation like on $\\theta$ or $i$ not seen during training time? Do they collapse? Time extrapolation seems like an important application but the paper does not even consider it.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose the use of latent diffusion models as emulators for physical systems governed by partial differential equations (PDEs). The primary objective of the paper is to investigate whether compression impacts the emulation capacity. The main finding is that the emulation quality remains high even in high-compression regimes, even when the autoencoder provides poor reconstructions. The authors evaluate their methods across several canonical PDES and provide qualitative and quantitative results.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- I believe that the experiments are well-motivated and extensive. In this regard, the paper's main finding, that good emulation does not require good reconstruction, is important, as it is, at least in my opinion, unexpected. \n- The paper is well written and quite easy to understand, which is great. The numerical experiments/applications are interesting and validate the research question.\n### Weaknesses\n- I believe a major weakness of this approach is the reduced interpretability, as the method does not incorporate any explicit physical knowledge. This issue is crucial when dealing with safety-critical applications, for instance, in medical imaging, where certain applications may lead to physically invalid outputs. \n- This is compounded with some missing baselines, for instance, https://arxiv.org/abs/2305.07671 proposes a PINN hybrid approach it would be good to compare against the physics-informed models counterparts.\n\nQUESTIONS:\n- Why did you not include comparisons to more established neural solvers such as Fourier Neural Operators (FNO), U-Nets, or Physics-Informed Neural Networks (PINNs)? \n\n- It would be good to understand what physical properties are preserved in the latent space. For instance, are the latent variables learning specific quantities, such as velocity, pressure, etc., and are these preserved despite the poor reconstruction?\n\n- Given that the decoder can hallucinate perceptually plausible outputs, how can we be sure that the generated fields are physically meaningful? Have you considered incorporating physics-based constraints or projection steps post-decoding? This also raises the question of how this model performs under different boundary conditions. Do I need to retrain the model, or does it generalize?\n\n- Can you provide an intuitive explanation or a theoretical perspective of why the emulation remains strong despite poor reconstructions?\n\n- Latent diffusion models can be very slow to train. Can you comment on the generalization capabilities of your model and how it would hold up in practice against more traditional alternatives?\n\n- Your model generates statistically correct trajectories. In applications requiring deterministic repeatability (e.g. control), how would you address the stochastic nature of diffusion outputs?\n\nLIMITATIONS:\nSee questions and weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies how physical emulators can exploit a latent space representationn for achieving an effective speedup. For this, the authors focus on two fluiddynamics datasets and compare different autoencoders (with different compression factors) for latent space representation, as well as diffusion models and a traditional neural solver for the prediction. As a baseline, they consider a neural solver in pixel space. The main result of the paper is to show empirically that latent space representations are robust to compression (up to reasonable compression factors) and that diffusion models are effective in reducing the compounding error problem. This study suggest practitioners to adopt latent space representations and diffusion models as backbone for future research in learning-based physical emulators.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is very well written, focused on a single issue, and has extensive numerical comparisons. It tackles an important and timely problem, and tries to do it empirically but with rigor and only simplifications due to the computational requirements needed for such a task. Given the intensive computational effort, sharing these results could possibly benefit the community and sparing people to repeat the same ablation studies.\n\nAlthough I believe the problem is important and I appreciate the focus of the paper, I believe there are also several weaknesses to be addressed:\n\n1. Scope\n\n- Dataset Diversity: Only two 2D fluid‐dynamics datasets are used, insufficient to support claims about physics emulators in general; key regimes (e.g., viscous flows, 3D phenomena) are omitted.\n- Time-Resolution Constraints: The reason for temporal down‐sampling (striding) is unclear (whether it’s a model limitation or training convenience) and it’s unknown if finer time‐steps would break the approach. A thorough empirical validation should nail all these details.\n\n2. Method\n\n- Patch-Size discrepancy: Latent-space tokens use 32×32 patches, while pixel-space tokens use 16×16, with no rationale or ablation study provided.\n- Binary-mask: Introduced without experiments backing up the claimed benefit.\n- Sampling strategy claims: Benefits are asserted but unquantified, with no experiments to back up the sampling scheme’s impact.\n- Omission of pixel-space Diffusion: Only latent-space diffusion is explored; a pixel-space diffusion model baseline is never evaluated.\n\n3. Metrics\n\nLack of downstream-task metrics: The study relies on VRMSE and power-spectrum similarity without assessing application-relevant quantities (e.g., in airfoil, lift/drag), so practical adequacy is unknown.\n\n4. Comparisons and baselines\n\nSpeed–accuracy trade‐off: No positioning of the emulator on the fidelity vs. runtime axis; no comparison to lower‐fidelity, faster simulators or a mechanism (“knob”) to adjust speed vs. accuracy. These are instead readily available in physics simulators.\n\n5. Literature and context\n\nLatent-representation survey: Despite citing latent-space literature, there is no critical engagement or experimental comparison of different representation strategies, leaving the empirical study feeling incomplete.\n\nQUESTIONS:\n1. Can the authors elaborate on why the two datasets presented should be sufficient to present a case for the general family of physics emulators? Or otherwise back up their claims with experiments on additional datasets?\n2. Is the stride in the datasets introduced because of limitations of the methods during training? Is this something avoidable? Or can these models only work with sufficiently coarse time-steps?\n3. Can the authors assess the prediction results with a downstream application metric? E.g., if one is analyzing an airfoil, they might be interested in the resulting lift and drag coefficients. For the datasets you considered, are the predictions accurate enough to be used in downstream applications? This is a much more important metric than VRMSE, power spectrum similarity, etc.\n4. Why did the authors use 32x32 patches per token in the latent space model but 16x16 in the pixel space model?\n5. Why did the authors introduce the binary mask? Can they argue about the benefits and necessity of training in a different scenario of what the downstream application will be? They write a sentence about this yielding benefits, did they run some ablation studies?\n6. They mention the benefits for their sampling strategy: Can they provide evidence? Making these claims without evidence to back up may steer future research in directions not necessarily thoroughly tested.\n7. Can the binary mask train the model for “inpainting”? i.e., filling in between a coarser time-scale?\n8. Why did the authors not consider a pixel-space diffusion model? What would the performances be?\n\nLIMITATIONS:\nIn general, the paper discuss its limitations and does not overpromise, which I appreciate. There are however some points that need some extra careful discussion, in my opinion. I believe that by discussing more openly these limitations would position the paper more strongly.\n\n1. The authors only consider two, 2D, fluiddynamics dataset (This point is mentioned by the author, but I think they should expand on it). They argue these are different, but fluiddynamics phenomena are also very rich. Since the study is empirical, providing sufficient empirical evidence is necessary for backing up the claims presented. Furthermore, the title of the work hints at physics emulation in general. The PDE governing fluiddynamics are general, but e.g. a primarily viscous fluid flow is excluded from the study.\n2. The quality of the prediction is assessed in terms of a few, different numerical metrics which make sense but are hard to evaluate: What does 0.01 VRMSE mean for the simulation considered? The qualitative images shown seem promising, but it is also clear that the models cut off higher frequency components. Are these important in downstream applications? These questions remain unanswered in the paper, and it remains unclear for which types of applications (if not all) it is a good idea to directly focus on latent space prediction without worrying (too much) about compression.\n3. Typically, physics simulator have tradeoff speed-accuracy. It is unclear in the paper where the physics emulator proposed are placing themselves in this axis. How do the performances compare with a lower accuracy, higher speed simulator? Is there a “knob” an user can tune in these physics emulators to reduce speed and improve the accuracy (and vice versa)? These questions remain open, leaving the study somewhat incomplete.\n4. There is a vast literature on latent representations (which the authors rightfully cite). But the paper main contribution is an empirical study of latent space for physics emulation, so not discussing and studying thesed representations seem incomplete.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper compares deterministic neural solvers with latent diffusion models (LDM) for solving partial differential equations (PDEs). Specifically, it compresses 2D image-grid physical simulation data into a latent space using an autoencoder, then reconstructs it through a diffusion model operating within this latent space. The study evaluates performance across varying compression rates and claims that LDM demonstrates robust performance and computational efficiency even at high compression rates.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- Recently, there has been active research applying generative models, especially latent diffusion models, in the field of physics simulation. This paper aligns with that trend by conducting an empirical comparison.\n\n- The systematic experimentation with varying compression rates is meaningful in terms of practical relevance.\n\n- Addressing the trade-off between efficiency and compression through latent space is an important consideration for large-scale simulations.\n\n- The attempt to empirically summarize the potential of LDM-based PDE simulations can be viewed positively.\n\n**Weaknesses**\n- Diffusion-based generative models have already been applied to PDE problems in the literature, so the novelty of this paper is very limited.\n\n- Baseline comparisons are highly restricted; additional comparisons with various state-of-the-art methods beyond deterministic neural solvers are needed.\n\n- The fundamental reasons why LDM exhibits superior performance over deterministic solvers across various compression rates are inadequately analyzed.\n\n- It is unclear what practical impact robustness across different compression rates would have in actual simulation applications.\n\n- The proposed method should be tested beyond simple 2D grid-based experiments, such as on irregular meshes or 3D simulations.\n\n- Further validation might be necessary to confirm whether the efficiency advantages can extend to real-world, large-scale problems, and discussion on this aspect is currently lacking.\n\nQUESTIONS:\nPlease see the above weaknesses.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-07-03"}