Title: DiffusionPDE: Generative PDE-Solving under Partial Observation
Venue: NeurIPS 2024
Accepted: yes
Keywords: Guided Diffusion Model

Partial Differential Equation

Sparse Observation

Inverse Problem
Abstract: We introduce a general framework for solving partial differential equations (PDEs) using generative diffusion models. In particular, we focus on the scenarios where we do not have the full knowledge of the scene necessary to apply classical solvers. Most existing forward or inverse PDE approaches perform poorly when the observations on the data or the underlying coefficients are incomplete, which is a common assumption for real-world measurements. In this work, we propose DiffusionPDE that can simultaneously fill in the missing information and solve a PDE by modeling the joint distribution of the solution and coefficient spaces. We show that the learned generative priors lead to a versatile framework for accurately solving a wide range of PDEs under partial observation, significantly outperforming the state-of-the-art methods for both forward and inverse directions.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces diffusion methods to tackle the partially observed PDEs, named DiffusionPDE. By learning the joint distribution of solution and coefficient space, the proposed model can handle both forward and inverse problems. The authors experiment with diverse PDEs and settings to demonstrate the model's effectiveness.\n\nSTRENGTHS:\n-\tThis paper successfully utilizes the diffusion methods in solving PDEs, covering both forward and inverse problems.\n\n-\tThe main text and supplementary materials provide diverse experiment settings, which can well support the model’s effectiveness on partial observations.\n\n-\tThis paper is overall clear and well-written.\n\nWEAKNESSES:\n1.\tThe technical contribution is limited.\n\nFrom a technical view, this paper is an application of the diffusion model in PDE solving. There are also some previous methods that also use diffusion methods and leverage the PDE loss [1]. Thus, I think the technical novelty is limited.\n\n[1] A Physics-informed Diffusion Model for High-fidelity Flow Field Reconstruction, JCP 2023\n\n2. Some powerful baselines are missing.\n\n- According to Figure 1, I think the base model of DiffusionPDE is U-Net. How about comparing it with a single U-Net? I think U-Net could be a powerful baseline.\n\n- There are also some latest models that are good at processing partially observed or irregularly placed PDEs, such as OFormer [1] and Transolver [2]. They should include them as baselines.\n\n[1] Transformer for Partial Differential Equations' Operator Learning, TMLR 2023\n\n[2] Transolver: A Fast Transformer Solver for PDEs on General Geometries, ICML 2024\n\n3. Model efficiency comparisons are needed, including GPU memory and running time.\n\n4. I think the proposed model cannot predict the future evolution of a time-dependent PDE. Current tasks are all about “reconstruction” or “imputation”.\n\nQUESTIONS:\n1.\tFigure 4, do both forward and inverse tasks use the same diffusion model? Or do we need to train two models for these two different tasks?\n\n2.\tI think the base model is U-Net. So how does DiffusionPDE handle the spatially scattered partial observations? Is the input still in the regular grid, but only the sampled locations have ground-truth values?\n\nLIMITATIONS:\nI appreciate that they have discussed the limitations. But I think the mentioned issues about efficiency and limitations on temporal modeling are not trivial. More discussions are expected.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes to solve PDEs given only sparse measurements by jointly modeling the solution and coefficient space (e.g. the initial conditions) using a diffusion model. By applying diffusion posterior sampling (DPS) the authors obtain samples that are consistent with the sparse measurements and the underlying PDE equations. Several experiments show superior performance of the method compared to standard baselines such as PINNs and FNOs.\n\nSTRENGTHS:\n- Solving PDEs under partial observation is an important problem in real-world applications\n- The proposed method is technically sound and improves upon existing baseline methods (PINN, FNO) that do not work well for sparse measurements\n- Leveraging a pretrained diffusion model as a generative prior to model the joint distribution of solution and coefficient space is a good idea\n- The presentation of the method is clear and supported by concise algorithms and equations. The paper is well written overall\n- Experiments consider standard baseline methods for PDEs and cover a sufficient range of different dynamics\n\n----\n\nPost-rebuttal: the authors have addressed quite a few of the initial concerns, and while some concerns (e.g. about the magnitude of the contributions remain), I'd be happy to support an accept. I've raised my score accordingly.\n\nWEAKNESSES:\n- The main weakness of the method is the limited novelty. Both sparse measurements and physics-based losses have been considered together with diffusion models, see e.g. Shu et al. (2023). So it seems to me that the main technical novelty is to apply diffusion models to model the joint distribution of two simulation states at different points in time and apply DPS during inference for consistency with the sparse measurements and PDE constraints.     \n- The experiments do not take into account any stochasticity or uncertainty. In principle, DPS will give a distribution of solutions, which is not the case for the other baseline methods, but this is not explored further in the paper. \n- Since the joint distribution models two states at time 0 and time T (for all experiments except Burgers' equation) and $0 \\ll T$, the authors need to simplify the PDE loss $\\mathcal{L}_{pde}$ to drop any time derivatives. This is a serious limitation.   \n- It is not clear if DPS works better than classifier-free guidance, as used e.g. in Shu et al. (2023), or other methods for solving inverse problems with diffusion models. \n- DPS requires a lot of compute during inference for calculating $\\mathcal{L}_{pde}$. For a fair comparison, it would be important to show the number of parameters, training time and inference time for all methods.\n\nQUESTIONS:\n- Algorithm 1 shows an adaption of DPS to EDM (Karras et al. 2022). Is this adaptation novel? Can the authors give some intuition why they apply the DPS losses in line 12 and 13 to the 2nd order correction (line 8) and not apply any trapezoidal rules in this case?  \n- Are sparse measurements located on a grid that matches the resolution of the diffusion model or do they have continuous coordinates? In the second case, how are they interpolated to match the data resolution of the diffusion model? Does that make classifier-free guidance difficult to apply?\n- As noted in the weaknesses: why not use classifier-free guidance? I would like to see a discussion of different methods for inverse problems and diffusion models that can be used here instead of DPS and what are the advantages of using DPS. Reconstructing the solution/coefficient space from sparse measurements alone is a linear inverse problem with a number of different methods that can be used (e.g. Denoising Diffusion Restoration Models; Kawar et al. 2022, among many others) which oftentimes have much nicer theoretical guarantees/higher quality reconstructions and faster sampling speed. When considering these methods, is adding the PDE loss $\\mathcal{L}_{pde}$ and thus making the problem a non-linear inverse problem really beneficial?\n- Likewise, as mentioned above: what are the parameter counts and runtimes of the method and the baselines?\n\nLIMITATIONS:\nThe authors have mentioned slow sampling speed as a limitation in the conclusion, but I think an extended discussion of this would be important to include.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work uses a guided diffusion process to solve the PDE forward and inverse problems with partial observations. Instead of learning the parameter-to-solution map ($a\\rightarrow u$) as in Neural Operators, the method learns the diffusion process on the joint distribution $(a,u)$, and use guided diffusion for inference under sparse observations. Compared with several baseline method, the proposed method shows improved performance for solving forward and inverse problem with sparse observations.\n\nSTRENGTHS:\nThe work uses a guided diffusion process to solve the PDE forward and inverse problems with partial observations.The authors compare with several baseline methods. The idea is clearly presented, and might be useful for the community.\n\nWEAKNESSES:\nThe paper presents an interesting approach to solving PDE forward or inverse problems with sparse observations, which is an appealing concept given the minimal data requirement. However, this approach raises some concerns about the well-posedness of the problem. For example, in forward problems where sparse observations of the parameter $a(x_i)$ are available, there are infinitely many ways to interpolate $a$ and solve the PDE to obtain $u$. They are all valid solutions that satisfy both the PDE and the observations. This suggests that the method's ability to achieve good recovery might heavily rely on the strong regularization imposed by the training dataset, potentially limiting its practical utility as it may only favor solutions resembling those in the training set.\n\nAdditionally, in Appendix C, Table 2, the weightings for observation and PDE loss are significantly higher (by two to six orders of magnitude) than those for $\\nabla_x \\log(p(x))$ as described in Equation 8, which might indicate a predominance of data fitting over the diffusion process. It would be beneficial if the authors could provide more guidance on how these weights were chosen and discuss the implications of using smaller weights. Understanding the rationale behind these choices could help clarify the model's dependency on these parameters and their impact on the solution's behavior.\n\nQUESTIONS:\n(1) The results from the baseline methods (PINO, DeepONet, PINNs, and FNO) are so bad. The claim that these methods are \"not easily extendable\" invites further scrutiny:\n\n(a) All the baseline models are supposed to represent smooth functions. However, in Figure 4, 7, 8, they look discontinuous at the training points.\nAn explanation of how these models were trained and how inference was conducted could clarify why these discrepancies appear.\n\n(b) Taking PINNs as an example in the Darcy flow problem.\nLet $\\hat{a}(x)$ and $\\hat{u}(x)$ be the (potentially noisy) observation at $x$.\nWe can represent $a(x)$ by a neural network, $a_V(x)$ (use neural net for convenience, could be other representations), and the PDE solution $u(x)$ by a neural net $u_W(x)$, where $V$ and $W$ are the weights of the neural network.\nWe can solve the following optimization problem:\n\n$$\\min_{V,W} \\sum_{x\\in T_d} (u_W(x) - \\hat{u}(x))^2 + (a_V(x) - \\hat{a}(x))^2 + \\sum_{x \\in T_r} (\\nabla \\cdot (a_V(x) \\nabla u_W(x)) - q(x))^2$$\n\nwhere $T_d$ is the observational point, and $T_r$ is the residual points, which does not need to be the same as $T_d$.\n\n(c) Similar, for a trained neural operator parametrized by W, $G_W[a] = u$. We can solve the following optimization problem:\n\n$$\\min_V \\sum_{x\\in T_d} (a_V(x) - \\hat{a}(x))^2 + (G\\[a_V\\](x)-\\hat{u}(x))^2$$\n\nwhere $G[a_V]$ is the solution of the PDE with parameter $a_V$.\n\nIt seems that all the baseline methods can be used for forward and inverse problems with sparse observation. \nIt is unclear why the proposed method would offer superior performance compared with the baselines. \n\nIn contexts where full observation are available, as shown in Table 4, one might intuitively expect methods like PINNs—which utilize residual losses to ensure adherence to PDE constraints—or Neural Operators—which establish a direct parameter-to-solution mapping—to deliver more accurate results compared to a method that relies on a diffusion process. This leads to a critical inquiry on why diffusion process gives better accuracy for PDE problems.\n\n(2) How is the PDE loss computed? Is it by finite difference on a regular grid? Detailing this in the main text could help readers assess the accuracy and applicability of the PDE loss in different scenarios.\n\nLIMITATIONS:\nThe author mention several limitation in the conclusion.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper uses score based generative diffusion models to find the forward and backwards solution of a set of PDEs given partial observations of the solution and/or incomplete knowledge of the coefficients. The method performs well, and outperforms other ML  methods such as FNO, as well as 'standard' FE type methods, for a range of standard test problems. The method reconstructs This is a novel approach, which delivers good performance, with low errors  at a competitive speed. Extensive tests are given, with careful analysis of the results.\n\nSTRENGTHS:\nThe use of score based generative methods in this context, where both the solution and the parameter estimates are updated, is novel. The method is clearly effective for the problems considered and should have good applications to real world examples. Extensive tests on a series of standard test problems show that the errors of the method are much lower than other ML based methods such as FNO.\n\nWEAKNESSES:\nThis paper suffers as do many similar papers from a limited range of examples. It concentrates on the usual examples of PDEs such as NS and Bergers, and in both cases of these it looks at problems with quite moderate viscosity, which are realtively east to solve. This is more or less inevitable for such a short paper as this, especially as comparisons are needed with other method. But I would have liked to have seen more novel examples than the usual ones. This is not really a criticism of this paper, but is something to consider for future work. It would be imporoved by a fairer comparison with other methods which work with incomplete data and measurements. A clear exanple of this being the data assimilation widely used in physical modelling for just this range of problems. These should be descibed somewhere in the introduction and in Section 2. (Although of course these latter methods are slow in comparison.) The method is also limited (see later) to looking at certain slices of the solution.\n\nQUESTIONS:\n1. How does this method compare with a data assimilation approach\n2. How easy would it be to extend the method to full time intervals \n3. How easy is it to extend the method to higher dimensions\n4. Have the authors tried out the method on more challenging PDE examples.\n5. Also consider tests on NS and Bergers' eqn with much smaller viscosity.\n\nLIMITATIONS:\nThe model as described only looks at slices of solutions of 2D problems. This has been clearly identified by the authors. In this sense it is vrather limited when compared to other ML based approaches, and of course traditional FE based methods. I am pleased that this is recognised and that the authors plan to address this. The DiffusionPDE method will only be truly competitive when this is done, but this paper is a good step in this direction.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-10"}