DiffusionPDE: Generative PDE-Solving under Partial Observation

Authors: jiahe  huang
guandao  yang
zichen  wang
jeong joon park
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 22:19:26

Abstract: We introduce a general framework for solving partial differential equations (PDEs) using generative diffusion models. In particular, we focus on the scenarios where we do not have the full knowledge of the scene necessary to apply classical solvers. Most existing forward or inverse PDE approaches perform poorly when the observations on the data or the underlying coefficients are incomplete, which is a common assumption for real-world measurements. In this work, we propose DiffusionPDE that can simultaneously fill in the missing information and solve a PDE by modeling the joint distribution of the solution and coefficient spaces. We show that the learned generative priors lead to a versatile framework for accurately solving a wide range of PDEs under partial observation, significantly outperforming the state-of-the-art methods for both forward and inverse directions.

Main paper: 
 introduction: 
 Partial differential equations (PDEs) are a cornerstone of modern science, underpinning many contemporary physical theories that explain natural phenomena. The ability to solve PDEs grants us the power to predict future states of a system (forward process) and estimate underlying physical properties from state measurements (inverse process).To date, numerous methods [1, 2] have been proposed to numerically solve PDEs for both the forward and inverse directions. However, the classical methods can be prohibitively slow, prompting the development of data-driven, learning-based solvers that are significantly faster and capable of handling a family of PDEs. These learning-based approaches [3] [4] [5] [6] typically learn a deterministic mapping between input coefficients and their solutions using deep neural networks.Despite the progress, existing learning-based approaches, much like classical solvers, rely on complete observations of the coefficients to map solutions. However, complete information on the underlying physical properties or the state of a system is rarely accessible; in reality, most measurements are sparse in space and time. Both classical solvers and the state-of-the-art data-driven models often overlook these scenarios and consequently fail when confronted with partial observations. This limitation confines their use primarily to synthetic simulations, where full scene configurations are available by design, making their application to real-world cases challenging.We present a comprehensive framework, DiffusionPDE, for solving PDEs in both forward and inverse directions under conditions of highly partial observations-typically just 1~3% of the total information. This task is particularly challenging due to the numerous possible ways to complete missing data and find subsequent solutions. Our approach uses a generative model to formulate the joint distribution of the coefficient and solution spaces, effectively managing the uncertainty and simultaneously reconstructing both spaces. During inference, we sample random noise and iteratively 38th Conference on Neural Information Processing Systems (NeurIPS 2024). denoise it following standard diffusion models [7] . However, we uniquely guide this denoising process with sparse observations and relevant PDE constraints, generating plausible outputs that adhere to the imposed constraints. Notably, DiffusionPDE can handle observations with arbitrary density and patterns with a single pre-trained generative network.We conduct extensive experiments to show the versatility of DiffusionPDE as a general PDE-solving framework. We evaluate it on a diverse set of static and temporal PDEs, including Darcy Flow, Poisson, Helmholtz, Burger's, and Navier-Stokes equations. DiffusionPDE significantly outperforms existing state-of-the-art learning-based methods for solving PDEs [3] [4] [5] [6] 8] in both forward and inverse directions with sparse measurements, while achieving comparable results with full observations. Highlighting the effectiveness of our model, DiffusionPDE accurately reconstructs the complete state of Burgers' equation using time-series data from just five sensors (Fig. 4 ), suggesting the potential of generative models to revolutionize physical modeling in real-world applications. 
related works: 
 Our work builds on the extensive literature of three areas: forward PDE solvers, inverse PDE solvers, and diffusion models. Please see relevant surveys for more information [9] [10] [11] [12] [13] .Forward PDE Solvers. PDE solvers take the specification of a physics system and predict its state in unseen space and time by solving an equation involving partial derivatives. Since Most PDEs are very challenging to solve analytically, people resolve to numerical techniques, such as Finite Element Method [14, 2] and Boundary Element Method [1, 15] . While these techniques show strong performance and versatility in some problems, they can be computationally expensive or difficult to set up for complex physics systems. Recently, advancements in deep-learning methods have inspired a new set of PDE solvers. Raissi et al. [16, 6] introduce Physics-Informed Neural Networks (PINNs), which optimize a neural network using PDE constraints as self-supervised losses to output the PDE solutions. PINNs have been extended to solving specific fluid [17, 18] , Reynolds-averaged Navier-Stokes equations [19] , heat equations [20] , and dynamic power systems [21] . While PINNs can tackle a wide range of complex PDE problems, they are difficult to scale due to the need for network optimization. An alternative approach, neural operators [3, 5] , directly learn the mapping from PDE parameters (e.g.initial and boundary condition) to the solution function. Once trained, this method avoids expensive network optimization and can instantly output the solution result. This idea has been extended to solve PDE in 3D [22, 23] , multiphase flow [24] , seismic wave [25, 26] , 3D turbulence [27, 28] , and spherical dynamics [29] . People have also explored using neural networks as part of the PDE solver, such as compressing the physics state [30] [31] [32] [33] . These solvers usually assume known PDE parameters, and applying them to solve the inverse problem can be challenging. PDE inverse problem. The inverse problem refers to finding the coefficients of a PDE that can induce certain observations, mapping from the solution of a PDE solver to its input parameters.People have tried to extend traditional numerical methods to this inverse problem [34] [35] [36] [37] [38] , but these extensions are non-trivial to implement efficiently. There are similar attempts to inverse deep-learning PDE solvers. For example, one can inverse PINNs by optimizing the network parameters such that their outputs satisfy both the observed data and the governing equations. iFNO [39] and NIO [40] tries to extend FNO [3] . Other methods [41, 42] directly learn the operator functions for the inverse problem. PINO [4] further combines neural operators with physics constraints to improve the performance of both forward and inverse problems. These methods assume full observations are available. To address the inverse problem with partial observations, people have tried to leverage generative priors with Graph neural networks [43, 8] . These works have not demonstrated the ability to solve high-resolution PDEs, possibly limited by the power of generative prior. We want to leverage the state-of-the-art generative model, diffusion models, to develop a better inverse PDE solver. Diffusion models. Diffusion models have shown great promise in learning the prior with higher resolutions by progressively estimating and removing noise. Models like DDIM [44] , DDPM [7] , and EDM [45] offer expressive generative capabilities but face challenges when sampling with specific constraints. Guided diffusion models [46] [47] [48] [49] enhance generation processes with constraints such as image inpainting, providing more stable and accurate solutions. Prior works on diffusion models for PDEs highlight the potential of diffusion approaches by generating PDE datasets such as 3D turbulence [50, 51] and Navier-Stokes equations [52] with diffusion models. Diffusion models can also be used to model frequency spectrum and denoise the solution space [53] , and conditional diffusion models are applied to solve 2D flows with sparse observation [54] . However, the application of diffusion models to solve inverse problems under partial observation remains underexplored. In this work, we aim to take the initial steps towards addressing this gap. 
overview: 
 To solve physics-informed forward and inverse problems under uncertainty, we start by pre-training a diffusion generative model on a family of partial differential equations (PDEs). This model is designed to learn the joint distribution of the PDE coefficients (or the initial state) and its corresponding solutions (or the final state). Our approach involves recovering full data in both spaces using sparse observations from either or both sides. We achieve this through the iterative denoising of random Gaussian noise as in regular diffusion models but with additional guidance from the sparse observations and the PDE function enforced during denoising. The schematic description of our approach is shown in Fig. 1 .Diffusion models involve a predefined forward process that gradually adds Gaussian noise to the data and a learned reverse process that denoises the data to reconstruct the original distribution. Specifically, Song et al. [55] propose a deterministic diffusion model that learns an N -step denoising process that eventually outputs a denoised data x N and satisfies the following ordinary differential equations (ODE) at each timestep t i where i ∈ {0, 1, ..., N -EQUATIONHere ∇ x log p x; σ(t) is the score function [56] that helps to transform samples from a normal distribution N (0, σ(t 0 ) 2 I) to a target probability distribution p(x; σ(t)). To estimate the score function, Karras et al. [45] propose to learn a denoiser function D(x; σ) such thatEQUATIONTo enable control over the generated data, guided diffusion methods [48] add guidance gradients to the score function during the denoising process. Recently, diffusion posterior sampling (DPS) [46] Algorithm 1 Sparse Observation and PDE Guided Diffusion Sampling Algorithm. xi N ← D θ (x i ; σ(t i ))▷ Estimate the denoised data at step t i 5:d i ← x i -xi N /σ(t i )▷ Evaluate dx/dσ(t) at step t i 6:x i+1 ← x i + (σ(t i+1 ) -σ(t i ))d i ▷ Take an Euler step from σ(t i ) to σ(t i+1 ) 7: if σ(t i+1 ) ̸ = 0 then 8: xi N ← D θ (x i+1 ; σ(t i+1 ))▷ Apply 2 nd order correlation unless σ = 0 9:d ′ i ← x i+1 -xi N /σ(t i+1 )▷ Evaluate dx/dσ(t) at step t i+1 10: x i+1 ← x i + (σ(t i+1 ) -σ(t i )) 1 2 d i + 1 2 d ′ i ▷L pde ← 1 m ∥0 -f (x i N )∥ 2 2▷ Evaluate the PDE loss of xi N 14: x i+1 ← x i+1 -ζ obs ∇ xi L obs -ζ pde ∇ xi L pde ▷ GuideDPS [46] showed that under Gaussian noise assumption of the sparse measurement operator M(•), i.e., y|x ∼ N (M(x), δ 2 I) with some S.D. δ, the log-likelihood function can be approximated with:EQUATIONwhere xi N := D(x i ; σ(t i )) denotes the estimation of the final denoised data at each denoising step i. Applying the Baye's rule, the gradient direction of the guided diffusion is therefore:EQUATIONwhere s(x) = ∇ x log p x is the original score function, and ζ = 1/δ 2 .Our work focuses on two classes of PDEs: static PDEs and dynamic time-dependent PDEs. Static systems (e.g., Darcy Flow or Poisson equations) are defined by a time-independent function f :EQUATIONwhere Ω is a bounded domain, c ∈ Ω is a spatial coordinate, a ∈ A is the PDE coefficient field, and u ∈ U is the solution field. ∂Ω is the boundary of the domain Ω and u| ∂Ω = g is the boundary constraint. We aim to recover both a and u from sparse observations on either a or u or both.Similarly, we consider the dynamic systems (e.g., Navier-Stokes):EQUATIONwhere τ is a temporal coordinate, a = u 0 ∈ A is the initial condition, u is the solution field, and u| Ω×(0,∞) = g is the boundary constraint. We aim to simultaneously recover both a and the solution u T := u(•, T ) at a specific time T from sparse observations on either a, u T , or both.Finally, we explore the recovery of the states across all timesteps u 0:T in 1D dynamic systems governed by Burger's equation. Our network D θ models the distribution of all 1D states, including the initial condition u 0 and solutions u 1:T stacked in the temporal dimension, forming a 2D dataset. Compared with the ground truth, we see that our method successfully recovers the PDE in all cases.Guided Diffusion Algorithm In the data-driven PDE literature, the above tasks can be achieved by learning directional mappings between a and u (or u T for dynamic systems). Thus, existing methods typically train separate neural networks for the forward solution operator F : A → U and the inverse solution operator I : U → A.Our method unifies the forward and inverse operators with a single network and an algorithm using the guided diffusion framework. DiffusionPDE can handle arbitrary sparsity patterns with one pre-trained diffusion model D θ that learns the joint distribution of A and U, concatenated on the channel dimension, denoted X . Thus, our data x ∈ X , where X := A × U. We follow the typical diffusion model procedures [45] to train our model on a family of PDEs.Once we train the diffusion model D θ , we employ our physics-informed DPS [46] formulation during inference to guide the sampling of x ∈ X that satisfies the sparse observations and the given PDE, as detailed in Algorithm 1. We follow Eq. 5 to modify the score function using the two guidance terms:EQUATIONwhere x i is the noisy data at denoising step i, y obs are the observed values, and f (•) = 0 is the underlying PDE condition. L obs and L pde respectively represent the MSE loss of the sparse observations and the PDE equation residuals:EQUATIONwhere xi N = D θ (x i ) is the clean image estimate at denoising timestep i, which can be split into coefficient ûi and solution âi . Here, m is the total number of grid points (i.e., pixels), n is the number of sparse observation points. o j represents the spatio-temporal coordinate of jth observation. Note that, without loss of generality, L pde can be accumulated for all applicable PDE function f in the system, and the time component τ k is ignored for static systems. 
pde problem settings: 
 We show the usefulness of DiffusionPDE across various PDEs for inverse and forward problems and compare it against recent learning-based techniques. We test on the following families of PDEs. EQUATIONHere the coefficient a has binary values. We set q(c) = 1 for constant force. The PDE guidance function is thus f = ∇ • (a(c)∇u(c)) + q(c). Inhomogeneous Helmholtz Equation. We consider the static inhomogeneous Helmholtz Equation with a no-slip boundary on ∂Ω, which describes wave propagation:EQUATIONThe coefficient a is a piecewise constant function and k is a constant. Note 11 is the Poisson equation when k = 0. Setting k = 1 for Helmholtz equations, the PDE guidance function isf = ∇ 2 u(c) + k 2 u(c) -a(c).Non-bounded Navier-Stokes Equation. We study the non-bounded incompressive Navier-Stokes equation regarding the vorticity.EQUATIONHere w = ∇ × v is the vorticity, v(c, τ ) is the velocity at c at time τ , and q(c) is a force field. We set the viscosity coefficient ν = 10 -3 and correspondingly the Reynolds number Re = 1 ν = 1000. DiffusionPDE learns the joint distribution of w 0 and w T and we take T = 10 which simulates 1 second. Since T ≫ 0, we cannot accurately compute the PDE loss from our model outputs. Therefore, given that ∇• w(c, τ ) = ∇ • (∇ × v) = 0, we use simplified f = ∇ • w(c, τ ).Bounded Navier-Stokes Equation. We study the bounded 2D imcompressive Navier Stokes regarding the velocity v and pressure p.EQUATIONWe set the viscosity coefficient ν = 0.001 and the fluid density ρ = 1.0. We generate 2D cylinders of random radius at random positions inside the grid. Random turbulence flows in from the top of the grid, with the velocity field satisfying no-slip boundary conditions at the left and right edges, as well as around the cylinder ∂Ω lef t,right,cylinder . DiffusionPDE learns the joint distribution of v 0 and v T at T = 4, which simulates 0.4 seconds. Therefore, we similarly use f = ∇ • v(c, τ ) as before.Burgers' Equation. We study the Burgers' equation with periodic boundary conditions on a 1D spatial domain of unit length Ω = (0, 1). We set the viscosity to ν = 0.01. In our experiment, the initial condition u 0 has a shape of 128 × 1, and we take 127 more time steps after the initial state to form a 2D u 0:T of size 128 × 128.EQUATIONWe can reliably computef = ∂ t u(c, τ ) + ∂ c (u 2 (c, τ )/2) -ν∂ cc u(c, τ) with finite difference since we model densely on the time dimension. We first test DiffusionPDE on jointly learning the forward mapping F : A → U and the inverse mapping I : U → A given sparse observations. In our experiments, we define our PDE over the unit square Ω = (0, 1) 2 , which we represent as a 128 × 128 grid. We utilize Finite Element Methods (FEM) to generate our training data. Specifically, we run FNO's [3] released scripts to generate Darcy Flows and the vorticities of the Navier-Stokes equation. Similarly, we generate the dataset of Poisson and Helmholtz using second-order finite difference schemes. To add more complex boundary conditions, we use Difftaichi [57] to generate the velocities of the bounded Navier-Stokes equation.We train the joint diffusion model for each PDE on three A40 GPUs for approximately 4 hours, using 50,000 data pairs. For Burgers' equation, we train the diffusion model on a dataset of 50,000 samples produced as outlined in FNO [3] . We randomly select 5 out of 128 spatial points on Ω to simulate sensors that provide measurements across time.We compare DiffusionPDE with state-of-the-art learning-based methods, including PINO [4] , Deep-ONet [5] , PINNs [6] , and FNO [3] . However, note that none of these methods show operation on partial observations. These methods can learn mappings between a and u or u 0 and u 1:T with full observations, allowing them to also solve the mapping between u 0 and u T . PINNs map input a to output u by optimizing a combined loss function that incorporates both the solution u and the PDE residuals. DeepONet employs a branch network to encode input function values sampled at discrete points and a trunk network to handle the coordinates of the evaluated outputs. FNO maps from the parametric space to the solution space using Fourier transforms. PINO enhances FNO by integrating PDE loss during training and refining the model with PDE loss finetuning. We train all four baseline methods on both forward and inverse mappings using full observation of a or u for both static and dynamic PDEs. We tried training the baseline models on partial observations, but we noticed degenerate training outcomes (see supplementary for details). Overall, they are intended for full observations and may not be suitable for sparse measurements.More closely related to our method, GraphPDE [8] demonstrates the ability to recover the initial state using sparse observations on the final state, a task that other baselines struggle with. Therefore, we compare against GraphPDE for the inverse problem of bounded Navier-Stokes (NS) equation, which is the setup used in their report. GraphPDE uses a trained latent space model and a bounded forward GNN model to solve the inverse problem with sparse sensors and thus is incompatible with unbounded Navier-Stokes. We create bounded meshes using our bounded grids to train the GNN model and train the latent prior with v 0:T for GraphPDE.While we employ guided sampling to reconstruct the solutions, Classifier-Free Guidance (CFG) [58] offers an alternative approach where the diffusion model is conditioned on sparse input data. Shu et al. [54] extend this method by developing an optimized CFG approach that conditions on the PDE loss, In the inverse set-up, we take observations of the final state and solve for the initial. For the Burgers' equation, we use 5 sensors throughout all time steps and want to recover the solution at all time steps. Note that we train on neighboring snapshot pairs for the baselines in order to add continuous observations of the Burgers' equation. Results show that existing methods do not support PDE solving under sparse observations, and we believe they are not easily extendable to do so. We refer readers to the supplementary for a complete set of visual results.using the observation as a low-resolution input. Additionally, OFormer [59] is another model designed to reconstruct the full solution using transformers, offering a shorter inference runtime. Consequently, we compare our approach against these methods for solving the unbounded Navier-Stokes equation.We respectively address the forward problem and the inverse problem with sparse observations of a or u. For the forward problem, we randomly select coefficients (initial states) as sparse observations and then compare the predicted solutions (final states) with the ground truth. Similarly, for the inverse problem, we randomly sample points on solutions (final states) as sparse observations, using the same number of observed points as in the forward model for each PDE.We show the relative errors of all methods regarding both forward and inverse problems in Table 1 . Since the coefficients of Darcy Flow are binary, we evaluate the error rates of our prediction. Non-binary data is evaluated using mean pixel-wise relative error. We report error numbers averaged across 1,000 random scenes and observations for each PDE. DiffusionPDE outperforms other methods including PINO [4] , DeepONet [5] , PINNs [6] , and FNO [3] for both directions with sparse observations, demonstrating the novelty and uniqueness of our approach. For the inverse problems of the Poisson and Helmholtz equations, DiffusionPDE exhibits higher error rates due to the insufficient constraints within the coefficient space, produced from random fields. In Fig. 4 , we visualize the results for solving both the forward and inverse problem of the non-bounded Navier-Stokes. We refer to the supplementary for additional visual results. While other methods may produce partially correct results, DiffusionPDE outperforms them and can recover results very close to the ground truth.For the inverse problem of the bounded Navier-Stokes equation, we further compare DiffusionPDE with GraphPDE, as illustrated in Fig. 5 . Our findings reveal that DiffusionPDE surpasses GraphPDE [8] in accuracy, reducing the relative error from 12.0% to 2.7% with only 1% observed points.We further show whether DiffusionPDE can jointly recover both a and u by analyzing the retrieved a and u with sparse observations on different sides as well as on both sides. In Fig. 2 , we recover the coefficients and solutions of Darcy Flow by randomly observing 500 points on only coefficient space, only space solution space, and both. Both coefficients and solutions can be recovered with low errors for each situation. We therefore conclude that DiffusionPDE can solve the forward problem and the inverse problem simultaneously with sparse observations at any side without retraining our network.To demonstrate the clear advantage of our guided sampling method, we evaluate both the forward and inverse processes of the unbounded Navier-Stokes equation, comparing our DiffusionPDE approach with Diffusion using CFG when considering only the initial and final states given 500 observation points, as illustrated in Fig. 6 . Our DiffusionPDE method consistently achieves lower relative errors across both evaluations. Furthermore, in Fig. 7 , we compare our results with those of Shu et al. [54] , where the full time intervals are solved autoregressively using an optimized CFG method. In their approach, the error in the final state increases to approximately 13%, which is notably higher than that of our two-state model. Additionally, the relative errors of the transformer-based approach, OFormer [59] , are around 17% and 23%, which are significantly larger than those observed with DiffusionPDE.We demonstrate that DiffusionPDE is capable of retrieving all time steps throughout the time interval [0, T ] from continuous observations on sparse sensors. To evaluate its ability to recover u 0:T with sparse sensors, we study the 1D dynamic Burgers' equation, where DiffusionPDE learns the distribution of u 0:T using a 2D diffusion model. To apply continuous observation on PINO, DeepONet, FNO, and PINNs, we train them on neighboring snapshot pairs. Our experiment results in a test relative error of 2.68%, depicted in Fig. 4 , which is significantly lower than other methods.We examine the effects of different components of our algorithm such as PDE loss and observation samplings. We strongly encourage readers to view the supplementary for more details of these analyses as well as additional experiments. PDE Loss. To verify the role of the PDE guidance loss of Eq. 8 during the denoising process, we visualize the errors of recovered a and u of Helmholtz equation with or without PDE loss. Here, we run our DPS algorithm with 500 sparse observed points on both the coefficient a and solution u and study the effect of the additional PDE loss guidance. The relative error of u reduces from 9.3% to 0.6%, and the relative error of a reduces from 13.2% to 9.4%. Therefore, we conclude that PDE guidance helps smooth the prediction and improve the accuracy.We examine the results of DiffusionPDE in solving forward and inverse problems when there are 100, 300, 500, and 1000 random observations on a, u, or both a and u. The error of DiffusionPDE decreases as the number of sparse observations increases. DiffusionPDE is capable of recovering both a and u with errors 1% ∼ 10% with approximately 6% observation points at any side for most PDE families. DiffusionPDE becomes insensitive to the number of observations and can solve the problems well once more than 3% of the points are observed. Observation Sampling Pattern. While CFG struggles with robustness, we show that DiffusionPDE is robust to different sampling patterns of the sparse observations, including grid and non-uniformly concentrated patterns. Note that even when conditioned on the full observations, our approach performs on par with the current best methods, likely due to the inherent resilience of our guided diffusion algorithm. Additionally, DiffusionPDE can leverage continuous coordinates with bilinear interpolation in the prediction space to obtain predicted values for points that do not lie directly on the grid, without compromising accuracy. 
conclusion and future work: 
 In this work, we develop DiffusionPDE, a diffusion-based PDE solver that addresses the challenge of solving PDEs from partial observations by filling in missing information using generative priors. We formulate a diffusion model that learns the joint distribution of the coefficient (or initial state) space and the solution (or final state) space. During the sampling process, DiffusionPDE can flexibly generate plausible data by guiding its denoising with sparse measurements and PDE constraints. Our new approach leads to significant improvements over existing state-of-the-art methods, advancing toward a general PDE-solving framework that leverages the power of generative models.Several promising directions for future research have emerged from this work. Currently, Diffusion-PDE is limited to solving slices of 2D dynamic PDEs; extending its capabilities to cover full time intervals of these equations presents a significant opportunity. Moreover, the model's struggle with accuracy in spaces that lack constraints is another critical area for exploration. DiffusionPDE also suffers from a slow sampling procedure, and a faster solution might be desired. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We include implementation details of data generation and sampling in the supplemental materials (Sections B and C).Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: We report the standard deviation of our DiffusionPDE in the supplemental materials (Section I).Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: Our work, which concentrates on solving both forward and inverse problems of partial differential equations (PDEs) with sparse observations, promises to significantly advance scientific and engineering disciplines that rely on these computations. The potential negative impacts are minimal, as this research primarily aims to provide more accurate and efficient computational methods without adverse effects on existing systems or practices.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [NA]Justification: The paper does not use existing assets.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: The paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: For experiments with sparse observations or sensors, we find that DiffusionPDE performs the best when weights ζ are selected as shown in Table 2 . During the initial 80% of iterations in the sampling process, guidance is exclusively provided by the observation loss L obs . Subsequently, after 80% of the iterations have been completed, we introduce the PDE loss L pde , and reduce the weighting factor ζ obs for the observation loss, by a factor of 10. This adjustment shifts the primary guiding influence to the PDE loss, thereby aligning the diffusion model more closely with the dynamics governed by the partial differential equations. DiffusionPDE performs better when we apply the PDE loss term L pde in addition to the observation loss term L obs as guidance, as shown in Table 3 . The errors in both the coefficients ( initial states) a and the solutions (final states) u significantly decrease. We also visualize the recovered a and u and corresponding absolute errors of Darcy Flow, Poisson equation, and Helmholtz equation in Fig. 8 . It is demonstrated that the prediction becomes more accurate with the combined guidance of PDE loss and observation loss than with only observation loss. We further refine the noisy outputs generated by baseline methods such as DeepONet, PINO, FNO, and PINNs. Specifically, given a partially observed parameter a for the PDE f (c; a, u) = 0 and a pre-trained forward operator F ′ , we address the problem by solving the optimization equation:EQUATION)and the results are shown in Table 5 and Fig. 15 . Optimization reduces errors and smooths the solutions. However, the resulting values are smaller due to the smoothing effect from minimizing PDE loss, and the overall error compared to the ground truth remains much higher than DiffusionPDE. This may be due to difficulty in optimizing the derivatives of noisy a and u.We further assess the significance of our DiffusionPDE by analyzing the standard deviations for forward and inverse problems under conditions of 500 observation points and full observation, respectively, as detailed in Table 6 . We evaluate our model using test sets comprising 1,000 samples for each PDE. Our findings confirm that full observation enhances the stability of the results, a predictable outcome as variability diminishes with an increase in observation points. The standard deviations are notably higher for more complex PDEs, such as the inverse problems of the Poisson and Helmholtz equations, reflecting the inherent challenges associated with these computations. Overall, DiffusionPDE demonstrates considerable stability, evidenced by relatively low standard deviations across various tests. We evaluate the computing cost during the inference stage by testing a single data point on a single A40 GPU for the Navier-Stokes equation, as shown in Table 7 . DiffusionPDE has a lower computing cost compared to Shu et al. [54] , which autoregressively solves the full time interval. This advantage becomes more significant when we increase the number of time steps. Further, we evaluate the inference runtimes on one single A40 GPU of vanilla full-observation-based methods and also the optimization time of them during the inference as introduced in Appendix H. The optimization runtimes are significantly slower, especially when using Fourier transforms. We find that DiffusionPDE is robust against sparse noisy observation. In Fig. 16 , we add Gaussian noise to the 500 observed points of Darcy Flow coefficients. Our DiffusionPDE can maintain a relative error of around 10% with a 15% noise level concerning the forward problem, and the recovered solutions are shown in Fig. 17 . Baseline methods such as PINO also exhibit robustness against random noise under sparse observation conditions; this is attributed to their limited applicability to sparse observation problems, leading them to address the problem in a more randomized manner. Robustness on Sampling Patterns Moreover, as mentioned in the main document, we investigate the robustness of DiffusionPDE on different sampling patterns of the observation points. Here, we address the forward problem of Darcy Flow using 500 observed coefficient points, which are non-uniformly concentrated on the left and sides or are regularly distributed across the grid, as depicted in Fig. 18 . Our results demonstrate that DiffusionPDE flexibly solves problems with arbitrary sparse observation locations within the spatial domain, without re-training the neural network model. However, the CFG method faces challenges when solving with varying sampling patterns, as demonstrated in Fig. 19 . In this figure, we compare the reconstruction results of DiffusionPDE and Diffusion with CFG for the unbounded Navier-Stokes equation, where all observation points are located on the left side of the grid. The CFG approach struggles with this asymmetric sampling pattern, while DiffusionPDE maintains more accurate reconstructions. Stochasticity Evaluation Since we employ a deterministic diffusion model, with partial observations as input, the only source of stochasticity or uncertainty in our approach arises from the initial random noise. To examine this, we conducted experiments to assess the impact of different noise seeds on both the initial and final states of the Navier-Stokes equations, as demonstrated in Fig. 20 . Our findings indicate that the diffusion model exhibits some degree of uncertainty in its predictions, despite the deterministic nature of the underlying framework. We also investigate how our DiffusionPDE handles varying degrees of sparse observation. Experiments are conducted on the Darcy Flow, Poisson equation, Helmholtz equation, and non-bounded Navier-Stokes equation. We examine the results of DiffusionPDE in solving forward and inverse problems when there are 100, 300, 500, and 1000 random observations on a, u, or both a and u, as shown in Fig. 21 . We have observed that the error of DiffusionPDE decreases as the number of sparse observations increases. Overall, we recover u better than a. DiffusionPDE can recover u with approximately 2% observation points at any side pretty well. DiffusionPDE is also capable of recovering both a and u with errors 1% ∼ 10% with approximately 6% observation points at any side for most PDE families. We also conclude that our DiffusionPDE becomes insensitive to the number of observations once more than 3% of the points are observed. To evaluate the generalizability of DiffusionPDE, we implemented the model on various resolutions, including 64 × 64 and 256 × 256, while maintaining the same percentage of observed points. For resolutions of 64 × 64, 128 × 128, and 256 × 256, we observe 125, 500, and 2000 points on a or u respectively, which are approximately 3% for each resolution. Overall, DiffusionPDE is capable of handling different resolutions effectively. For instance, Table 9 presents the forward relative errors of the solution u and inverse error rates of the coefficient a for the Darcy Flow, demonstrating that DiffusionPDE performs consistently well with similar error rates across various resolutions. We have compared the results using the RBF kernel [60] , as shown in Fig. 22 . For the forward process of solving the Poisson, Helmholtz, and Darcy Flow equations, the RBF kernel achieved solution errors of approximately 14.3%, 23.1%, and 18.4%, respectively, with 500 random observation points. However, when addressing the inverse problem, the errors increased significantly to 141.2%, 143.1%, and 34.0%, respectively. This increase in error is likely due to the inherent challenges of solving inverse problems with such a straightforward method. Additionally, we compare our DiffusionPDE method with a single U-Net model. The U-Net is trained based on our EDM diffusion model, where we initially train it to map between 500 fixed input points and the full output space, as illustrated in Fig. 23 . For the Navier-Stokes equation, the prediction of the final state results in an average test error of approximately 39%, which is significantly higher than the error produced by our diffusion model. Furthermore, when making predictions using 500 different sampling points, the relative error increases to approximately 49%. We also train another U-Net model to map between 500 random input points and the full output space, but this model results in a test error of 101%, indicating that the U-Net struggles to adapt to varying sampling patterns and fails to flexibly solve different configurations. Justification: In the Abstract, we include the URL to our project page, which provides access to the data and code.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 


Decision: This diffusion models proposed here solve PDEs with unknown coefficients by modeling the joint distribution of the coefficients and solution space.  An advantage of this generative approach over some other learning-based PDE solvers is that it can flexibly handle partial observations with irregular patterns.  While one borderline-negative review is skeptical of this probabilistic approach, overall we found the authors' rebuttal to these concerns to be convincing.  There were some concerns about the range of experiments and baseline methods compared to in the original manuscript, but a thorough author rebuttal addressed most of these points.  Please be sure that your final manuscript incorporates these additional experiments and comparisons.

Reviews: review: paper_summary: paper_summary: The paper uses score based generative diffusion models to find the forward and backwards solution of a set of PDEs given partial observations of the solution and/or incomplete knowledge of the coefficients. The method performs well, and outperforms other ML  methods such as FNO, as well as 'standard' FE type methods, for a range of standard test problems. The method reconstructs This is a novel approach, which delivers good performance, with low errors  at a competitive speed. Extensive tests are given, with careful analysis of the results.
strength_weakness: strength_weakness: This paper suffers as do many similar papers from a limited range of examples. It concentrates on the usual examples of PDEs such as NS and Bergers, and in both cases of these it looks at problems with quite moderate viscosity, which are realtively east to solve. This is more or less inevitable for such a short paper as this, especially as comparisons are needed with other method. But I would have liked to have seen more novel examples than the usual ones. This is not really a criticism of this paper, but is something to consider for future work. It would be imporoved by a fairer comparison with other methods which work with incomplete data and measurements. A clear exanple of this being the data assimilation widely used in physical modelling for just this range of problems. These should be descibed somewhere in the introduction and in Section 2. (Although of course these latter methods are slow in comparison.) The method is also limited (see later) to looking at certain slices of the solution.
questions: questions: 1. How does this method compare with a data assimilation approach
2. How easy would it be to extend the method to full time intervals 
3. How easy is it to extend the method to higher dimensions
4. Have the authors tried out the method on more challenging PDE examples.
5. Also consider tests on NS and Bergers' eqn with much smaller viscosity.
limitations: limitations: The model as described only looks at slices of solutions of 2D problems. This has been clearly identified by the authors. In this sense it is vrather limited when compared to other ML based approaches, and of course traditional FE based methods. I am pleased that this is recognised and that the authors plan to address this. The DiffusionPDE method will only be truly competitive when this is done, but this paper is a good step in this direction.
paper_summary: paper_summary: The work uses a guided diffusion process to solve the PDE forward and inverse problems with partial observations. Instead of learning the parameter-to-solution map ($a\rightarrow u$) as in Neural Operators, the method learns the diffusion process on the joint distribution $(a,u)$, and use guided diffusion for inference under sparse observations. Compared with several baseline method, the proposed method shows improved performance for solving forward and inverse problem with sparse observations.
strength_weakness: strength_weakness: The paper presents an interesting approach to solving PDE forward or inverse problems with sparse observations, which is an appealing concept given the minimal data requirement. However, this approach raises some concerns about the well-posedness of the problem. For example, in forward problems where sparse observations of the parameter $a(x_i)$ are available, there are infinitely many ways to interpolate $a$ and solve the PDE to obtain $u$. They are all valid solutions that satisfy both the PDE and the observations. This suggests that the method's ability to achieve good recovery might heavily rely on the strong regularization imposed by the training dataset, potentially limiting its practical utility as it may only favor solutions resembling those in the training set.

Additionally, in Appendix C, Table 2, the weightings for observation and PDE loss are significantly higher (by two to six orders of magnitude) than those for $\nabla_x \log(p(x))$ as described in Equation 8, which might indicate a predominance of data fitting over the diffusion process. It would be beneficial if the authors could provide more guidance on how these weights were chosen and discuss the implications of using smaller weights. Understanding the rationale behind these choices could help clarify the model's dependency on these parameters and their impact on the solution's behavior.
questions: questions: (1) The results from the baseline methods (PINO, DeepONet, PINNs, and FNO) are so bad. The claim that these methods are "not easily extendable" invites further scrutiny:

(a) All the baseline models are supposed to represent smooth functions. However, in Figure 4, 7, 8, they look discontinuous at the training points.
An explanation of how these models were trained and how inference was conducted could clarify why these discrepancies appear.

(b) Taking PINNs as an example in the Darcy flow problem.
Let $\hat{a}(x)$ and $\hat{u}(x)$ be the (potentially noisy) observation at $x$.
We can represent $a(x)$ by a neural network, $a_V(x)$ (use neural net for convenience, could be other representations), and the PDE solution $u(x)$ by a neural net $u_W(x)$, where $V$ and $W$ are the weights of the neural network.
We can solve the following optimization problem:

$$\min_{V,W} \sum_{x\in T_d} (u_W(x) - \hat{u}(x))^2 + (a_V(x) - \hat{a}(x))^2 + \sum_{x \in T_r} (\nabla \cdot (a_V(x) \nabla u_W(x)) - q(x))^2$$

where $T_d$ is the observational point, and $T_r$ is the residual points, which does not need to be the same as $T_d$.

(c) Similar, for a trained neural operator parametrized by W, $G_W[a] = u$. We can solve the following optimization problem:

$$\min_V \sum_{x\in T_d} (a_V(x) - \hat{a}(x))^2 + (G\[a_V\](x)-\hat{u}(x))^2$$

where $G[a_V]$ is the solution of the PDE with parameter $a_V$.

It seems that all the baseline methods can be used for forward and inverse problems with sparse observation. 
It is unclear why the proposed method would offer superior performance compared with the baselines. 

In contexts where full observation are available, as shown in Table 4, one might intuitively expect methods like PINNs—which utilize residual losses to ensure adherence to PDE constraints—or Neural Operators—which establish a direct parameter-to-solution mapping—to deliver more accurate results compared to a method that relies on a diffusion process. This leads to a critical inquiry on why diffusion process gives better accuracy for PDE problems.

(2) How is the PDE loss computed? Is it by finite difference on a regular grid? Detailing this in the main text could help readers assess the accuracy and applicability of the PDE loss in different scenarios.
limitations: limitations: The author mention several limitation in the conclusion.
paper_summary: paper_summary: The paper proposes to solve PDEs given only sparse measurements by jointly modeling the solution and coefficient space (e.g. the initial conditions) using a diffusion model. By applying diffusion posterior sampling (DPS) the authors obtain samples that are consistent with the sparse measurements and the underlying PDE equations. Several experiments show superior performance of the method compared to standard baselines such as PINNs and FNOs.
strength_weakness: strength_weakness: - The main weakness of the method is the limited novelty. Both sparse measurements and physics-based losses have been considered together with diffusion models, see e.g. Shu et al. (2023). So it seems to me that the main technical novelty is to apply diffusion models to model the joint distribution of two simulation states at different points in time and apply DPS during inference for consistency with the sparse measurements and PDE constraints.     
- The experiments do not take into account any stochasticity or uncertainty. In principle, DPS will give a distribution of solutions, which is not the case for the other baseline methods, but this is not explored further in the paper. 
- Since the joint distribution models two states at time 0 and time T (for all experiments except Burgers' equation) and $0 \ll T$, the authors need to simplify the PDE loss $\mathcal{L}_{pde}$ to drop any time derivatives. This is a serious limitation.   
- It is not clear if DPS works better than classifier-free guidance, as used e.g. in Shu et al. (2023), or other methods for solving inverse problems with diffusion models. 
- DPS requires a lot of compute during inference for calculating $\mathcal{L}_{pde}$. For a fair comparison, it would be important to show the number of parameters, training time and inference time for all methods.
questions: questions: - Algorithm 1 shows an adaption of DPS to EDM (Karras et al. 2022). Is this adaptation novel? Can the authors give some intuition why they apply the DPS losses in line 12 and 13 to the 2nd order correction (line 8) and not apply any trapezoidal rules in this case?  
- Are sparse measurements located on a grid that matches the resolution of the diffusion model or do they have continuous coordinates? In the second case, how are they interpolated to match the data resolution of the diffusion model? Does that make classifier-free guidance difficult to apply?
- As noted in the weaknesses: why not use classifier-free guidance? I would like to see a discussion of different methods for inverse problems and diffusion models that can be used here instead of DPS and what are the advantages of using DPS. Reconstructing the solution/coefficient space from sparse measurements alone is a linear inverse problem with a number of different methods that can be used (e.g. Denoising Diffusion Restoration Models; Kawar et al. 2022, among many others) which oftentimes have much nicer theoretical guarantees/higher quality reconstructions and faster sampling speed. When considering these methods, is adding the PDE loss $\mathcal{L}_{pde}$ and thus making the problem a non-linear inverse problem really beneficial?
- Likewise, as mentioned above: what are the parameter counts and runtimes of the method and the baselines?
limitations: limitations: The authors have mentioned slow sampling speed as a limitation in the conclusion, but I think an extended discussion of this would be important to include.
paper_summary: paper_summary: This paper introduces diffusion methods to tackle the partially observed PDEs, named DiffusionPDE. By learning the joint distribution of solution and coefficient space, the proposed model can handle both forward and inverse problems. The authors experiment with diverse PDEs and settings to demonstrate the model's effectiveness.
strength_weakness: strength_weakness: 1.	The technical contribution is limited.

From a technical view, this paper is an application of the diffusion model in PDE solving. There are also some previous methods that also use diffusion methods and leverage the PDE loss [1]. Thus, I think the technical novelty is limited.

[1] A Physics-informed Diffusion Model for High-fidelity Flow Field Reconstruction, JCP 2023

2. Some powerful baselines are missing.

- According to Figure 1, I think the base model of DiffusionPDE is U-Net. How about comparing it with a single U-Net? I think U-Net could be a powerful baseline.

- There are also some latest models that are good at processing partially observed or irregularly placed PDEs, such as OFormer [1] and Transolver [2]. They should include them as baselines.

[1] Transformer for Partial Differential Equations' Operator Learning, TMLR 2023

[2] Transolver: A Fast Transformer Solver for PDEs on General Geometries, ICML 2024

3. Model efficiency comparisons are needed, including GPU memory and running time.

4. I think the proposed model cannot predict the future evolution of a time-dependent PDE. Current tasks are all about “reconstruction” or “imputation”.
questions: questions: 1.	Figure 4, do both forward and inverse tasks use the same diffusion model? Or do we need to train two models for these two different tasks?

2.	I think the base model is U-Net. So how does DiffusionPDE handle the spatially scattered partial observations? Is the input still in the regular grid, but only the sampled locations have ground-truth values?
limitations: limitations: I appreciate that they have discussed the limitations. But I think the mentioned issues about efficiency and limitations on temporal modeling are not trivial. More discussions are expected.

Comments: title: Rebuttal
Clarification on DiffusionPDE’s Robust Performance Using Continuous Coordinates and Bilinear Interpolation
comment: We greatly appreciate your response. We would like to clarify a few points:

> "the well-posedness of the problem"

We agree that there are infinitely many valid solutions to the partial observations. However, we assume that there are statistical patterns in the coefficient and solution spaces, which means that each valid solution has its own 'probability' of actually being the solution. We aim to learn this probability distribution with our diffusion model, which excels at sampling a highly likely full state given partial observations, as reflected by the much lower error compared to all prior methods suggested by ourselves and other reviewers, including a simple baseline (suggested by reviewer nBsW) of completing the coefficient space using an RBF kernel. Such problems of statistically inferring unobserved data have been widely studied across domains, such as image inpainting in computer vision and matrix completion in machine learning, which are deemed well-posed problems of maximizing probability.

> "Why do we mask the non-observed input values to zero?"

For our method and other baselines that require optimization during inference, we indeed exclude non-observed values. For baseline methods that expect a complete input space, e.g., FNO, we have to fill in those values. We have tried filling in the missing values with constant zero values (default) or using RBF kernel or nearest neighbor interpolation, all resulting in lower performance than DiffusionPDE.

> "As the PDE loss can be computed and minimized without any data. / This seems to suggest that PINN failed to train in these cases..."

PINN is trained well according to the original paper, and the reviewer is correct in noting that PINN can handle partial observations and automatically complete the missing data. However, it’s generally understood that PINN tends to have a test error between 10% and 30%, even in fully observed situations, because the training loss never perfectly converges to 0. This high error is one of the reasons why there is growing interest in neural operators [1].

> "This statement appears to conflict with the typical characterization of neural operators (FNO or DeepONet) as being 'resolution invariant'... Could you provide more details on the difficulties encountered?"

FNO can solve problems across different resolutions, and the reviewer is correct in noting that the output of neural operators can be evaluated at any point. However, we cannot assume that the input to neural operators can be sparse like ours. In fact, neural operators require a complete continuous grid as input, whereas we only have very sparse observation points (approximately 3%), which are highly discrete and do not meet this requirement. As a result, the model struggles to learn higher-frequency features. To the best of our knowledge, neural operators have not been shown to be effective on highly sparse inputs.

> "It seems somewhat counter-intuitive that learning a joint distribution could be more effective than learning this mapping directly."

The neural operators that learn the mapping directly indeed perform comparably to DiffusionPDE given "full" observation of $a$. However, when the observation of $a$ is not complete, the solution $u$ is not uniquely determined. Hence, we need to sample from the conditional distribution of $u$, a task in which DiffusionPDE outperforms neural operators and their variants suggested by the reviewers.

[1] Physics-Informed Neural Operator for Learning Partial Differential Equations. Li et. al. ACM/JMS Journal of Data Science 2024.
We greatly appreciate your response. We would appreciate it if you could let us know of any specific questions we can address to help raise the score.
Thank you for the updates and the interesting new results in the PDF.

It's definitely an interesting direction, and I'm open to supporting acceptance by raising my score. I still find it difficult to strongly argue for acceptance given the limited scope of the technical contributions, though.
Thanks for the the hard work and for conducting additional experiments. I appreciate the comprehensive benchmarking and the detailed responses to the reviewers' comments. 

> we know of no existing method that performs well under sparse observations without assumptions about PDE coefficients to the best of our knowledge. 

I agree. But I'm not asking for out-of-sample distribution.
My questions is on the well-posedness of the problem. 

Using the Darcy flow as an example. Suppose $a$ is sampled from some Gaussian random field with certain correlation structure.
If we only have partial observation of $\{x_i, a(x_i)\}$, for i = 1,...,N, and N is relatively small. Suppose the ground truth is $a_{GT}$. However, there could be $a_1$, ..., $a_M$ that are all consistent with the observation. That is, $a_j(x_i) = a_{GT}(x_i)$ for i = 1,...,N. and j = 1,...,M.

In this case, DiffusePDE might recover $a_{GT}$ (hypothetically), while FNO, DeepOnet, PINN, etc, might recover $a_1$, ..., $a_M$, which will certainly have error with respect to $a_GT$, but they are all consistent with the observation data and the PDE. Therefore, the low accuracy of other baseline methods does not necessarily mean they are worse. They might be different but still valid solutions to the problem.

As an even more straightforward approach in this case, we can sample the Gaussian random field conditional on the observation, and solve the PDE numerically, this would give infinite number of valid solutions to the problem.

> During inference, we apply a mask to set non-observed input values to zero.

Why do we mask the non-observed input values to zero? It seems a more reasonable approach is to exclude those in the loss function.

> For partial observations, PINNs [1] does not learn any knowledge about the PDE, making it difficult to inpaint the missing parts

My understanding is that PINN, or any method employing PDE loss, should be able to inpaint missing parts in ways that remain consistent with the PDE. As the PDE loss can be computed and minimized without any data.

> Neural operators [2] learn to map the entire coefficient space to the entire solution space and hence have difficulty taking partial observations as input

This statement appears to conflict with the typical characterization of neural operators (FNO or DeepONet) as being "resolution invariant" -- the neural operator can be evaluated at any point, even those not in the training points.
Could you provide more details on the difficulties encountered?


> For full observations, PINNs require iterations of optimization to converge to the solution, which can be vulnerable to local minimal or failure to convergence, leading to higher error.

This seems to suggest that PINN fail to train in these cases, which invite questions on how is PINN trained, As the problems (Darcy flow, Poisson, etc) are standard examples in PINN literatures [1,2,3,4].

> Iterative generative models, such as diffusion models, tend to beat feedforward models like GANs.

I'm not concerned about GANs. To my understanding, in your approach, the joint distribution between the parameter $a$ and the solution $u$ is modeled. That is, for any particular $a$, there is a conditional distribution of $u$. 
However, for PDEs, the solution $u$ should be uniquely determined by the parameter $a$. And neural operators aim to learn this mapping.
It seems somewhat counter-intuitive that learning a joint distribution could be more effective than learning this mapping directly.

As a summary, due to ongoing concerns regarding the problem formulation, implementation of some of the baseline method, and an lack of understanding of advantages of the diffusion-based approach, I will maintain my current score for this review.

[1] M. Raissi, P. Perdikaris, and G. E. Karniadakis, “Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations,” 

[2] DeepXDE(https://deepxde.readthedocs.io/en/latest/demos/pinn_forward.html)

[3] Q. He and A. M. Tartakovsky, “Physics-Informed Neural Network Method for Forward and Backward Advection-Dispersion Equations,”

[4] S. Wang, S. Sankaran, H. Wang, and P. Perdikaris, “An Expert’s Guide to Training Physics-informed Neural Networks,”
We greatly appreciate your response and recognition of our method. We would like to clarify that DiffusionPDE can utilize continuous coordinates with bilinear interpolation in our prediction space to obtain predicted values for points that are not on the grid. By doing so, for example, the forward problem of the non-bounded Navier-Stokes equation results in a relative error of approximately 6.0%, which is comparable to the scenario without interpolation (approximately 6.9%) and even smaller as more points are considered during the calculation of observation loss, while still remaining significantly lower than OFormer (approximately 16.2%). Overall, the diffusion model is powerful, enabling DiffusionPDE to consistently outperform OFormer. We are happy to mention all these discussions in the revised paper.
I would like to thank the authors' response. After rebuttal, I am clear about the paper's technical contribution.

There are still some questions remain. I think the extension of DiffusionPDE to a prediction model is non-trivial. The current framework cannot support this. Also, as the author mentioned, they treat the sparse observation in uniform grids, which is also not a generalizable design since the partial observations can be placed in any position of the domain. Some techniques, such as OFormer, may be more applicable than U-Net.

After rebuttal, I think the application scope of DiffusionPDE is limited. The authors should discuss more about the above-mentioned limitations in the revised paper. However, I appreciate the design in the joint modeling of forward-inverse problems. Thus, I improve my score to 5.