PIG: Physics-Informed Gaussians as Adaptive Parametric Mesh Representations

Authors: Namgyu Kang, Jaemin Oh, Youngjoon Hong, Eunbyung Park
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: Gaussians, Physics-informed Deep Learning

Abstract:
The numerical approximation of partial differential equations (PDEs) using neural networks has seen significant advancements through Physics-Informed Neural Networks (PINNs). Despite their straightforward optimization framework and flexibility in implementing various PDEs, PINNs often suffer from limited accuracy due to the spectral bias of Multi-Layer Perceptrons (MLPs), which struggle to effectively learn high-frequency and nonlinear components. Recently, parametric mesh representations in combination with neural networks have been investigated as a promising approach to eliminate the inductive bias of MLPs. However, they usually require high-resolution grids and a large number of collocation points to achieve high accuracy while avoiding overfitting. In addition, the fixed positions of the mesh parameters restrict their flexibility, making accurate approximation of complex PDEs challenging. To overcome these limitations, we propose Physics-Informed Gaussians (PIGs), which combine feature embeddings using Gaussian functions with a lightweight neural network. Our approach uses trainable parameters for the mean and variance of each Gaussian, allowing for dynamic adjustment of their positions and shapes during training. This adaptability enables our model to optimally approximate PDE solutions, unlike models with fixed parameter positions. Furthermore, the proposed approach maintains the same optimization framework used in PINNs, allowing us to benefit from their excellent properties. Experimental results show the competitive performance of our model across various PDEs, demonstrating its potential as a robust tool for solving complex PDEs. Our project page is available at https://namgyukang.github.io/Physics-Informed-Gaussians/

Decision: accept (poster)
Reviewer scores: 6, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1729086870825

thread_type:
official_review

<thread_object>
creator: reviewer_uwk5

<content>
summary:
This paper proposes Physics-Informed Gaussians (PIGs), which incorporates Gaussian embeddings in the PINNs framework, to eliminate the inductive biases of neural networks.

strengths:
The strength of this paper is to overcomes the limitations of traditional PINNs.  The proposed method maintains the benefits of PINNs, improves performance in PDE approximation, and can be applied to various PDEs.

weaknesses:
As mentioned in Discussion and Limitations, theoretical understanding of the convergence properties is lack, although the universality of PIG is discussed in Section 3.3.

questions:
I may miss something but, is there any reason why choose Gaussian functions as feature embedding? Would other embedding functions fail in the proposed framework?
</content>
</thread_object>

</thread>
<thread>
creation date:
1729238309008

thread_type:
official_review

<thread_object>
creator: reviewer_vutu

<content>
summary:
Physics-Informed Gaussians (PIG) is proposed, an efficient and accurate PDE solver that utilizes
learnable Gaussian feature embeddings and a lightweight neural network, counteracting the problems inherent in previous static parametric grid approaches. PIG model achieves competitive accuracy and faster convergence with fewer parameters compared to state-of-the-art methods.

strengths:
- The paper is well-written and easy to follow. I enjoyed reading it.

- Motivation is clear.

- Comparisons with previous works are well-discussed.

- The proposed model is supported by both theoretical and empirical evidence.

- Error bars are provided, enhancing the reliability of the experimental results.

- Code is provided for reproducibility.

weaknesses:
- (Minor) Figures should be vector images to avoid pixelization.

- While I found no significant weaknesses, there may be room to further expand the discussion on the novelty and its potential impact on future research.

questions:
- (Line 245-) "Similar to the previous parametric grid methods, which obtain feature embeddings by interpolating only neighboring vertices, this locality encourages the model to capture high-frequency details by effectively alleviating spectral bias": Is it possible to mathematically prove that "this locality encourages the model to capture high-frequency details by effectively alleviating spectral bias" in some sense? This is just a question out of curiosity.

- Is the title of Section 3.2.2 relevant?

- Why is tanh used in the architecture (it is fine if there is no specific reason)? What about sin or swish activations?

- Is there any issues about mode collapses ($\mu_i \rightarrow \mu_j$ for all $i$ and $j$ and/or $\sigma_i \rightarrow 0$ or $\infty$)?

- I think the update scales of $W$ and $\theta$ differ significantly from those of $\mu_i$ and $\sigma_i$. The optimal updates for $\mu_i$ and $\sigma_i$ likely depend on the scale of the simulation. Did the authors encounter any issues related to this?

- It may be off-topic, but is the proposed method applicable to high-dimensional PDEs, such as $d\gtrsim100$? How significant is the computational cost?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730660840289

thread_type:
official_review

<thread_object>
creator: reviewer_sq8v

<content>
summary:
The authors propose the Physics-Informed Gaussian (PIG) that learns feature embeddings of input coordinates, using a mixture of Gaussian functions. Compared to PINN, this paper applies Gaussians to input coordinates and makes the mean and covariance matrix learnable. The locality of Gaussians encourages the model to capture high-frequency details by effectively alleviating spectral bias.

strengths:
The paper is well-written and easy to follow. The code and experiments are of good quality. It provides a more concise and more parameter efficient feature embedding method compared to previous parametric grid representations method.

weaknesses:
- The primary concern lies in the originality of this method. Though it’s considered as a feature embedding, mathematically, it is in the same spirit of a KAN layer with RBF. Also, it is not uncommon to consider Gaussian embedding in ML. 

- The paper states that the approach "dynamically adjusts the positions and shapes of the Gaussians during training" with parameters "iteratively updated throughout the training process." The claims in the paper give an impression of adaptive sampling method based on the training. However, the approach does not actually take into account the residual or the loss function landscape. It would be better to provide a more rigid description on this. Also, in the experiments, the method is compared with different PINN approaches for each PDE, and the authors report the results without conducting experiments by themselves again. Though “the sensitivity of PINN variants to hyperparameters complicates fair comparisons”, it’s still necessary to conduct experiments with same setup and number of parameters to ensure a fair comparison. The average error of PIG is generally higher, but the authors only highlight the best of PIG. PIG is claimed to “enable more accurate and efficient approximations”, but the efficiency is not clear. Hence, it seems a lot of conclusions in this paper are overstated.

- I would like to see how the spectral bias can be alleviated, either through experimental evidence or analytical justification.

- For the 2D and 3D cases, what is the mathematical formulation for the Gaussian embedding? How is the parameter being initialized? Additionally, how are the boundary conditions (BC) being constrained?

- The paper mentions that "computational costs per iteration of our method are significantly lower than JAX-PI." A detailed report of these computational costs would be beneficial.

questions:
- I would be interested to see how the authors address the weaknesses above. Could authors provide additional evidence or context to support the novelty? 
- Have the authors conducted any experiments to demonstrate how the method alleviates spectral bias?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730733798369

thread_type:
official_review

<thread_object>
creator: reviewer_77oa

<content>
summary:
In this paper, the authors use deep Gaussian mixture models as parametric mesh representations to replace MLP in PINN, so that it performs better for high-frequency and non-linear components. Experiments for PDEs such as Allen-Cahn, Helmholtz, Nonlinear diffusion, etc., are conducted compared with several previous PINN derivatives, showing that the proposed method achieves sota solution accuracy.

strengths:
- A theoretical proof of the universal approximation capability for the proposed method is provided.
- The experiments are diverse, the baselines are sufficient, and the ablation study is detailed.
- The writing is overall clear and detailed.

weaknesses:
- Though diverse and rich, the experimental settings are all relatively simple, and lack comparisons to traditional methods.
- Correct me if I am wrong, but I think only the solution accuracy is provided, without the computation time?

questions:
- With the work SIREN in mind, I wonder if the Gaussian basis is the best choice?
- The idea is quite similar to 3D Gaussian splatting in rendering. However, there are differences. One easy way to explain is that, we can think of the volume rendering equation as a specific type of governing physics equation. Then the success of applying parametric mixed Gaussian in one specific type of equations doesn't mean it is the best idea for others. For example, there can be long-range interactions for certain PDEs, while Gaussian functions are local and lack such representation capability. I would like to have the authors' opinions on such comments.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732246309658

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Thank you to all the reviewers for their reviews and constructive feedback. Before addressing each review individually, we will clarify some commonly raised points in this general response. All revised parts in the revised paper have been marked in red.

1. We anticipated potential questions regarding experiments involving full covariance cases, and as a result, we promptly initiated these experiments. Through these efforts, we have successfully enhanced the performance of the Full Covariance model. The final results of these experiments are presented in the table below. 

|   | 2D Helmholtz | 3D Klein-Gordon  | 3D Nonlinear-Diffusion | 3D Flow-Mixing | 
|---|---|---|---|---|
| Diagonal  | 1.10e-04 | 2.24e-03 | 4.13e-03 | 8.23e-04 |
| Full | 5.17e-05 | 1.81e-03 | 3.86e-03 | 3.48e-04 |
| # Gaussians  | 500 | 100 | 50 | 400 |


2. We conducted additional experiments on the 100-dimensional Allen-Cahn and Poisson equations. For the 100-dimensional Allen-Cahn equation, PIG achieved an $L^2$ relative error of $8.88 \times 10^{-3}$. Similarly, for the 100-dimensional Poisson equation, PIG achieved an $L^2$ relative error of $8.42 \times 10^{-3}$. The $L^2$ relative error curves for these experiments are provided in Appendix Figure 11.

3. We tested the Lid-Driven Cavity equation, a well-known challenging problem due to its singular behavior. The results demonstrated that our method (PIG) achieved superior $L^2$ relative error performance compared to the baseline model PGCAN using the Parametric Grid method (PIG: $4.04 \times 10^{-4}$, PGCAN: $1.22 \times 10^{-3}\)$. The detailed results and figures can be found in Appendix Figures 12 and 13.


4. We performed additional experiments on the inverse problem of the Allen-Cahn equation. These experiments showed that PIG converged significantly faster, as illustrated in Appendix Figure 10.

5. To address spectral bias, we conducted experiments on the high-frequency Helmholtz equation $(a_1 = 10, a_2 = 10\)$. Our method achieved an $L^2$ relative error of $7.09\times 10^{-3}$. Notably, this equation cannot be learned at all using standard PINNs. The results are presented in Appendix Figure 14.

7. Finally, we compared the computational time and memory usage of our method. For the Allen-Cahn equation, JAX-PI required 0.01666 seconds per iteration, which is **$2.30\times$** slower compared to the baseline, and consumed 22.9GB of memory, which is **$11.68\times$** larger. These details are summarized in the table below for clarity.

|   | 3D Klein-Gordon | 2D Allen-Cahn  | Nonlinear-Diffusion  | 2D Helmholtz | 3D Flow-Mixing |
|---|---|---|---|---|---|
| Time per iteration (s) | 0.0025 | 0.0074 | 0.042 | 0.050 | 0.12 |
| Memory (GB) | 0.7 | 1.96 | 4.9 | 3 | 17.8 |
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a new version of PINNs to solve PDEs. The proposed method utilizes the Gaussian feature embeddings with a lightweight neural networks. A universal approximation theory is provided. Numerical experiments show the proposed method is effective compared with other standard methods.  

This paper is well written. The theoretical analysis justifies plausibleness of the proposed method. The numerical experiments are comprehensive and well highlight effectiveness of the proposed method in comparison with the other existing methods. A detailed ablation study is given.  

Thus, I recommend acceptance for this paper.