L-PINN: A Langevin Dynamics Approach with Balanced Sampling to Improve Learning Stability in Physics-Informed Neural Networks

Authors: Minseok Jeong, Giup Seo, Euiseok Hwang
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: Physics-informed neural network, Langevin dynamics, Adaptive sampling method

Abstract:
Physics-informed neural networks (PINNs) have emerged as a promising technique solving partial differential equations (PDEs). However, PINNs face challenges in resource efficiency (e.g., repeatedly sampling of collocation points) and achieving fast convergence to accurate solutions. To address these issues, adaptive sampling methods that focus on collocation points with high residual values have been proposed, enhancing both resource efficiency and solution accuracy. While these high residual-based sampling methods have demonstrated exceptional performance in solving certain stiff PDEs, their potential drawbacks, particularly the relative neglect of points with medium and low residuals, remain under-explored. In this paper, we investigate the limitations of high residual-based methods concerning learning stability as model complexity increases. We provide a theoretical analysis demonstrating that high residual-based methods require tighter upper bound on the learning rate to maintain stability. To overcome this limitation, we present a novel Langevin dynamics-based PINN (L-PINN) framework for adaptive sampling of collocation points, which is designed to improve learning stability and convergence speed. To validate the effectiveness, we evaluated the L-PINN framework against existing adaptive sampling approaches for PINNs. Our results indicate that the L-PINN framework achieves superior relative $L^{2}$ error performance in solutions while demonstrating faster or comparable convergence stability. Furthermore, we showed that our framework maintains robust performance across varying model complexities, suggesting its potential for compatibility with larger, more complex neural network architectures.

Decision: reject
Reviewer scores: 5, 8, 6, 5

Reviews and discussion:
<thread>
creation date:
1729502647953

thread_type:
official_review

<thread_object>
creator: reviewer_zj8i

<content>
summary:
In this paper the authors highlight some potential shortcomings of residual-based methods for training PINNs and the lack of theoretical understanding thereof. In particular, the authors show theoretically that convergence requires a tighter upper bound on the learning rate. Furthermore, the authors propose a novel algorithm to train PINNs that exploits Langevin dynamics. The main idea is, instead of resampling the collocation points at each iteration (proportionally to the residual loss), to update existing sample position based on residual loss gradient.

strengths:
- the paper is clearly written and easy to read.
- the idea of using Langevin dynamics to update sample points instead of resampling is elegant.

weaknesses:
- while the theoretical contribution on the learning rate is interesting, it seems to be not very useful in practice. Based on the ablation study in Figure 3a and 4a it seems like that the steepness changes only slightly and that picking a smaller learning rate would already help convergence of current methods.
- furthermore, as the authors highlighted in the paper, Langevin dynamics comes at the price of two new additional parameters, which need to be fine tuned. As the authors themselves note, experiments in Appendix E suggest that the choice of these two parameters highly influence the performance of the proposed method.

questions:
- Theorem 4.1 relies on an asymptotic limit for reaching collocation sample population. Therefore, I expected that performance would improve as the number of Langevin steps increases. Can you elaborate on why this is not the case based on the results in Appendix E? (the authors note that this behaviour is particularly true for small $\beta$ but also in the other cases it is hard to see an improvement if $l_L$ is increased). Somehow these results seem to imply that reaching collocation sample population is not useful in practice.
- It would be interesting to see quantitatively how much the choice of the step size $\tau$ in L-PINN affects convergence. It might be useful to have a comparison along the lines of the evaluation in Appendix E for $\beta$ and $l_L$, but in this case as $\tau$ varies.
- Instead of sampling new collocation points to find high-residuals, the proposed approach updates points according to the residual gradient. Intuitively, this seems to be more intense computationally, since sampling can usually be done rather cheaply. Did you run any experiments in this direction? 
- I find Figure 5 a bit confusing. Could you clarify whether you are showing the error with respect to the solution or the solution itself?

Some minor typos:
- line 88: I would add $R_\theta(x)$ after "residual" such that it is clear what $R_\theta(x)$ is
- line 105: should use a comma instead of a point before "i.e."
- line 148: missing a point before "Additionally"
- line 430: for L-PINN, RAD, R3 and L* "exact solution" should be "predicted solution"
- line 460-466: results fro the Burgers' equation with 8 layers for Random-R and R3 should also be in boldface
</content>
</thread_object>

</thread>
<thread>
creation date:
1729972882798

thread_type:
official_review

<thread_object>
creator: reviewer_skj6

<content>
summary:
This paper examines the relationship between sampling strategies and learning stability in PINNs. The authors provide theoretical analysis showing that sampling methods focused on high-residual points require stricter learning rate constraints for stability, especially with increased model complexity. They present a Langevin dynamics-based PINN (L-PINN) framework that implements balanced sampling proportional to PDE residuals. They vaildate the effectiveness of the proposed sampling method across multiple PDEs, showing that  L-PINN achieves comparable or better relative L2 error performance while maintaining stability across different model complexities and learning rates compared to existing methods.

strengths:
1.  The proposed method is novel and provides rigorous theoretical analysis of stability issues in high-residual sampling methods. The authors establish clear mathematical connections between sampling strategies and learning rate constraints, backing their theoretical claims with detailed proofs. This helps explain the stability challenges that emerge as model complexity increases.

2. Another strength is the paper's thorough empirical validation. The experiments span multiple representative PDEs (Burgers', Convection, Allen-Cahn). The authors carefully compare their approach against various sampling methods, with comprehensive ablation studies on learning rates and network depth.

weaknesses:
1.  The assumption 3.1 seems to be too strong for nonlinear PDEs. But I think it is ok for linear PDEs. 

2. Section 5.2's experimental results are dense and difficult to follow - this content could be better organized with supporting details moved to the appendix. 

3.  The authors overlook relevant prior work, particularly PirateNet [1], which addresses similar scaling challenges in training deep PINN models through adaptive skip connections.

[1] Wang, S., Li, B., Chen, Y. and Perdikaris, P., 2024. PirateNets: Physics-informed Deep Learning with Residual Adaptive Networks. arXiv preprint arXiv:2402.00326.

questions:
1. What exactly are the feature vectors being visualized in Figure 6? The paper relies on Assumption 3.1 to express PDE residuals as linear combinations of feature-mapped vectors, but does not clearly define how these feature vectors ϕ(x) are obtained for nonlinear PDEs. Are these simply outputs from hidden layers, or do they incorporate PDE residual information? The authors should provide precise mathematical expressions for the visualized quantities.

2.  If these feature vectors are merely hidden layer outputs without considering PDE residuals, there appears to be a significant gap in the paper's logic. How do these empirical visualizations justify Assumption 3.2 about heavy-tailed distributions of feature vectors that appear in Eq 3.1?  This disconnect between the theoretical framework and empirical validation needs to be addressed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730061464287

thread_type:
official_review

<thread_object>
creator: reviewer_sawy

<content>
summary:
This paper investigates the limitations of high residual-based methods concerning learning stability as model complexity increases. The authors claim two questions which remain unclear: lack of theoretical analysis of the balancing effect and potential risks of the high residual method. They provide a theoretical analysis and propose Langevin dynamics-based PINN (L-PINN) framework for adaptive sampling of collocation points. The paper also compares the performance between L-PINN and other adaptive sampling methods.

strengths:
- This paper investigates the limitations of high residual-based methods concerning learning stability as model complexity increases.

- This work provides a theoretical analysis and propose Langevin dynamics-based PINN (L-PINN) framework for adaptive sampling of collocation points.

- The paper is well-written and easy to follow.

- The figures and tables are clear.

weaknesses:
- The other adaptive sampling methods (baselines) compared in the paper have some important hyperparameters which could significantly affect the performance. For example, in RAD, $k$ and $c$ are important and improper values of them will fail the method. I cannot find any description about the hyperparameters of the baseline methods. I am concerned about if the baselines are well-trained.

- It is true that applicability to large models is important for these methods. However, Applying simple 1-D PDEs to deep MLPs may not be a good choice.
  - These PDEs do not need such deep MLPs at all. Two or three hidden layers are enough for training the model. 
  - Increasing the depth of MLPs will make the training unstable. 
  - On the other hand, increasing width will also increase the size of the MLPs and will not make the MLPs unstable. The paper could do a comparison on this.
  - It might be more meaningful to adopt other multiple layers like attention layers and Fourier layers instead of deep MLPs. I am concerned about whether L-PINN can be applied to the scenarios that really need large models and still be robust.

questions:
- “Adaptive sampling based on residual distribution” is introduced at first and then the “Adaptive sampling focused on high residuals”. Why do the authors claim the latter method is used to address convergence issues of the former method? Actually, RAR can be regarded as a special case of RAD or RAR-D. The RAD method is used to address the issue of RAR focusing too much on high residuals. RAD has less convergence issues than RAR.

- RAD methods have experimentally discussed the “Unresolved questions 2, Potential risks of the high residual method” (lines 136-139). I admit this paper discusses this problem from a different aspect. But I would like to see what is unclarified or unsolved of RAD with respect to “Unresolved questions 2”.

- A period is missing in line 148.

- In figure 4, I suggest adding standard deviations. For figure 4b, it seems L-PINN only performs better when the learning rate is 0.003. This might result from randomness. I would like to see more cases when the learning rate is between 0.002 and 0.004.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730614666778

thread_type:
official_review

<thread_object>
creator: reviewer_u9zx

<content>
summary:
The paper presents a method to adaptively select the collocation points for solving partial differential equations (PDEs) through physics-informed neural networks (PINNs). In PINNs, the selection of the number and location of collocation points impacts the model training and is a well-known issue in the literature. Various methods have already been proposed in the literature, some of which have been compared in this paper. The paper presents a novel method for selecting the collocation points through empirical results and theory. The method named Langevin PINNs aims to focus not only on high-residual-based locations, but also proposes to balance it with selecting locations with low or medium residual values. Overall, the paper's motivation aligns with improving PINNs for simulating PDEs, showcasing its effectiveness on canonical examples.

strengths:
The paper is easy to follow for non-theory parts and presents a concise overview of the literature in this domain.

The choice of PDEs is diverse and incorporates diverse challenges observed while training PINNs for simulating PDEs.

Sample trajectory plots help observe the method's performance.

weaknesses:
The rationale for showcasing the method's performance on deep networks is unclear. The advantage gained by the algorithm with a deep neural network is unclear. From Table 1, it seems like the baselines achieve a similar result even with a small network, so why would one opt for a further deep network which is even harder to optimize?

The method is not clearly explained, and it is unclear how to implement the proposed method. It would be appreciated if the authors could provide a more detailed explanation of the method and implementation. 

The discussion on the computational complexity of the baseline methods is presented. However, the manuscript does not compare the proposed method's computational cost with the baselines. Can the authors provide such a comparison of the computational cost? It would help the readers analyze the advantages of the proposed method in terms of computational cost.

Although compared with similar methods, can the authors compare their method with RAR-D presented in [1] or justify why the comparison is/should not be performed?

It is difficult to understand what Fig. 5 shows. It seems like the performance of the proposed method is completely off for a deeper network, contrary to the text in the article.

Although a sensitivity analysis of the proposed method is carried out with different learning rates, the range seems limited. How do the methods perform when trained at an even smaller learning rate?

Limitations:

The proposed method is presented for low-dimensional problems, and validating its performance on high-dimensional problems is not performed. The selection of collocation points in higher dimensions is also a complex problem. The paper does not discuss how the method scales with the rise in dimensionality.

Along similar lines, the method is not performed for multiscale systems of PDEs, which have challenges in choosing the right collocation points.  

[1] Wu, Chenxi, et al. "A comprehensive study of non-adaptive and residual-based adaptive sampling for physics-informed neural networks." Computer Methods in Applied Mechanics and Engineering 403 (2023): 115671.

questions:
Included along with weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732086560083

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Regarding the rebuttal response format

comment:
We sincerely thank the reviewers for your detailed and insightful comments. We have made every effort to address as many of the raised issues as possible, and the updated version of the paper has been uploaded.

Our rebuttal responses reference this updated version of the paper. Consequently, the locations of specific content addressing prior concerns may have shifted slightly, and we kindly ask the reviewers to take note of this.

Additionally, we will incorporate further experimental results into the paper whenever possible before the rebuttal deadline.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732580402321

thread_type:
official_comment

<thread_object>
creator: area_chair_dlka

<content>
comment:
Dear Reviewers u9zX, SaWy,
If not already, could you please take a look at the authors' rebuttal? Thank you for this important service.
-AC
</content>
</thread_object>

</thread>
<thread>
creation date:
1733194291601

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Final Summary of the Rebuttal

comment:
We sincerely appreciate the reviewers' insightful feedback, which has been instrumental in enhancing the quality, depth, and clarity of our work. Below, we summarize the key strengths of our approach, address the primary concerns raised, and highlight the additional efforts made during the discussion phase.

---

### Key Strengths of Our Approach

1. **Theoretical Contributions:**  
   While previous studies have demonstrated the success of adaptive sampling methods in improving model performance, a rigorous theoretical framework to analyze these sampling strategies has been limited. Our work bridges this gap by proposing a detailed theoretical analysis that examines sampling methods in relation to model complexity, residual concentration, and stability. This framework not only provides insights into existing methods but also lays the groundwork for advancing adaptive sampling strategies in PINNs.

2. **Novelty:**  
   By integrating Langevin dynamics into Physics-Informed Neural Networks (PINNs), our approach introduces a novel mechanism that fundamentally enhances learning stability. Unlike existing methods that rely on Monte Carlo integration, our method avoids direct reliance on these techniques, which makes it inherently more robust when applied to higher-dimensional PDEs. This innovation addresses critical challenges in scaling PINNs to complex problems.

3. **Comprehensive Validation:**  
   To validate the proposed theoretical framework, we designed experiments explicitly aligned with the analysis presented in the manuscript. These experiments evaluate key aspects, including the impact of model complexity, residual concentration, and adaptive sampling on learning stability. The results not only reinforce the theoretical insights but also demonstrate the robustness and generalizability of L-PINN across a wide range of PDE scenarios.

---

### Addressing Reviewer Concerns

1. **Mathematical Clarification of Feature Vectors:**  
   To address feedback regarding feature vector construction, we revisited the mathematical formulation and clarified the derivation process in **Appendix A**. Specifically, we refined the representation of the feature vectors to better align with assumptions. This update resolves ambiguities and ensures consistency with the proposed theoretical framework, strengthening the connection between theory and practice.

2. **Scalability to Complex Scenarios:**  
   To address concerns regarding scalability, we conducted new experiments on two-dimensional PDEs, including Burgers' and Heat equations, which are outlined in **Appendix I**. Additionally, we performed a computational complexity analysis, detailed in **Appendix H**, to evaluate the efficiency of L-PINN in complex scenarios (more collocation points, higher dimension). These results highlight L-PINN's ability to efficiently scale to complex scenarios without compromising stability or performance.

3. **Comparative Analysis with Alternative Architectures:**  
   In response to reviewer feedback, we expanded our comparative analysis to include alternative architectures, such as random Fourier features, attention mechanisms, and modified MLPs (e.g., incorporating residual blocks) detailed in **Appendix J**. These additional experiments confirmed that L-PINN maintains compatibility and effectiveness across diverse model structures, further underscoring its adaptability and versatility.

4. **Clarification of Explanations:**  
   We recognize the ambiguities in certain parts of our manuscript, particularly regarding distinctions between methods like RAD and RAR-D and their theoretical implications. Additionally, we acknowledge that the interpretation of **Figure 4-(a)** may have introduced confusion. While immediate revisions are not feasible, we will address these points in the final version by clearly separating and structuring the relevant discussions and improving the explanation of experimental results, especially those in Figure 4-(a). These refinements aim to enhance the clarity and accessibility of our explanations.

---

### Final Remarks

The reviewers' constructive feedback has been invaluable in improving our manuscript and resolving potential ambiguities. Through additional experiments, expanded comparative analyses, and planned improvements to explanations, we have strengthened both the theoretical and experimental foundation of L-PINN.

We believe that our work offers a meaningful contribution to the field by providing a robust framework that improves learning stability in PINNs through balanced sampling strategies. Once again, we sincerely thank the reviewers for their thoughtful comments and constructive suggestions, which have greatly enriched our work.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper considers improving PINN, which uses a neural network to represent the solution to a PDE and seek it by minimizing an integration of the PDE residual. Accurately evaluating the integrated residual is an outstanding challenge, because PINN is believed to particularly suit high dimensional problems, for which full integrations are however too expensive to perform. The authors proposed a Langevin SDE based approach to randomly sample collocation points for evaluating the residual in a balanced fashion, so that high residual locations are emphasized but low residual locations are not ignored either. Reviewers and I all agree this is an interesting idea. However, there were concerns about robustness / details of the implementation and inadequate evidence of improved performance over existing approaches, which remained unresolved after the rebuttal. Moreover, although the sampling problem matters more in high dimensions, only low dimensional PDEs were tested in the paper. In addition, there was insufficient comparison with non-PINN approaches. Therefore, I encourage the authors take these discussions into consideration and submit again.