Title: Mitigating Instability in High Residual Adaptive Sampling for PINNs via Langevin Dynamics
Venue: NeurIPS 2025
Accepted: yes
Keywords: Langevin dynamics

Adaptive sampling

Physics-informed neural network

Learning stability
Abstract: Recently, physics-informed neural networks (PINNs) have gained attention in the scientific community for their potential to solve partial differential equations (PDEs). However, they face challenges related to resource efficiency and slow convergence. Adaptive sampling methods, which prioritize collocation points with high residuals, improve both efficiency and accuracy. However, these methods often neglect points with medium or low residuals, which can affect stability as the complexity of the model increases. In this paper, we investigate this limitation and show that high residual-based approaches require stricter learning rate bounds to ensure stability. To address this, we propose a Langevin dynamics-based Adaptive Sampling (LAS) framework that is robust to various learning rates and model complexities. Our experiments demonstrate that the proposed method outperforms existing approaches in terms of relative $L^{2}$ error, and stability across a range of environments, including high-dimensional PDEs where Monte Carlo integration-based methods typically suffer from instability.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a mechanism for training PINNs using Langevin sampling to learn the distribution of residuals. The authors provide theoretical guarantees on the convergence properties of the method (i.e. distributional convergence and flatness). A comprehensive set of experiments were conducted, and LAS shows state-of-the-art results in six out of ten benchmarks, including the notably challenging high-dimensional PDEs.\n\nAlthough I think this work is interesting from an experimental perspective and well-informed in the existing literature, I find the theoretical aspects of this paper concerning. I am willing to increase my score if the authors provide improvements in the proofs in a rebuttal.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n* I believe this paper explores some interesting directions that improve the training of PINNs. The search for collocation points for PINNs is relevant, especially in high dimensions.\n* The assumptions (3.1, 3.2, 3.3) are well-formulated and fairly weak so that they don't compromise the generalizability of the theoretical results.\n\n**Weaknesses**\n\n* This paper, although theoretically correct, is inconsistent with continuous-time and discrete-time formulations of Langevin dynamics. For example, Equation (4.1) is the Euler-Maruyama discretization of the Langevin equation $dX_t=\\nabla_{x}|R_{\\theta}(x_n^l)|^2+\\beta dB_t$. I should expect to see both formulations in the paper.\n* Although Theorem 4.1 is correct, its proof is wrong. Moreover, in a continuous-time FP equation, I should not expect to see any discretization in time ($\\tau$) in the proof. In the rebuttal, I want to see the authors use the continuous-time SDE formulation in the proof. I recommend the following sources [1] [2].\n* There are serious inconsistencies in the structure of the residual. In the proof of Proposition 3.1, you write that $R_\\theta(x)=\\theta^T\\phi(x)$, which is different from Equation (3.3) in Assumption 3.1. The proof of Proposition 3.2 has the same issue.\n* Theorem 3.1 also seems wrong (see questions)\n\n[1] Jordan, R., D. Kinderlehrer, and F. Otto (1998). “The Variational Formulation of the Fokker–Planck Equation”. SIAM Journal on Mathematical Analysis, vol. 29, no. 1, pp. 1–17.\n\n[2] Ambrosio, L., N. Gigli, and G. Savaré (2008). “Gradient Flows in Metric Spaces and in the Space of Probability Measures”. 2nd. Springer.\n\nQUESTIONS:\n* Can the authors clarify the inconsistency between propositions 3.1 and 3.2, and assumption 3.1? There seems to be a contradiction.\n* Every theorem/proposition, by convention, should list all the underlying assumptions explicitly instead of making the reader guess which assumptions hold.\n* I think the authors are missing a proof when bridging assumptions 3.2 and 3.3. Heavy-tailed distributions, by definition, prevent tight concentration, so these two cannot simultaneously hold. What are the conditions that the weights or $\\beta$ must satisfy for both assumptions to hold?\n* Theorem 3.1 also seems unclear. If $\\max\\|\\phi\\|\\sim D^{1/\\alpha}$, then $(\\max\\|\\phi\\|)^2\\sim D^{2/\\alpha}\\sim\\lambda_{max}(\\Sigma)$, so I don't see why you argue that $(\\max\\|\\phi\\|)^2 >> \\lambda_{max}(\\Sigma)$\n\nLIMITATIONS:\nA practical concern is that the benchmarking methods require fewer neural network evaluations compared to the proposed method, which involves solving an SDE with neural network evaluations at every time step. But I appreciate that the authors addressed this in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a Langevin dynamics-based Adaptive\n Sampling (LAS) for the solution of PDEs under the PINN framework. This work is part of a wider literature on adaptive sampling strategies for PINNs colloration point residuals, roughly divided into distribution based sampling and high-residual based sampling. The authors focus on the latter, proceeding by providing a systematical analysis of the theoretical properties of high residual sampling and their connection to learning instability. They identify sharp gradients as one the min source of instability and suggest the use of Langevin dynamics to update the residuals. This is a change from previous approaches based on sampling independent residuals from a target distribution at every iteration. The use of noise in the LAS step has a smoothing effect which the authors argue leads to flatter local optima and improved algorithm stability.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The paper is very well written, and very clear. Morevoer, while the main body is dense with results, both theoretical and empirical analysis, this does not affect the flow of the exposition.\n- The authors have been methodic and thorough in their analysis: they develop a theoretical analysis under some assumptions, which are validated eitehr with references to prior work, or in the appendix. The main theorems are also assessed empirical to show additional evidence (e.g. Figure 3).\n- The experimental evaluation of their model is similarly precise. Eight different PDE systems are assessed which comprise increasing complexities and a wide range of behaviors. The performance is systematically compared with four competing approaches.\n- The experimental results prove the validity of the approach, point out the theoretical features of the algorithm described in the first half of the paper, and highlight the pitfalls of the competing algorithms.\n- Extensive ablation studies are reported, and algorithm runtimes are provided.\n- Finally, clear implementation details are provided, which allow other researchers to replicate and build on their work, and practictioners to use their methodology.\n\nOverall, I am satisfied with the quality of this paper.\n\n**Weaknesses**\n- The authors fail to provide a clear paragraph covering the limitations of their proposed approach.\n- While algorithm runtimes are reported in the appendix, these are not commented in the text. However, LAS is found to be more computationally expensive than the other approaches, except than L^\\infty. Generally, LAS is 1.2 to 6 times slower, using 1 to 10 number of layers. However, results in the main body are only reported for 8 and 10 layers, which are significantly more expensive and more prone to out-of-memory (OOM) issues for high problem dimensions, as reported in Table 8. Note that OOM is recorded using a RTX 4090 GPU with 24GB memory, so for practitioners with more limited hardware, they may encounter memory bottlenecks much sooner.\n\nQUESTIONS:\nAs I mentioned, the paper looks good. However, in my opinion it critically lacks two things:\n1. A discussion on the computational requirements. This should involve both comments on the hardware used, and viability of the algorithm on other kind of hardware (e.g. smaller GPUs, CPUs). Moreover, it should mention the computational complexity (empirical or theoretical) and how the algorithm performs against the baselines. Part of this is present in the appendix but, in my opinion, it is very important to give visibility to this issues in the main body with a high-level discussion. Details can be provided in the appendix.\n2. The presentation of the LAS performance compared to the baseline (or its own performance with 8 and 10 layers) for a smaller number of layers (1, and 5). This provides a fairer comparison in terms of compute, and provides evidence to practitioners on whether this approach can be used in more resource costraiend environments.\n\nLIMITATIONS:\nLimitations were not mentioned. Crucially, the authors failed to comment on the computational cost-accuracy tradeoff.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
In this paper, a new adaptive sampling method called LAS is proposed to enhance the accuracy of Physics-Informed Neural Networks (PINNs). LAS addresses the issue of neglecting points with medium or low residuals, which can compromise stability. The experiments demonstrate that the proposed method outperforms several existing approaches in terms of relative error and stability across a variety of environments, including high-dimensional partial differential equations (PDEs), where Monte Carlo integration-based methods typically struggle with instability.

STRENGTHS AND WEAKNESSES:
engths: The paper is well-written, and the motivation is clearly articulated.

Weaknesses: The numerical analysis is insufficient to demonstrate the value of this paper, particularly due to the lack of comparisons with closely related works such as 

[1] Failure-informed adaptive sampling for PINNs
[2] Failure-informed adaptive sampling for PINNs, Part II: Combining with Re-sampling and Subset Simulation
[3] A Gaussian Mixture Distribution-Based Adaptive Sampling Method for Physics-Informed Neural Networks

QUESTIONS:
See weakness.

LIMITATIONS:
More numerical comparision is needed,  see weakness.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-02

comments: SUMMARY:
This paper proposes to use Langevin dynamics to update collocation points when training PINNs. The authors analyzed the gradient behaviors for training with high residual points and training with random samples, and showed that there could be big difference between these two approaches regarding convergence and stability. The authors consequently proposed to update the collocation points locally via Langevin dynamics, which may favor flat regions and enjoy better training stability. The proposed method is tested on several 1D PDEs as well as high-dimension PDEs.

STRENGTHS AND WEAKNESSES:
Strength

- The paper is well written and easy to follow, despite having certain theoretical depth.

- The paper empirically benchmarked several collocation sampling methods, which are tested on both simple and high-dimensional PDEs.

Weakness

- Although the overall experimental results show favorable performance for the proposed method. I feel the gain in the performance seems not very strong. For example, as shown in Table 1, for some PDEs, random sampling still yields the best results. The results for high-dimensional PDEs are significantly better for the proposed method. However, the error seems to increase with the dimensionality, and the benchmarked existing methods do not really work and they may not be tailored for such problems.

- Nevertheless, although I am not expert on tuning PINNs, I think the proposed Langevin dynamics sampling is an interesting idea and its motivation is well analyzed.

QUESTIONS:
- What is $a(\theta)$ in equation 3.2?

LIMITATIONS:
- Limitation is discussed adequately.
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-05