FP64 is All You Need: Rethinking Failure Modes in Physics-Informed Neural Networks
Authors: Chenhui Xu
Dancheng Liu
Amir Nassereldine
Jinjun Xiong
Venue: NeurIPS 2025 poster ()
Keywords: Physics-Informed Neural Networks
Failure Modes
Primary area: machine_learning_for_sciences

Abstract: Physics‑Informed Neural Networks (PINNs) often exhibit “failure modes” in which the PDE residual loss converges while the solution error stays large, a phenomenon traditionally blamed on local optima separated from the true solution by steep loss barriers.
We challenge this understanding by demonstrate that the real culprit is insufficient arithmetic precision: with standard FP32, the L‑BFGS optimizer prematurely satisfies its convergence test, freezing the network in a spurious failure phase.
Simply upgrading to FP64 rescues optimization, enabling vanilla PINNs to solve PDEs without any failure modes.
These results reframe PINN failure modes as precision‑induced stalls rather than inescapable local minima and expose a three‑stage training dynamic—un‑converged, failure, success—whose boundaries shift with numerical precision.
Our findings emphasize that rigorous arithmetic precision is the key to dependable PDE solving with neural networks.
Our code is available at Supplementary Material.

FP64 is All You Need: Rethinking Failure Modes in Physics-Informed Neural Networks
Chenhui Xu
Dancheng Liu
Amir Nassereldine University at Buffalo, SUNY Buffalo, NY, USA, 14226 edu
Jinjun Xiong*
Abstract
Physics-Informn Informed Neural Networks (PINNs) often exhibit "failure modes" in which the PDE residual loss converges while the solution error stays large, a phenomenon traditionally blamed on local optima separated from the true solution by steep loss barriers. We challenge this understanding by demonstrate that the real culprit is insufficient arithmetic precision: with standard FP32, the L-BFGS optimizer prematurely satisfies its convergence test, freezing the network in spurious failure phase. Simply upgrading to FP64 rescues optimization, enabling vanilla PINNs to solve PDEs without any failure modes. These results reframe PINN failure modes rrecisoon-inuuce stalls rather than inescapable local minima and expose three-stage training dynamic- dynamic-un-comee un-converged, failure, success- whose boundaries shift with numerical precision. Our findings emphasize that rigorous arithmetic precision the key to dependable PDE solving with neural networks. Our code is available httpt::///gtt h/ttth/////t......oee teigthhb.... wwwm/mmmniiiiii Hui COMiiiiPNNNNFFPPP
Introduction
Informee Neural Networks (PINNs) 27 have gained wide attention and applications in recent years as a novel numerical solver for partial differential equations (PDEs). PINNs can find a numerical solution by optimizing the residual loss defined by the PDE, leveraging the nature of the universal approximation [16] of neural networks and the automatic differentiation 4 provided by the deep learning framework such PyTorch [26). Although theoretically it is capable of providing exact numerical solutions for PDEs, the researchers have identified several failure modes for PINNs [20]. In these cases presenting failure mode, the PDE residual loss is optimized to very small value, but the numerical solution provided by PINN is over trivial and has a huge error with the true solution.
To mitigate these failure modes, various methods based optimization 29 28 regional gradient 34), sampling 212 32 9. and model architectures [35] 36 23) have been proposed. These methods are based consensual understanding of the PINN's loss landscape: such failure modes occur when the model becomes trapped local optimum located within an extremely sharply descending loss basin 2 20 The optimizer typically has difficulty climbing out of this loss basin, resulting in model that eventually converges to a totally wrong solution. Based on this hypothesis, there should be a significant loss barrier [II] between the failure modes and the true solution of the PDE.
[
However, our empirical results show that there is no such a loss barrier between the failure mode and the true solution that can block the optimizer from surmounting. As shown in Fig. (a))(b), we find that the model that eventually converges to an ideal numerical solution also has experienced failure mode pattern during the training process. There is a similar non-synchronous decrease in residual loss and error during the training process in both success and failure mode cases. As in Fig [ (c)) the expected sudden rise in loss in line with the traditional loss landscape understanding does not occur during the transition of the model from a failure mode pattern to a success case.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Peskcre 14411
Prod x007
(a) Prediction Change Training Itcration
Failure Modes Case
0:0
04-
(b) Prediction Change
over Training Iteration
Success Casc
Iteration
Ieraaton 700
Iteration 1400
Ineration 2000
0.0
255
753
1000
1590 1750
250
500
1009
1250
1500
2758
(c) Failure Modes Training
(d) Success Case Training
(c) Loss Landscape
( Eroo Landscape
Dynamic/ / FP32 Precision
Dynamic/ FP64 Precision
Figure 1: Training Dynamic of PINN's failure mode case and success case.
This phenomenon directly challenges the current understanding of the loss landscape of PINN failure cases in the machine learning community. The cause of the PINN failure case is not a local optimal solution. Instead, as shown in Fig. (f), there significant discrepancy between the loss landscape and error landscape of a PINN. It suggests that the so-called failure mode may just be an intermediate stage of PINN optimization. The model ends up in failure mode due an unanticipated stop in the optimization process. Therefore, this leads to fundamental question:
What makes PINN's 's optimization process unexpectedly stop at a faiture mode?
We found the answer to this question surprisingly simple: The arithmetic precision of the model not enough to maintain the optimization process any longer. Arithmetic precision is generally considered not to play decisive role in neural networks in vision and language, where using FP32, FP16 and even lower precision is common practice. But the problem PINN is dealing with is scientific computing with high precision requirements. By using the default precision setting provided in the deep learning framework, the convergence condition of L-BFGS [21], which is the mostly used robust optimizer for PINN, is triggered prematurely due to the lack of enough arithmetic precision. To illustrate that, we show in Fig. [ with exactly the same problem setting, models with FP32 precision consistently present failure modes, while the models with FP64 precision always succeed.
Regarding the training dynamic, we identify that the training process of PINN will undergo three phases: un-converged phase, failure phase, and success phase. With the same initialization and optimization setup, models with different precisions will finally stop in different stages of optimization. Failure modes problems that used to be considered more difficult (c.g., with a nigher-frequent convection parameter) have longer failure phases and more gentle loss plain. As a result, the required arithmetic precision is positively correlated with the difficulty of solving a failure mode problem.
Further, we find that all these known failure modes can be solved by vanilla PINN with sufficient arith- metic precision using the L-BFGS optimizer. Performance of vanilla PINNs with FP64 precision on PDEs like convection can surpass most ttate-of-the-art model architectures that claim enhancement on PINN failure modes. With reduced arithmetic precision, models like PINNsFormer [36] show a typical failure mode feature again, indicating that may not solve the failure modes fundamentally.
Contributions. In this paper, we mainly made the following contributions:
We revolutionize the understanding of PINN failure modes. We reveal that, on the loss landscape of PINN, instead of being in separate loss basin, the failure mode has a pathway to the optimal solution that can be found by the optimizer.

---

66
01
0
(a) Convection Equation, Loss 0.0239
(b) Reaction Equation, Loss = 0.0199
Figure 2: Some cases of PINN failure modes.
We reveal that unexpected stopping of the optimizer due to insufficient arithmetic precision is the core cause of PINN failure modes. With suitable precision, vanilla PINN is strong enough to avoid failure modes with the L-BFGS optimizer.
We show that the optimization process of PINN is divided into three stages related to difficulty, and different precisions will make the optimization stop at different stages.
2 Related Works
Physics-Informee Neural Networks. differential equations of a physical system directly into neural-networt training objective. Instead of fitting network solely to observational data, PINNS minimize residual loss that penalizes violations of the PDE (or ODE/integro differential law) at set of collocation points, This simple idea turns -iiffreerrodifffemmttaal
Neural Networks 2 embed the governing
neural network uo (ar 1) into a mesh-free free solver that learns a function satisfying
F(uo (z, t)) = 0, V(2, t) E Ω2 Blus (ar, t)) 0, V(x, t) 0n,
where (x. t) E Q is the ppatial-tempore coordinate, ön is the boundary of ft. The F, and B denote the operators defined by PDE equations, initial conditions, and boundary conditions, respectively.
The training of PINNS leverages automatic differentiation techniques
25 available within contem-
porary deep learning frameworks, such as PyTorch [26], to perform numerical differentiation and thereby construct the residual loss at selected collection points. Recent studies have demonstrated that second-order optimization methods, noteably quasi-Newton methods such as SSBFGS [24). Broyden [5] and L-BFGS 2T], can significantly enhance the stability and fitting accuracy of PINN
training 18 15
33
20
compared to relying exclusively on first-order optimizers such as Adam 17
Failure Modes in Physics-Ifformed Neural Networks. Despite the promising performance and broad applicability, PINNs are susceptible to several failure modes that hinder their effectiveness in practice 20 The failure modes of PINN refer to a phenomenon in which the PDE residual loss of a model is optimized to a very low level (close to 0), but its resulting solution is still far from the expected true solution. As in Fig. 2
1-d convection and reaction equations optimized close-zero level, but their approximations to the equation's 's true solutions present the over-simplified patterns.
In response to this phenomenon, researchers have come up with a very wide range of conjectures. Early attempts involve curriculum regularization and breaks down complex PINN problems into simple ones and conquers them one at time. Subsequently, R3 [ tries to address this issue with resampling on the failure area. Then, several model architectures like PINNsFormer [36], ProPINN [34] PirateNet [30]. and PINNMamba [35] are proposed to help the propagation of the correct pattern among the collection points. From an optimization perspective, researchers also propose regional optimization 33 and Adam+L-BFGS 28 to help the convergence.
learning (00). which
Yet, these approaches rely on a hypothesis that the failure mode of PINN is a local minimal that completely isolated from the true solution in loss landscape. During the optimization, model either goes towards a failure mode loss basin or a true solution basin. These approaches intend to induce the model towards the true solutions with an inductive bias. While these methods have achieved some results on some part of failure mode cases, their understanding is not intrinsic. We argue that the failure modes are in the same loss basin as the true solution, and are caused by early stopping.
Arithmetic Precision in Neural Networks. Arithmetic precision is commonly thought to have a sub- sidiary impact on the mainstream applications of neural networks [4] Although early deep-learning systems relied almost exclusively on 32-bit doating-point arithmetic, a large body of empirical work now shows that reducing numerical precision rarely degrades the accuracy of modern models.
3

---

Pmi
X
Univensal Basin
Success Pottern Dusin
Faiture
Araiture
Failure Mode Basin
Loss-Barrier Hypothesis
(b) Same-Basin Hypothesis
Figure 3: A schematic that illustrates two kinds of loss landscape hypothesis.
Production-scadd scale vision, speech, and language pipelines routinely deploy INTS/FPS/FPI//FII kemels mixed-precee precision execution and deliver FP32-level quality with multi-fold improvements in speed and energy efficiency (37 13 10 On the GPU-design side, hardware is now co-engineered
for these lowpprecision formats: NVIDIA's Hopper architecture [8] adds FP8/INT8 Tensor Cores and a Transformer Engine for satriinncy-cccrran trade-off. This co-exploration lower-presd swver-precision from both software and hardware hints that, in general application of neural networks, arithmetic precision might not be an essential determinant of the model performance. However, we identify that this trend may not fit the scientific machine learning. Numerical methods based neural networks need to take the same high accuracy calculations as traditional methods (e.g., finite element methods (3)).
Problem and Experiment Settings
3.1 Problem Settings
The PINN employs a neural network parameterized by to approximate the solution of a PDE defined as Eq. by optimizing the following residual loss on 101 x 101 collection points (x4, 4:):
minimize
C(uo) Ar + XBLs(ue),
where (uo)
Σ
(T;,)(1;,44));;: = 10xl (xx,4))0x
Σ
[B(uo(E;;4a)) 4:11 (2)
Ixl
(xx,6))xx
To evaluate the performance of the models, take relative Mean Absolute Error (rMAE, a.k.a & relative error) and relative Root Mean Square Error (rRMSE, a.k. lg relative error) formulated as:
>N [u (Ins tn) (nn)
DN lue (Ins bn) (Ins tn
tMAE (f)
IRMSE (ñ)
E Iu (In, (n)
Em= Iu (In, tn)
(3)
where N is the number of test points, u(x, f) is the ground truth, and ú(x, 1) is the model's prediction.
In this paper, we investigate four famous failure mode PDEs that are widely studied the PINN community: convection equations, reaction equations, wave equations, and Allen-Cahn 20 36 34)
The descriptions and details these problems can be found the Appendix
3.2 Experiments Settings
We initialize the vanilla PINN with an MLP with 3 hidden layers and 512 neurons in each layer. We
include advanced PINN neural architectures like PINNsFormer [36], KAN [22], PINNMamba [35) to test the generalization ability of our findings, following their original settings. We train the models with L.-BGGS optimizer following common practice 20 33 36 35). All the experiments are
implemented NVIDIA H100 GPU, with CUDA version 12.8 and Pytorch vesion 2.I.I.

---

Error Distribution wint zatiion
Loss and Error over toochs
Loss Landscape
1.00
0.75
0.5
72
54
-025
0.2
-0.2
34
00 2.2
0.1
0.6
0.
1.2
3
250
520
750
1000 Spach
1250
1503
1750
2003
-100
-1.0
-0.5
0.0
(a) Error Distribution w.r. Initialization
(b) Absence of Loss-Barrier
(e)Universal Loss Basin
Figure 4: Several empirical results on convection support Same-Base Basinn Hypothesis.
4 Understanding PINN Failure Modes from Loss Landscape Perspective
In this section, we demonstrate that the failure modes of PINNs are not attributed to the local minimal with extremely steep loss landscape, which is view widely accepted by scholars who have studied this area. Instead, we point out that, the failure modes are in the same loss basin as the true solution. The model finally present failure mode is ascribed to the early stopping of the optimization process.
4.1 Loss-Barrier Hypothesis vs Same-Basin Hypothesis: Which is the True Story?
We first state two hypotheses for failure modes of PINNs: (1) the
Hypothesis, which is widely accepted by the PINN researchers, and (2) the Same-Batt Hypothesis, which we propose.
Loss-Barrier Hypothesis. PINN failures are initially attributed that "the optimizer has gotten stuck in a local minima with a very high loss function" [20]. This assertion is then been overthrown because the observation of near-zero empirical residual loss in failure modes experiment cases 33 34 35 36).
This hints that the cause of PINN failure modes is not that PDE residual loss cannot be optimized. Instead, shown in Fig. (a). it seems that the failure modes are because the optimizer is trapped in a local minimum of the loss landscape, which is nearly as low as the true solution's 's empirical residual loss on collection points. Under this hypothesis, the optimizer is not able to distinguish the failure modes and the true solution minimum. Therefore, models cannot find the true solution, because there a loss barrier between the failure mode local minimal and the true solution minimal. The model's loss faces huge penalty if its parameters are optimized to cross such loss barrier.
Same- Basin Hypothesis. Based the intuition that the loss basins generally connected with [1] we propose an alternative same-basin hypothesis. It says that the failure modes parameters are the same loss basin as the true solution, as shown in Fig. (b). There a large flat plain at the bottom of this loss basin, and the loss of the true solution is only slightly lower than that of the failure modes. Due to some reasons, the optimizer early stops when it reaches the flat plain in the iterated optimization process. In this hypothesis, there should not be significant loss barrier between the failure mode and the true solution. As a result, the optimizer with a global convergence guarantee should be able to find the global minimum empirically with ideal computational conditions.
a simple perturbation
4.2 Physics-Informe Neural Networks Follow Same-Basin Hypothesis
Although the Loss-Barrier Hypothesis is generally believed to hold, we identify three important clues pointing to the Same-Base Hypothesis: (1) initialization insensitivity, (2) absence of loss barrier, and (3) loss connectivity. This hints that the failure modes are caused by insufficient optimization.
PINNS Fail with any Random Initializations. If the failure mode patterns and the true solution patterns are in separate loss basins, it should be the case that, with some initialization, the model can be optimized to the true solution. But in practice, the situation is that the model is always trapped in the failure mode pattern, no matter how we conduct the random initializations. As shown in Fig (a). with 2000 different random initializations, the trained PINNs always fail on convection equations. This implies the extreme sparsity of the global optimum over the loss landscape. Under this implication, Barrier Hypothesis completely contradicts the intrinsic lottery ticket phenomena of neural networks (T] This suggests choosing the alternative hypothesis: Same-Base Hypothesis.
5

---

(I) ConveCe Phasc
Fa)luie Phase
(3)Success Phase
brie over toods
Prodited
Pneiiced
1966
1250
1710
Eguch
(a) Convection Problem solved by PINNsFormer
High Loss
Low Loss Iligh Error
Low Loss
High Error
Low Error
wyd 4000
weddictdd aoul
boss und Bris ovr Epocs
0.0
(b) Wave Problem solved by PINNMamba
Figure 5: Loss-Error Dynamics of PINN models can be in three phases.
Non-existent Loss Barrier from Failure Mode to Optimized Solution. A gravely important finding is that the expected loss barrier in the illustrate this finding, we initialize the PINN model with failure mode parameters that have been fully optimized for the convection problem, where there is a convergence signal of L-BFGS optimizer. Then we turn the arithmetic precision from FP32 FP64 (an important finding we will discuss in Section 5). and continue the training process for 2000 iterations. We find that, unlike the loss function that is longer updated all shown in Fig. (c), the loss shows further trace decline. As shown in Fig. (b), during this further decline process, we observe steep decline of the error (rMAE). This direct evidence against the Loss-Barrics Hypothesis. The absence of loss barrier in practice hints that the failure mode minimal and the true solution minimal are connected in the loss landscape.
Hypothesis doesn't empirically exist. To
Universal Trough in Loss Landscape. Further, if the Boss-Barrier Hypothesis holds, there should be a very large number of loss basins that are close to zero at their lowest point. However, as shown in Fig. A (c), we can only observe one trough in the loss landscape. The minimum basins not separated, instead, there a connected trough in the loss landscape. This directly disproves the assertion that PINN's loss is too rugged, making model optimization difficult. This phenomenon suggests that the failure modes from PINN and the true solution should be in the same basin, and there should be an optimization pathway between them. We further visualized the loss and error landscape of a well-trained model, as shown in Fig. M (1). We found that the loss is relatively level and smooth, and does not have the expected extremely rugged loss landscape, In contrast, the error landscape is much steeper at the position near the true solution optimal. This means that a small perturbation to a parameter can have a much larger effect on the error of the model than the loss. At the same time, the true solution exists in only very small region on this large loss plain.
The three clues above show, directly or indirectly, that the real reason why PINNs can have failure modes is not an insurmountable barrier in the loss landscape, in other words, is not due to being stuck in a local optimum. Instead, the PINN true solution of the equation and the failure modes are in the same loss basin. This means that when considering the optimization problem of PINN, the researcher should accept the Same-Basin Hypothesis for model design and tuning.
4.3 The Loss-Error Dynamics of PINN's Training
Based on Same-Basi Hypothesis, the whole training dynamics of the PINN model can be divided into three phases: (1) Un-Converged phase, where both the loss and error are still in high level, suggesting the model parameters are still not optimized; (2) Failure phase, where the loss is optimized to near-zero level but the error is still the high level, and the model presents failure modes such as naive solution; and (3) Success phase, where the model parameter can both achieve near-zero level loss and error. This implies that every model that is optimizable to arrive at near-true solution also experiences pattern of failure modes at some point during the training process, yet the model is able

---

Ground Truth
PINN FP32 Training
Brror Map
PINN FP64 Training
Error Map
Convection
Bma 444
Reaction
Wave
Allen- Cabn
130
535
675
Figure 6: Results for solving 4 failure mode PDE problems with different arithmetic precision.
to be further optimized and eventually arrives at the optimal solution of the equations that the model capable of giving. Thus, the failure modes are intermediate states of the PINNS training dynamics.
We empirically confirm these phases with different PINN models and PDE problem settings. As shown in Fig. proven to solve the failure modes problem, for the convection and wave problem, respectively. We note that the division of training dynamics of PINN into three phases common in various problem settings. Different problem settings have different phase duration characteristics. Those PDEs with longer failure phase are more sensitive and likely be affected by failure modes. When the PINN failure modes were first identified, it was concluded that the difficulty of modeling the PDE increases with the PDE parameter (00). We attribute this that the larger PDE parameter will extend the duration of the failure phase. Therefore, the optimization processes are more likely to stop at the failure phase.
train PINNsFormer (36] and PINNMamba [35] which have been empirically
5 Precision Serves a Key Factor in PINN's Training
We now know that the cause of failure modes is the premature stopping of the model optimization process. This leads another question: what contributes to the premature stopping of PINN model optimization? In this section, we reveal that the arithmetic precision is a key issue of the PINN optimization. Unlike neural networks in CV and NLP, single-precision (FP32) training does not guarantce that all neural networks corresponding to PDE problems are efficiently optimized. Instead, double-precision (FP64) training stable strategy for various model and problem settings.
5.1 Failure Modes Disappear when Using Double-Precisiie Training
We first show that the most basic vanilla PINN can solve all currently known failure modes problems when using double precision training. As shown in Fig. 6 with all these PDE problems where the vanilla PINN exhibits failure modes, the use of double preeils-peiiioonn training yields solutions that are highly consistent with ground truth. This suggests that using double -reniiiire training allows the model optimization to go through all three phases, rather than stopping at the failure phase. Moreover, as shown in [ the vanilla PINN trained with double precision can outperform all state-of-the-art methods in terms of prediction error. hints that the inductive-biasee approach that was once based on the understanding of the Loss Barrier Hypothesis is not fundamental to the addressing of PINN
7

---

Table 1: Comparison with baseline methods on 4 failure mode problems.
Convection
Reaction
Model
Loss
rMAB
IRMSE
Loss
IMAE
IRMSE
PINN
0.0133 +: 0.0055 0.6904 +: 0.0826 0.7640 0.0694
0.1991 + 0.0001
0,9788 0.0019
0.9778
QRes
6
0.0153 + 0.0027 0.7498 + 0.0464 0.8184
0.0382
0.1991 + 0.0001
0.9826 + 0.0023 0.9830
0.0026
ForMsFormer
36
0.0009 + 0.0001 0.0327 + 0.0068 0.0435= 0.0073
3.0c-6 = 1.0c-6
0.0147 1: 0.0013 0.0296 : 0.0027
KAN
22
0.0250 + 0.0042 0.6213 + 0,0675 0.6985 = 0.0701
7,0e-6 = 1.0e-6
0.0167 + 0.0014 0.0312 0.0034
Net
0.0347 + 0.0061 0.9704 +: 0.1826 0.9740
0.1894
4.0e-6 = 1.0c-6 0.0178 0.0023 0.0443 0.0064
RoPINN
33
0.0189 + 0.0062 0.6251 +: 0.0940 0.7204
0.0941
5.8e-5 + 9.0e-6 010589
+ 0.0161 0.0965 + 0.0310
PINNMamba
35
0.0001 + 2.0c-5
0.0184 + 0.0037 0.0197 0.0038
1.0e6 = 1 1.0c-6 0.0092 + 0.0017 0.0213 0.0036
PINN FP64
5.0c-6 = 1.0e-6 0.00S9 + 0.0013 0.0072 0.0017
5 5.0c-6 0.0271 0.0063 0.0502 0.0111
Wave
Allen-Cahn
Model
Loss
:MAE
IRMSE
Loss
iMAE
IRMSE
PINN
27
0.0174 + 0.0061 0.2746 +: 0,0574 0,2837 = 0.0571
0.4703 = 0.2986 0,9720 +: 0.0370 0,9662= 0.0300
QRes
0.0965 + 0.0192 0.5335 + 0.1230 0.5273 0.1172
0.9887 0.0021 0.9821 0.0089 0.9846 0.0092
Former
3G
0.0231 + 0.0017 0.3492 + 0.0871 0.3571
0.0872
0.4625 = 0.2875 0.9908 + 0.0401
0.9913 # 0.0420
KAN
22
0.0067 + 0.0012 0.1475 + 0.0354 0.1489 = 0.0357
0.0234 = 0.0031
0,3166 + 0.0233 0,5661 + 0.0440
Pirate Net
0.0153 + 0.00S1
0.2544 + 0.0471 0.2637
0.0480
0.0017 + 0.0001
0.1088
0.0109
0.1889
0.0180
RoPINN
38
0.0015 + 0.0005
0,0631 + 0.0226
0.0642
0.0238
PINNMamba
35
0.00012 + 3e-5
0.0193 + 0.0033 0.0195
0.0033
0.0027 + 0.0002
0.1432 + 0.0123 0.2645 + 0.0201
PINN_
4.2e-5 : 1.6e-5
0,0080 + 0.0052
0.0081
0.0031
1.3e-5=4.0c-6
0.0157 + 0.0036
0.0545
0.0112
failure modes. Instead, insufficient arithmetic precision is a straightforward contributor to the failure modes of the PINN. As with traditional numerical methods such as Finite Element Analysis, PINN also requires the use of double precision to achieve accurate and trustworthy results.
5.2 Optimization Stops Early without Sufficient Arithmetic Precision
Based on the analysis Section 4 the failure modes of PINN are due to premature stopping of the optimization process. In order to find the reason why arithmetic precision affects the optimization stopping of the PINN, we target the optimizer and its implementation in a deep learning framework.
PINNs Commonly Use L-BFGS Optimizers. The PDE residual contains high-order differential operators whose spectra can span many orders of magnitude, giving a Hessian with extreme eigenvalue ratios. First-order methods (Adam/SGD) therefore move in directions dominated by the largest eigenvalues and make little progress on the stiff directions. Rightly, second-order optimizers such as Newton, BFGS, and L-BFGS implicitly (or explicitly) invert an estimate of the Hessian, scaling each parameter update by the local curvature and thereby shrinking the stiff directions while enlarging the flat ones. Among all the second-order optimizers, L-BFGS is the most commonly utilized for its excellent performance while maintaining relatively low-level time and space complexity. L.- BFGS (21] approximates the inverse Hessian using history of the past updates of the position and gradient. At each iteration, it constructs the Hessian approximation implicitly using the stored history. We include the details of the L-BFGS optimizer in Appendix B
Existing Implementation of L-BFGS Optimizer. In deep learning framework's implementation of the L-BFGS algorithm, such as 'torch.optim. LBFGS algorithmic control is divided between user- defined outer loops and library-defined inner loops. The user governs the outer loop by repeatedly invoking spppllizeesteppcloouureee where each call corresponds to a single quasi-Newton update that assimilates one curvature pair for the inverse Hessian. During a single call to 'optinizeersteplloouree) it may perform up to 'max_ite _ix_trrrrrr inner loops that repeatedly reevaluate the objective and gradient, compute a quasi-Newton scarch direction via the two-loop recursion with the latest 'history_ sizee curvature pairs, optionally conduct Strong-Wolfe Wolfe line search [31) and update both parameters and curvature history. The inner loop halts when convergence criteria on gradient norm or successive parameter changes are met, after which control returns to the user-level outer loop.
Low-Precision Causes the Inner Loop Early Stopping. We found that insufficient arithmetic precision prematurely triggers the convergence criteria of the inner loop. As shown in Fig. (a), FP32 triggers the inner loop early stopping (the count of inner loops <= 5) much earlier than FP64 does. This is due to the fact that the trigger condition for convergence in PINN ("tolerance "rolerance_change"" has a value of 1e-7 that is smaller than the machine unit & for single precision floating point numbers. Machine unit is the smallest positive floating-poit point number that, when added to 1, yields representable
8

---

During Training
Gradient
Counts of Inner Loops wit Outer Interation
Weight
25
Crodden Norm
Weight
weephn
P732
30
P764
2005
500
1000
1500
2000
2500
3000
500
1000
1500
2000
2500
3000
(a) Counts of Inner Loops
(b) Weights Norms
(c) Gradient Norms
Figure 7: The values of the PINN weights orders of magnitude different than the values of the gradients, and the ratio is also incremental, causing single precision to underflow earlier.
FP64
$
PINN
0.7910
0.0059
1,2670
0.2644
0.0237
0.0037
QRes
NaN
0.7870
0.7235
0.7498
0.0179
10
NuNN
0.9670
0.3133
0.0631
0.0043
PINNsFormer
NeN
0.9128
0.4233
0.0231
0.0087
NeNN
0.9920
0.4923
0.0047
KAN
1.2789
0.9901
0.6723
30
NoN
0.7910
0.6904
0.0059
PirateNet
1.1614
0.8210
0.5881
0.0759
50
100
NaN
NaN
0.9010
0.8933
0.0192
PINNMamba
NaN
NaN
0.2410
0.0188
0.0042
(a) Convection PDE Parameters Arithmetic Precision
(b) PINN Models Arithmetic Precision
Figure 8: Results on convection equations with different arithmetic precision: (a) PINN's rMAE various convection parameters ,3. (b) Different models' rMAE on convection with B = 50.
value strictly greater than 1 on a given hardware/format:
min & > 0: 1 + & is representable and 1 + 8 1}.
(4)
For radix 3 (= 2) and p binary digits in the significand, & = 3 which means the FP32 has machine 1.19 e-7 "tolerance_ "erranceechhnge' 1e-7, while FP64's ss machine 2.22 e-16.
As shown in Fig. (b), during the optimization of PINN, the Norms of the weights are incrementally increasing in the order of magnitude of le+1 level, which is greater than 1. This means that the computation effective numerical precision is even more roarse-graineh than machine E, and thus computing the changes of weights will be easier to underflow. In Fig. (c), we show that the norms of the gradient are decreasing to near-zero level gradually, which will cause the PINN optimization problem to be extremely ill- ill-conditioned. Therefore, the amount of weight change can quickly drop below machine with FP32 training, since the weight change positively correlated with gradient.
5.3 Harder Equations Require Higher Arithmetic Precision
has been shown that, for parameterized PDE, the PINN optimization of the equations becomes progressively more difficult as the parameters of the equations become larger, and even failure modes occur (20) 71. The reason for this phenomenon is that the frequency of model changes will become higher, and the pattern of the model will become more versatile. We attribute this to the fact that as the parameter of the equation gets larger, the separation of its error and loss landscapes also gets progressively larger. This means that for more difficult problems with larger parameter, an early stopping point the optimization process will result in a larger error. As shown Fig, (a), we confirm this problem's 's complex trend with varying degrees of arithmetic precision from BF16 to FP64. We find that for more complex equation problems (like 3 100), increasing the arithmetic precision of the model to double precision effectively solve the failure modes of PINN.

---

Table 2: Comparison of FP32 and FP64 configurations in terms of relative RMSE, training time iteration, and memory usage across PDE benchmarks.
FP32
Equation
rRMSE
Training Time/Iter (s)
Memory (MB)
convection
0.7640
0.29
1609
reaction
0.9778
0.26
1629
0.2837
0.47
2295
Allen-Cahn
0.9662
0.40
1975
FP64
Equation
rRMSE
Training Time/Iter (s)
Memory (MB)
convection
0.0072
0.32
2441
reaction
0.0502
0.29
2481
wave
0.0081
0.61
3845
Allen-Cahn
0.0545
0,48
3167
5.4 Arithmetic Precision's Impact Various Model Architectures.
In order to verify whether the multiple understandings of model architectures 36 35
claiming to address PINN failure modes intrinsic, and whether our proposed method for improving numerical precision is orthogonal to other methods, as shown in Fig. g we evaluate lot of models on convection equations with parameter 3 being PINNsFormer and PINNMamba (from FP32 to TF32) can make the failure modes reproduce, which suggests that the method of constructing inductive bias based on the intrinsic solution to the failure modes problem. Using double- precision training, all models converge the correct solution, which illustrates that our understanding of loss landscape and model precision holds the correctness, generalizability, and orthogonality to other methods.
50. We find that just slight decrease the precision of
Hypothesis is not an
double-precision
5.5 Discussion of Computation
evaluate the FP32 and FP64 training overhead on a Nvidia H100 PCIe GPU, The FP64 latency per iteration is roughly 1.1-1.3 times that of FP32, and the GPU memory footprint about 1.5-1.7 times that of FP32. This is due the fact that, with Tensor Core, the FP64 peak arithmetic on H100 PCIe is 60 TFLOPS, the same as the FP32 peak arithmetic. As result, FP64 is not very much slower than FP32. We observe about 10-30% more latency of FP64 than FP32 that we believe is due to larger memory accesses. This indicates that training with FP64 does incur an increase in computational overhead, but under modern GPU architectures the increase is marginal.
As shown in Table
Conclusion
In this paper, through hypothesis testing on the loss landscape of PINN, demystify that the real cause of PINN's failure modes the early stopping of the optimization process. We reveal that this early stopping phenomenon can be easily solved by improving the training arithmetic precision of the model. Training with double precision will enable the vanilla PINN to beat state-of-the art PINN model architectures various PDE problem settings. This prompts the broader AI for Science community that PINN is still essentially a numerical solution method for PDEs, and that computational precision plays just crucial role it does in traditional methods.
Acknowledgements
This work is supported, in part. by the National Science Foundation and the Institute of Education Sciences under Grant 2229873 (AI4ExceptionalEd), National Science Foundation under Grant 2235364 (FuSe-TG), and SUNY-IBM AI Collaborative Research Award. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the sponsors.
10

---

References
[1] S. Ainsworth, J. Hayase, and S. Srinivasa. Git re-basin: Merging models modulo permutation symmetries. In The Eleventh International Conference on Learning Representations, 2023.
[2] S. Basir and I. Senocak. Critical investigation of failure modes in physics-informed neural networks. In AiAA SCITECH 2022 Forum, page 2353, 2022.
[3] F. Bassi and S. Rebay. A high-order accurate discontinuous finite element method for the numerical solution of the compressible navier-stokes equations. Journal of computational physics, 131(2)2677279) 1997.
[4] A. G. Baydin, B. A. Pearlmutter, A. A. Radul, and J. M. Siskind, Automatic differentiation in machine learning: a survey. Journal of machine learning research, 18(153):1-43, 2018.
[5] C. G. Broyden. A class of methods for solving nonlinear simultaneous equations. Mathematics of computation, 19792)5577-593 1965.
[6] J. Bu and A. Karpatne. Quadratic residual networks: A new class of neural networks for solving forward and inverse problems in physics involving pdes. In Proceedings of the 2021 SIAM International Conference on Data Mining (SDM). pages 675 683. SIAM, 2021.
[7] W. Cho, M. Jo, H. Lim, K. Lec, D. Lee, S. Hong, and N. Park. Parameterized physics-infrmed neural networks for parameterized pdes. In International Conference on Machine Learning, pages 85 15-8833.. PMLR, 2024.
[8] J. Choquette. Nvidia hopper h100 gpu: Scaling performance. IEEE Micro, 43(3):9 2023.
[9] A. Daw, J. Bu, S. Wang, P. Perdikaris, and A. Karpatne. Mitigating propagation failures in thysics-infommn neural networks using retain- www.-ree-aaeeeeeeee release (13) sampling. In International Conference Machine Learning, pages 7264 7302. PMLR, 2023.
[10] M. Fishman, B. Chmiel, R. Banner, and D. Soudry. Scaling fp8 training to trillion-token 1lms. In Proceedings of the International Conference Learning Representations (ICLR), 2025.
[11] J. Frankle, G. K. Dziugaite, D. Roy, and M. Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference Machine Learning, pages 3259-3269. PMLR. 2020.
[12] Z. Gao, L. Yan, and T. Zhou. Failure-in informeed adaptive sampling for pinns. SIAM Journal on Scientific Computing, 45(4):A1971 AA9994 2023.
[13] A. Gholami, S. Kim, Z. Dong, Z. Yao, M. W. Mahoney, and K. Keutzer. A survey of quantization methods for efficient neural network inference. arXiv preprint arXiv:2103. 103.133033 2021.
[14] S. Gupta, A. Agrawal, K. Gopalakrishhan, and P. Narayanan. Deep learning with limited numerical precision. In International conference on machine learning, pages 1737-1746. PMLR, 2015.
[15] Z. Hao, C. Ying. H. Su, J. Zhu, J. Song, and Z. Cheng. Bi-level physics-informed neural networks for pde constrained optimization using broyden's 's hypergradients. In International Conference on Learning Representations, 2023.
[16] K. Hornik, M. Stinchcombe, and H. White. Multilayer feedforward networks are universal approximators. Neural networks, 2(5):359-366 1989.
[17] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint 141. 2014.
[18] E. Kiyani, K. Shukla, J. E. Urbán, J. Darbon, and G. E. Karniadakis. Which optimizer works best for physics
informed -oysicc-informed 2025.
neural networks and kolmogorov- networks? arXiv preprint 2
[19] D. A. Kopriva. Implementing spectral methods for partial differential equations: Algorithms for scientists and engineers. Springer Science & Business Media, 2009.
[20] A. Krishnapriyan, A. Gholami, S. Zhe, R. Kirby, and M. W. Mahoney. Characterizing possible failure modes in physics-informed neural networks. Advances in neurul information processing 26560, 2021.
processnn systems. 34:26548-
[21] D. C. Liu and J. Nocedal, On the limited memory bigs method for large scale optimization. Mathematical programming, 45(1):503-528 1989.

---

[22] Z. Liu, Y. Wang. S. Vaidya, F. Ruehle, J. Halverson. M. Soljacic, T. Y. Hou, and M. Tegmark. KAN: Kolmogorov-ariold networks. In The Thirleenth International Conference on Learning Representations, 2025.
[23] P. C. Nguyen, X. Cheng, Azarfar, P. Seshadri, Y. T. Nguyen, M. Kim, Choi, H. Udaykumar. and S. Back. Parev2: Physics-aware recurrent convolutional neural networks for spatiotemporal dynamics modeling. In International Conference on Machine Learning, pages 37649-37666. PMLR, 2024.
[24] Nocedal and Y-xx. Yuan. Analysis of self-scaling quasi-newton method. Mathematical Programming. 61(1):19-35 37, 1993.
[25] A. Paszke, S. Gross, Chintala, Chanan, Yang. Z. VeVito, Lin, Desmaison, Antiga, and
A. Lerer. Automatic differentiation in pytorch. 2017.
[26] A. Paszke, S. Gross, Massa, Lerer. J. Bradbury, G. Chanan, Killeen, Lin, N. Gimelshein.
L. Antiga, A. Desmaison, A. Kopt. E. Yang. Z. VeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner,
L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high hegh-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Bue, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[27] M. Raissi, P. Perdikaris, and G. E. Karniadakis. Physics-informed informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686-707, 2019.
[28] P. Rathore, W. Lei, Frangella, L. Lu, and Udell. Challenges in training pinns: loss landscape perspective. arXiv preprint ar Xiv:2402.0 2222402201866 868, 2024.
[29] Wang. S. Li, D. He, and L. Wang. Is 12 physies informed loss always suitable for training physics informed neural network? Advances in Neural Information Processing Systems, 35:8278-8290, 2022.
[30] S. Wang, B. Li, Y. Chen, and P. Perdikaris. Piratenets: Physics-informee deep learning with residual adaptive networks. Journal of Machine Learning Research, 25(402):1 51, 2024.
[31] P. Wolfe. Convergence conditions for ascent methods. SIAM review, 11(2):226-53) 1969.
[32] C. Wu, M. Zhu, Q. Tan, Y. Kartha, and L. Lu. A comprehensive study of adaative and residual-based adaptive sampling for physies-informed
informed neural networks. Computer Methods in Applied Mechanics and Engineering, 403:1 15671, 2023.
[33] H. Wu, H. Luo, Ma, J. Wang, and Long. Ropinn: Region optimized physics-informed neural networks. arXiv preprint arXiv:2405. 14369, 2024.
[34] H. Wu, Y. Ma. H. Zhou, H. Weng, J. Wang, and M. Long. Propinn: Demystifying propagation failures physics-informen neural networks. arXiv preprint arXiv:2502.00800 2025.
[35] C. Xu, D. Liu, Y. Hu, J. Li, R. Qin, Q. Zheng, and J. Xiong. Sub-sequential physics-inforit learning with state space model. In International Conference on Machine Learning. PMLR, 2025.
[36] Z. Zhao, X. Ding, and A. Prakash. Pinnsformer: transformer- framework for physics-informed neural networks. In The Twelfth International Conference on Learning Representations, 2024.
[37] N. Zmora, Wu, and Rodge, Achieving fp32 accuracy for int8 inference using quantization aware training with NVIDIA TensorRT, 2021.
12

---

A PDE Setups
A.1 1-D Convection
The oneddimmalional convection (or advection) equation characterises the transport of a scalar field u(x, suchuch as temperature, concentration, or momentum under a uniform velocity B. Widely studied in fluid dynamics and transport theory, it is expressed
du du Ōt
+ B
0, Væ (0,2 [0,
Ox
w( 0) sin a, Ve E 2T,, w0, 1) = uu2 f), E [0, 1].
(S)
where B denotes the constant advection speed. Larger values of 3 increase the difficulty for PINNs, making this equation common benchmark with known failure modes. Following prevailing prac- tice 36 33,, we set 3 = 50.
Its analytical solution is
Uana (I, 1) = sin(x -8t).
(6)
A.2 1-D Reaction
The dimennsnnio reaction equation models the temporal evolution a reacting species along a single spatial dimension:
du 06
pu(1 u) 0, Vo E [0, 2m), E 1],
u1x 0) expl (x
T)2 T1-
Va [0, 2T),
2(
(0, u(2x, f), VI E [0, 1].
with p representing the growth-rate coefficient. Increasing likewise poses greater challenges for
PINNs; we adopt the standard choice = 5 in accordance with [36 33]. The closed-form solution is
cxp
(x--))² 2(1/1)2)
exp(pt)
Mana
2 2(T/4)² (exp(pl) - 1) +1
(8)
exp
A.3 1-D Wave
The one me-diimnsioona wave equation describes wave propagation for example, vibrations on string-ane and in our study takes the form
8u
8u 01 0, Vx E 10, e [0,
012
u(x, 0) sin(TT)) + sin(3 Vo E [0, 1),
(9)
Ōu(x, 0) = 0, Vx E [0, 1],
Øt
w00,t) = (1,, = 0, E [0, 1].
where 3 controls the second harmonic; we set second-order derivatives and the initial condition contains first-order derivatives, optimisation is notoriously difficult [33]. Nevertheless, this case demonstrates that PINNMamba's matrix-defined time differentiation with with uniform scaling across derivative orders better capture temporal dynamics.
following [36 33). Because the PDE involves
Its analytical solution is
Mona (x, t) sin(TT.) cos(oortt (2rt) + sin(Sna) cos (23xt)).
(10)
13

---

A.4 Allen-Cahn
The Allen-Cahn equation a canonical reaction reaction-diffusion benchmark:
Du
Ou Or² 5u 5u 0, a (-1,1), te (0,1),
0.0001
uxx 0) = x² cos((T)), x _E [-1,1)
(11)
((-1,,t) u(1,t), t [0.1].
du(-1, 0u(1,t) te [0, 1]. dr
dr
Because no closed-form analytic solution exists, we adopt a high-resolution spectral approxima- tion [9] reference data, following (27 (4). The sharp interface and double entrance makes this PDE a widely used stress test for PINNs and a representative failure mode.
B L-BFGS Optimizer
For an unconstrained, twice-differentibble objective f:: R + RR, Newton's 's method uses the exact Hessian H₁ 22 ((xx) to obtain the step
I - H fxxx),
(12)
yielding quadratic local model and (under standard assumptions) quadratic convergence. Quasi- Newton methods avoid forming or inverting the expensive H₃ by maintaining an approximation B& s Hy (or its inverse Hg ss B that is updated from first-order information only. The most successful member is the BFGS update
Ha+1 (1 -1--pss (P.Pk) H4 Pk3)U5)) + PESASK Pk
(13)
yg 8k
with S4 Tx+1 FK and yk super-linear local convergence, but it stores a dense n x n matrix requiring Onn?) memory and time per iteration-makinn it impractical when is large.
vf(=++1) Vx(x)). BFGS enjoys global convergence and matrix-reqiring
L-BEGS eliminates the quadratic storage by discarding all but the most recent m curvature pairs {(s; k-1 Instead of storing H₃ explicitly, it represents the inverse Hessian implicitly through
these vectors and a diagonal scaling He (often 7 with 7/
-2- 196-1
2/ 99--1
In compact format, let:
St 3 (sk-m 88-1) 8k-1]
E Raxm Y8 14k ms ... M.k-l)
R2 S.Y, Yu (upper-triangular),
(14)
D₁ = diag(s ( -m ......... 1
(15)
oh-1 the inverse Hessian can be
With an initial diagonal scaling H = T (usually written without loss of information as
" 2/
H₂ H + [& HYYK]
R R--(D) T(D + +YTTH
R
ST Y H
(16)
R
The memory requirement reduces to (mm)) floats, while the computational cost per iteration becomes ((n)) flops. This scaling is linear in the number of parameters when is fixed, with typical values e [5, 20).
14

---

C Broader Social Impact
Making FP64 the default for PINNS turns a previously brittle solver into a dependable scientific tool: the same vanilla architecture that failed at FP32 converges robustly on convection, reaction, wave and Allen-Cahn equations when upgraded to FP64. Reliable neural PDE solvers promise faster, mesh-free simulations for climate modelling, renewable-energy design, and personalised biomedical devices, potentially shortening innovation cycles and widening access to high-fidelity analysis in domains where traditional finite-llemett codes are prohibitively slow.
The work also reframes precision first-class hyper-paramettr in numerical machine learning. aligning PINNs with classical finite-element practice where double precision is standard for safety- critical engineering. This alignment reduces the risk of deploying under-resolved models in high- stakes settings. Yet the same fidelity could be misapplied- for instance, to accelerate the design of destructive fluid-dynamic systems. We therefore release code under a license prohibiting military applications and encourage transparent reporting of energy usage and precision settings to foster reproducibility and responsible adoption.
D Limitations
However, double precision is more energy-intensive and currently restricted to high-end hardware such as NVIDIA's H100 GPUs, on which our experiments were run. Increased demand for FP64 compute could raise the carbon footprint of scientific machine learning and exacerbate resource inequity between well-funded labs and those with limited budgets. Conversely, our findings may motivate handware-software co-design efforts that deliver energy-efficient FP64 pipelines, ultimately mitigating this environmental cost.
15

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We attribute the failure modes of PINN to insufficient arithmetic precision, and analyzed the loss landscape of PINN. We include that the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
16

---

Justification:
We create section in Appendix to discuss the limitation of the paper.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
No theory in mathematical format is proposed in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide an anonymous link the code.
Guidelines:
The answer NA means that the paper does not include experiments.
17

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. w
For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We provide an anonymous link the code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ 7/nips. /ttppp.cc/pbliccguideeee http://oeeuuiiisssoooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
18

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
In section 3.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We include the CI in the main results in Table
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
show in tables
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Section experiments be conduct on any GPU with double precision support, but Nvidia A 1100 or H100 is recommended.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
19

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttgs///eerrrss..pppbbbbbbiiiie
Answer: [Yes]
Justification: Yes.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In Appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
20

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
Use the code of PINNMamba, which is CC-BY 4.0 and MIT.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects
21

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: http:::///euripp cc/ //oofffrencees/2255555 for what should or should be described.
22


Reviews:
[{"review_id": "rVIEwVl8Iu", "reviewer": "Reviewer_y1Ny", "summary": "The paper analyzes the failure modes of training physics-informed neural networks. It finds that the reason for failure in the loss function is precision limitation rather than getting stuck in local minima during optimization. To achieve this new insight, the authors propose the same-basin hypothesis of the loss landscape, where the correct values exist in the same basin, but since the loss landscape becomes flat due to precision limitations, optimization comes to a halt. This hypothesis is supported by analyses of physics-informed neural network training with various initializations and the absence of loss barriers from failure mode to optimization approach. To address this issue, the paper suggests using FP64 instead of FP32. Moreover, the paper demonstrates the efficacy of FP64 in solving more challenging equations with physics-informed neural networks.", "questions": "Why does the PINNMamba model in the reaction benchmark show better performance compared to the PINN_FP64 model?", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n\nThe analysis of the loss landscape of physics-informed neural networks and the same-basin hypothesis is interesting and novel.\n\nThe paper is well-written and well-organized.\n\nWeaknesses: \n\n1- The overhead of using FP64 for training physics-informed neural networks is not explored quantitatively. For example, this overhead could be evaluated on a simple model that works with both FP32 and FP64 on GPU.\n\n2- The FP64 solution proposed by the author is a rudimentary approach and moves in the opposite direction of optimizing neural networks toward low-precision arithmetic. The author should analyze the gradients, weights, and activation distributions more thoroughly and demonstrate whether simple approaches, such as FP32 with scaling factors or grouping parameters, work or not. The author can also draw inspiration from many approaches used for FP16 and BF16 training such as stochastic rounding and apply them to FP32 for PINNs.", "was_revised": true, "final_justification": "The authors evaluate the overhead of FP64 training compared to FP32 training on various GPUs and show that the overhead is negligible. Various quantization approaches, such as using scaling factors and stochastic rounding, are also evaluated, but have shown limited progress. Although better quantization approaches might be found for this problem, the FP64 solution is sufficient as an introductory solution and opens valuable research directions for the future.", "rebuttal": "We sincerely appreciate the constructive comments from reviewer y1Ny and the time spent on reviewing this paper. We address the questions and clarify the issues accordingly as described below.\n\n>**[Weakness 1]**:  The overhead of using FP64 for training physics-informed neural networks is not explored quantitatively. For example, this overhead could be evaluated on a simple model that works with both FP32 and FP64 on GPU.\n\n**[Response to W1]**: Thank you for your comments. We evaluate the FP32 and FP64 training overhead on a Nvidia H100 PCIe GPU, the results are reported in the following table.\n\n|Equation|FP32 rRMSE|FP32 Training Time/Iteration (second)|FP32 Memory (MB)|FP64 rRMSE|FP64 Training Time/Iteration (second)|FP64 Memory (MB)|\n|---|---|---|---|---|---|---|\n|convection| 0.7640 | 0.29 | 1609| 0.0072 |0.32 | 2441|\n|reaction|0.9778|0.26|1629|0.0502|0.29|2481|\n|wave|0.2837|0.47|2295|0.0081|0.61|3845|\n|Allen-Cahn|0.9662|0.40|1975|0.0545|0.48|3167|\n\nBy observation, we find that on H100 PCIe, the FP64 latency per iteration is roughly 1.1-1.3 times that of FP32, and the GPU memory footprint is about 1.5-1.7 times that of FP32. This is due to the fact that, with Tensor Core, the FP64 peak arithmetic on H100 PCIe is 60 TFLOPS, the same as the FP32 peak arithmetic. As a result, FP64 is not very much slower than FP32. We observe about 10-30% more latency of FP64 than FP32 that we believe is due to larger memory accesses.\n\nAlso, we note that on the Nvidia A6000 GPU, the latency of FP64 is about 10 times that of FP32. This is due to the fact that only few GPU types support high-performance double-precision computation. GPUs known to be suitable for double-precision computing contain the Nvidia A100/A800, H100/H200, and B200.\n\n>**[Weakness 2]**: The FP64 solution proposed by the author is a rudimentary approach and moves in the opposite direction of optimizing neural networks toward low-precision arithmetic. The author should analyze the gradients, weights, and activation distributions more thoroughly and demonstrate whether simple approaches, such as FP32 with scaling factors or grouping parameters, work or not. The author can also draw inspiration from many approaches used for FP16 and BF16 training such as stochastic rounding and apply them to FP32 for PINNs.\n\n**[Response to W2]**: We agree that the current trend in AI is towards efficient low-precision computation. The development of Quantization techniques has played an important role in areas such as NLP, CV, and especially LLM. But with this paper we would like to draw the attention of the AI community to the possible role of numerical precision. As an example, Pytorch defaulted to matmul.allow_tf32 = True in version 1.7, which is feasible for general AI computational problems, but fatal for scientific computation.\n\nWe did an analysis of the gradient, weights for the PINN model under multiple precision runs. We plotted visualizations like Fig. 7(b),(c) for the variation of statistics such as the norm, mean, and variance of the gradients and weights by iteration. Unfortunately, due to the new NeurIPS rebuttal rules this year, we cannot provide these visualizations at the rebuttal stage. We will put up these visualizations in the next release. What we have found in these visualizations is that at low precision, the gradient goes to zero prematurely, resulting in the weights not changing, which is consistent with our analysis.\n\n\nWe take the convection equation as an example and append some common ways to change the scale of weight and gradient as shown in the table below. The model is trained with FP32 precision. We find that layer normalization, simply stochastic rounding from FP32 to FP64, and scaling factors do not inherently improve the failure modes of PINN. We believe that the core reason for the failure of these methods is that the gradients on spacetime are involved in the computation of residual loss in PINN, which leads to the failure of the scaling methods in terms of the higher-order derivatives taken when the gradients continue to be solved for these differentials. The failure of the simple quantization approach that we tested indicates that for the successful operation of PINNs at low precision, it is still a new area to explore and a new research direction.  \n\n|Method|Training Loss|rMAE|rRMSE|\n|---|---|---|---|\n|Original FP32 Training|0.0133|0.6904|0.7640|\n|Layer Normalization|0.016598|0.7775|0.8394|\n|Scaling Factors (1e7)|0.014238|0.7431|0.8125|\n|Stochastic Rounding|0.013105|0.7178|0.7903|\n|FP64 Training|5e-6|0.0059|0.0072|\n\n\n\n\n>**[Question 1]**: Why does the PINNMamba model in the reaction benchmark show better performance compared to the PINN_FP64 model?\n\n**[Response to Q1]**: A core contribution of our paper is that we understand PINN failure modes from the perspective of PINN optimization. Our theory illustrates that the nature of PINN failure modes is an optimization problem in the flat plane on the loss landscape. Training with FP64 is one perspective to improve optimization, meanwhile, good model architecture, good sampling method, good optimizer, etc. are all ways to improve optimization. Although there is a slight numerical difference, both FP64 and PINNMamba solve the failure modes of PINN, while our method is more straightforward and has stronger generalization ability than PINNMamba. PINNMamba has a more optimizable model architecture than PINN, and we illustrate in Fig. 8 (b) the combined effectiveness of PINNMamba ( as well as other model structures) and FP64, which are two independent perspectives to improve the optimization of PINN, and can be used in conjunction with each other. Our main thrust in this paper is to make the AI-based scientific computing community more aware of the impact of numerical precision."}, {"review_id": "FIr4tbcJbG", "reviewer": "Reviewer_R6jD", "summary": "This paper proposes solving physics informed neural networks in double precision (fp64) instead of the standard float (fp32) or half precision (fp16). Experiments demonstrate that in double precision, many PINNs that typically diverge, instead stably converge to very accurate solves.", "questions": "Would it be possible to successfully train PINNs using autocasted fp64 (much like how many torch models are trained using automatic mixed precision between fp32 and fp16/bf16)? E.g. https://docs.pytorch.org/docs/stable/amp.html\nWhile the loss surface analysis is quite interesting, what’s missing is “why” the gradient basins seem so difficult to get underneath / the flatness of the loss landscape. Could the authors display differences in weight matrices in fp32 and fp64 that display what’s going on? Is it just one specific layer or is it all layers, etc.?\nAlternatively, is it the imprecision in gradient updates or weight matrices, etc. that are causing the failure modes?\n\nHow much slower and less memory efficient is training directly in fp64 instead of fp32? I understand this is most likely 2x the memory and something like 3x slower on the H100s that you used.\n\nDo all of the methods (KAN, PINNMamba, PirateNet, etc..) also work better when run in double precision or is it just PINN in Table 1? I see that there are some clear precision differences in one problem on Figure 8b but don’t really know how that maps.\nGiven Figure 7b) would a weight normalization or other normalization layer instead improve the training stability of these models?\n\nIs the failure phase in the loss dynamics due to imprecision in the loss function itself? That is, it seems like the model doesn’t generalize well at that point, but is still able to produce very low loss values? What loss function is typically used to train these models?\n\nAre there still cases when fp64 is not enough precision? I know that there are some (mostly Julia unfortunately https://github.com/JuliaMath/DoubleFloats.jl) packages for even higher precision; will these perform even better in MSE? The one missing thing is a bit of a show-stopping experiment – using double allows solving a very difficult scientific PDE that was previously unsolveable in fp32…\n\nWriting / presentation comments:\n\n-\tFigure 4: no need to title those plots in matplotlib, and keep the same theme for a/b. Then editorialize the caption some more (also same in Figure 3).\n\n-\tEditorialize the caption in Figure 1.\n\n-\tL65: “revolutionize” is a bit oversold. Let’s tone that down.\n\n-\tL303: “we demystify that the real cause”  “we show that …” and then “This early stopping …”\n\n-\tL306: “will enable” - enables\n\n-\tL308: honestly, I’m not sure what you’re trying to say here; just delete that sentence.\n\n-\tL145: “is then been overthrown” -> rewrite to “[33-36] show that there is near-zero empirical residual loss in many PINN failure modes.”\n\n-\tL144: “[20] attributed PINN failures to the “optimizer [getting] stuck … “\n\n-\tYou probably don’t need both Figures 2 and 6; suggest moving Figure 6 to the front and deleting Figure 2. Then you can editorialize the caption in Figure 2 a lot more.", "limitations": "n/a", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\nI appreciate the simplicity of this paper – simply running everything in double makes things work better.\n\nI appreciate the loss landscape analysis quite a lot as starts pointing out descriptively out to “why” the optimization fails. Furthermore, I like the “non-existent” loss barrier experiment as it clearly shows that the failure is somehow that of optimization (even the second order optimizers aren’t really able to push through).\n\nI like the difficulty experiment (section 5.3) as well.\n\nWeaknesses:\n\nThe paper is a bit too descriptive in nature – there’s not enough “why” experiments pointing out “why” double precision manages to solve things. What should we do when we try this and things still fail to converge (interestingly, we would need harder benchmarks as a result)?\nUnfortunately, the experiments are only on one PINN benchmark and not most of them, limiting the generality of the paper.\n\nFigure 5 just shows the optimization landscape becomes very flat for those specific architectures, not that the base PINN also has this issue. Instead showing the PINN loss curve in fp32 and fp64 is probably more interesting.\n\nThere's unfortunately a fair number of weird grammatical issues that detract from the meaning of the paper a bit. I've caught a bunch in the questions section, but don't think that's all of them.", "was_revised": true, "final_justification": "After rebuttal, I raise score to accept. My questions are answered quite well.\n\nOn the H100s, the tensor core fp64 should make things not too slow as compared to fp32. This paper opens an interesting line of research - both practical (how to best run high precisions and emulate them on new gpus) and theoretical (why does gradient underflow occur).", "rebuttal": "We sincerely appreciate the constructive comments from reviewer R6jD and the time spent on reviewing this paper. We address the questions and clarify the issues accordingly as described below.\n\n\n>**[Weakness 1]**: The paper is a bit too descriptive in nature – there’s not enough “why” experiments pointing out “why” double precision manages to solve things. What should we do when we try this and things still fail to converge (interestingly, we would need harder benchmarks as a result)? Unfortunately, the experiments are only on one PINN benchmark and not most of them, limiting the generality of the paper.\n\n**[Response to W1]**: We attributed the double-precision problem solving in Section 5.2 to the optimizer's premature determination of model convergence, and we proved our point by analyzing the model's machine epsilon, an analysis whose soundness is backed up by the experiments shown in Figure 7. In further experiments, we determined the variation of computation effective numerical precision with iteration in each benchmark equations. We made some visualization plots, but due to the new rules of NeurIPS rebuttal, we could not present them in the rebuttal stage.  So we do some description here. We found that the computation effective numerical precision of the model will be lower than the machine epsilon after a certain stage of the optimization, accompanied by the gradient of the model being underflowed to 0 in the corresponding case, and thus the weights of the model stop being updated. We guarantee that these new “why” experiments will be presented in camera-ready.\n\nFor more challenging equations that are harder to optimise (e.g., a convection equation with an extremely large $\\beta$ parameter, such as 200), double precision alone cannot always eliminate the failure modes. Fortunately, our method is orthogonal to all existing remedies for PINN failure modes, so it can be combined with them to further improve the model. We demonstrate the effectiveness of such a combination with a stronger PINN architecture in Section 5.4 and Fig. 8 (b).\n\nOn the other hand, for even tougher cases, increasing the arithmetic precision further may resolve the problem; Fig. 8 already shows this trend, and we expect it to generalise to higher precisions. Although mainstream deep-learning frameworks—PyTorch, for example—do not yet support precisions beyond FP64, we are actively working on extending PyTorch and CUDA to higher precisions. Our solution is designed to scale: the new data-type infrastructure can be extended to 128-, 256-bit, or even higher precision. This research is under way, and we plan to release and open-source this support to the community in the near future. We will promptly evaluate the PINN model’s performance at higher numerical precision on more challenging problems and leave this as future work.\n\nIn our follow-up work, we ran experiments analogous to those in Figures 4, 5, and 7 on all the additional benchmarks we evaluated. The outcomes were consistent with the conclusions reported in the paper. We will add these results in the next release.\n\n\n\n>**[Weakness 2]**: Figure 5 just shows the optimization landscape becomes very flat for those specific architectures...\n\n**[Response to W2]**: For the loss curves of base PINN, we have presented in Fig. 1, on both FP32 and FP64. The purpose of Fig. 5 is to show what kind of curve a successful optimization of the PINN problem should present.\n\n>**[Weakness 3]**: There's unfortunately a fair number of weird grammatical issues that detract from the meaning of the paper a bit. I've caught a bunch in the questions section, but don't think that's all of them.\n\n**[Response to W3]**: We thank you very much for these comments. We will carefully check the text/grammar word by word before next release. For the captions of individual figures, we will carefully verify and provide self-contained captions. \n\nFor example, we will delete the Fig.2 and move Fig. 6 to the front, then add a caption to describe which are failure modes.\n\n>**[Question 1]**: Would it be possible to successfully train PINNs using autocasted fp64 (much like how many torch models are trained using automatic mixed precision between fp32 and fp16/bf16)? While the loss surface analysis is quite interesting, what’s missing is “why” the gradient basins seem so difficult to get underneath / the flatness of the loss landscape. Could the authors display differences in weight matrices in fp32 and fp64 that display what’s going on? Is it just one specific layer or is it all layers, etc.? Alternatively, is it the imprecision in gradient updates or weight matrices, etc. that are causing the failure modes?\n\n**[Response to Q1]**:Unfortunately torch.autocast doesn't support FP64 at the moment. Our team is working on this.\n\nWe provide the norms of weights and gradients for FP64 training in Figure 7. We also plot the statistics (mean, std) of the weights and gradients at different numerical precisions. The differences in weights and gradients are reflected on all layers, not some specific layers. These plots will be inserted into the appendix in camera-ready.\n\nBased on the evidence and analysis, we believe that the imprecision in gradient updates is the main reason for the failure modes. Of course, the weights also change due to the imprecision in gradient updates, and the cumulative effect leads to a large difference between the weight matrices of FP32 and FP64. But this difference is not reflected in the norm and statistics of the weights.\n\n\n>**[Question 2]**:How much slower and less memory efficient is training directly in fp64 instead of fp32? I understand this is most likely 2x the memory and something like 3x slower on the H100s that you used.\n\n**[Response to Q2]** We evaluate the FP32 and FP64 training overhead on a Nvidia H100 PCIe GPU, the results are reported in the following table.\n\n|Equation|FP32 rRMSE|FP32 Training Time/Iteration (second)|FP32 Memory (MB)|FP64 rRMSE|FP64 Training Time/Iteration (second)|FP64 Memory (MB)|\n|---|---|---|---|---|---|---|\n|convection| 0.7640 | 0.29 | 1609| 0.0072 |0.32 | 2441|\n|reaction|0.9778|0.26|1629|0.0502|0.29|2481|\n|wave|0.2837|0.47|2295|0.0081|0.61|3845|\n|Allen-Cahn|0.9662|0.40|1975|0.0545|0.48|3167|\n\nBy observation, we find that on H100 PCIe, the FP64 latency per iteration is roughly 1.1-1.3 times that of FP32, and the GPU memory footprint is about 1.5-1.7 times that of FP32. This is due to the fact that, with Tensor Core, the FP64 peak arithmetic on H100 PCIe is 60 TFLOPS, the same as the FP32 peak arithmetic. As a result, FP64 is not very much slower than FP32. We observe about 10-30% more latency of FP64 than FP32 that we believe is due to larger memory accesses.\n\nAlso, we note that on the Nvidia A6000 GPU, the latency of FP64 is about 10 times that of FP32. This is due to the fact that only few GPU types support high-performance double-precision computation. GPUs known to be suitable for double-precision computing contain the Nvidia A100/A800, H100/H200, and B200.\n\n>**[Question 3]**: Do all of the methods (KAN, PINNMamba, PirateNet, etc..) also work better when run in double precision or is it just PINN in Table 1? ...\n\n**[Response to Q3]** The boost of raising to double precision for the listed baseline model architectures is consistent across these failure mode cases. We provide experimental results on wave equation in the following table. All the models run with double precision.\n\n\n|Model|Loss|rMAE|rRMSE|\n|---|---|---|---|\n|QRes|4.4e-5|0.0092|0.0093|\n|PINNsFormer|4.8e-5|0.0100|0.0102|\n|KAN|3.7e-5|0.0071|0.0072|\n|PirateNet|6.8e-5|0.0135|0.0138|\n|PINNMamba|3.3e-5|0.0062|0.0063|\n\nWe also run PINNMamba on FP64 across 4 benchmarks we use. \n\n|Task|Precision|Loss|rMAE|rRMSE|\n|---|---|---|---|---|\n|convection|FP32|0.0001|0.0184|0.0197|\n||FP64|4e-6|0.0042|0.0047|\n|reaction|FP32|1e-6|0.0092|0.0217|\n||FP64|1e-6|0.0081|0.0189|\n|wave|FP32|0.0002|0.0193|0.0195|\n||FP64|4.8e-5|0.0100|0.0102|\n|allen-cahn|FP32|0.0027|0.1432|0.2645|\n||FP64|9e-6|0.0114|0.0318|\n\nFrom the above two sets of experiments, we can assume that raising to double precision is an approach with strong generalization capabilities. The results of these experiments, as well as ongoing experiments on all baseline models over all failure mode equations, will be added in camera-ready.\n\nWe verified that layer normalization does not work. We speculate that this is due to the fact that the gradient is involved in the computation of the residual loss in PINN, which leads to a distorted mapping of the gradient to the physical information after normalization.\n\n|Method|Training Loss|rMAE|rRMSE|\n|---|---|---|---|\n|Original FP32 Training|0.0133|0.6904|0.7640|\n|Layer Normalization|0.016598|0.7775|0.8394|\n\n>**[Question 4]**: Is the failure phase in the loss dynamics due to imprecision in the loss function itself? ...\n\n**[Response to Q4]** As described in Sec 3.1 and Appendix A, the PINN loss function consists of the physics residual: the network’s predicted solution is substituted into the governing PDE and its initial/boundary conditions, the resulting residual is squared (or taken in absolute value) and summed/averaged over the collocation points, thereby minimizing deviation from the physical laws. The failure modes of PINN appear not only in the inference but also in the training stage, so it is not a generalization problem. This is due to the fact that the training loss of PINN does not directly reflect the absolute error of the solution, but rather the residuals of the governing PDE.\n\n>**[Question 5]**: Are there still cases when fp64 is not enough precision? ...\n\n**[Response to Q5]** Yes, the convection equation with beta=200 can not be solved with PINN with FP64. Unfortunately, these computational libraries for fp128 only support cpu, and as mentioned in response to W1, we are developing Pytorch support for higher precision. We see Allen-Cahn as a problem that FP32 failed to solve, but worked wonderfully well with FP64."}, {"review_id": "xW8Ze0213I", "reviewer": "Reviewer_QHbM", "summary": "This paper explores \"failure modes\" in physics-informed neural networks, a phenomenon commonly blamed for local optima separated from the true solution by steep loss barriers. The authors instead raises the \"same-basin hypothesis\" stating that the failure modes parameters are in the same loss basin as the true solution. The authors further analyze why failure modes exist with the commonly used L-BFGS optimizer, and find that the coarse machine precision of FP32 is to blame. The authors empirically demonstrate their hypothesis by replacing the FP32 training with FP64 training, and show that FP64 resolves all model's failure modes in a convection equation case.", "questions": "No.", "limitations": "No.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n- Clear motivation: the paper tries to understand the important and common phenomenon of failure modes in PINN training. The local optima hypothesis is questioned with clear evidence by shifting FP32 training to FP64 training, and the network is trainable again.\n- The authors deeply dig into the L-BFGS optimizer and provide clear explanations on the reason why FP32 is insufficient for training PINNs.\n- The experiments on some commonly used benchmarks of PINNs clearly show that simply shifting the FP32 training to FP64 training resolves the failure modes of vanilla PINNs, without modifications to the model architecture.\n\nWeaknesses:\n\n- The authors could consider apply the FP64 PINN training to some more complex PDEs, like Navier-Stokes equations, to better demonstrate the capability of PINNs with higher precisions.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the constructive comments from reviewer QHbM and the time spent on reviewing this paper. We address the questions and clarify the issues accordingly as described below.\n\n>**[Weakness 1]**: The authors could consider apply the FP64 PINN training to some more complex PDEs, like Navier-Stokes equations, to better demonstrate the capability of PINNs with higher precisions.\n\n**[Response to W1]**: We present some experimental results on 2D Naiver-Stoke and Fluid Dynamics in the table below. We find that FP64 consistently improves the performance of the model.\n\n\n\n\n|Problem|Precision|rMAE|rRMSE|\n|---|---|---|---|\n|2D Naiver-Stoke|FP32|13.08|9.08|\n||FP64|0.0397|0.0285|\n|Fluid Dynamics|FP32|0.3659|0.4082|\n||FP64|0.1475|0.1721|\n\nThe PDE problem are defined as following: \n\n2D Naiver-Stoke: \n\n\n$\\frac{\\partial u}{\\partial t}+\\left(u \\frac{\\partial u}{\\partial x}+v \\frac{\\partial u}{\\partial y}\\right)  =-\\frac{\\partial p}{\\partial x}+0.01\\left(\\frac{\\partial^2 u}{\\partial x^2}+\\frac{\\partial^2 u}{\\partial y^2}\\right)$\n\n$\\frac{\\partial v}{\\partial t}+\\left(u \\frac{\\partial v}{\\partial x}+v \\frac{\\partial v}{\\partial y}\\right)  =-\\frac{\\partial p}{\\partial y}+0.01\\left(\\frac{\\partial^2 v}{\\partial x^2}+\\frac{\\partial^2 v}{\\partial y^2}\\right) $\n\n$\n\\frac{\\partial u}{\\partial x}+\\frac{\\partial v}{\\partial y}  =0$\n\nwhere $u,v$ denotes the velocity on the x and y-axis respectively, and $p$ denotes the pressure field.\n\nFluid Dynamics:\n\n$\\frac{\\partial w}{\\partial t}+\\left(u \\frac{\\partial w}{\\partial x}+v \\frac{\\partial w}{\\partial y}\\right)  =\\frac{1}{\\operatorname{Re}}\\left(\\frac{\\partial^2 w}{\\partial x^2}+\\frac{\\partial^2 w}{\\partial y^2}\\right), \\quad(t, x, y) \\in[0, T] \\times \\Omega $\n\n$ \\frac{\\partial u}{\\partial x}+\\frac{\\partial v}{\\partial y}  =0, \\quad(t, x, y) \\in[0, T] \\times \\Omega $ \n\n$ w(0, x, y)  =w_0(x, y),(x, y) \\in \\Omega$\n\nwhere $u,v$ denotes the velocity  of the fluid on the x and y-axis respectively, $w$ is the vorticity, $T=10$ and $\\Omega = [0,2\\pi]^2$. $Re = 100$ is Reynolds number, $w_0$ is initial condition."}, {"review_id": "vEpXA3ohVi", "reviewer": "Reviewer_wrna", "summary": "This paper restudied the cause of PINN's training by L-BFGS optimizer is the early stopping of the optimization process.\n The low-precision such as FP32 causes the inner loop of L-BFGS optimizer early stopping.\nOn the other hand, by simple replacing FP 32 by double precision FP64 can surprisingly make vanilla PINN convergent for various PDE problems.", "questions": "1. Is the training failure caused by the L-BFGS optimizer? Does the same training failure exist for Adam optimizer of PINN?\n\n2. How does the training challenge for the long time prediction of PINN? This work only predict to time $t=1$, how about the long time prediction such as $t=10$, $t=50$?\n\n3. Although the four equations (convection equations, reaction equations, wave equations, and Allen-Cahn) have been widely studied in the PINN community, I don't think they are suitable for bechmarks nowdays. These four equations can be well-solved by PINN's variants, as well as traditional numerical methods. There exists PINN variants and training methods [1] which are superior than the baseline models compared in this paper. Complex PDE problems such as the Navier-Stokes flow with vortex, high dimensional PDEs, inverse PDE problems, etc, should be compared to show the improvements by using high-precision. \n\n[1]Wang S, Sankaran S, Wang H, et al. An expert's guide to training physics-informed neural networks[J]. arXiv preprint arXiv:2308.08468, 2023.", "limitations": "The limitations are stated in Appendix D.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The strengths of this paper is obvious, it improves the PINN training by L-BFGS optimizers simply and significantly.\nThe phenomenon is studied and analyzed adequately.\nHowever, my concern is whether double-precision changes the training failures by first order optimizers such as Adam.\nThe authors only concerned the L-BFGS optimizers on commonly studied PDE problems which have been well solved by PINN's variants.\nThese problems can also be easily solved by traditional numerical methods, and it can not reflect the advantages of PINNs.", "was_revised": true, "final_justification": "Most of my concerns have been resolved. The method is simple and effective, deserves to be seen in the PINNs' community. \n\nHowever, I think the widely used Adam optimizer deserves more exploration. The paper still need improvement on experiments. \n\nSo I change the rating from 3 to 4, without higher scores.", "rebuttal": "We sincerely appreciate the constructive comments from reviewer wrna and the time spent on reviewing this paper. We address the questions and clarify the issues accordingly as described below.\n\n>**[Weakness 1]**: However, my concern is whether double-precision changes the training failures by first order optimizers such as Adam. \n\n**[Response to W1]**: The PINN optimized by Adam using double precision has smaller loss, rMAE, and rRMSE than the single-precision case. This indicates that double precision is also useful for first-order optimizers. We show the results on convection equations optimized with Adam for 100000 iteration in the following table.\n\n|Optimizer|Precision|Training Loss|rMAE|rRMSE|\n|---|---|---|---|---|\n|Adam|FP32|0.095331|0.9022|0.9256|\n||FP64|0.000174|0.1363|0.1716|\n|L-BFGS|FP32|0.0133|0.6904|0.7640|\n||FP64|5e-6|0.0059|0.0072|\n\nDespite having lower loss and error, we find that the Adam-optimized model still fails to fully address failure modes with double-precision training.\nSeveral studies have demonstrated the superiority of second-order optimizers over first-order optimizers for PINN. [1,2] We strongly recommend using L-BFGS or Natural Gradient Descent for PINN optimization.\n\n\n\n>**[Weakness 2]**: The authors only concerned the L-BFGS optimizers on commonly studied PDE problems which have been well solved by PINN's variants. \n\n**[Response to W2]**: The resolving of PINN's variant for PINN failure modes is not intrinsic, as shown in Fig. 8(b), these PINN's variant still exhibit failure modes with reduced computational precision. The success of these variants is due to the introduction of some sort of prior, or inductive bias, into the model architecture of these variants. These prior and inductive bias are not necessarily generalizable. As shown in Table 1, these variants usually solve only part of the failure mode problems. \n\nThe contribution of our paper is not only to provide a solution to failure modes, but also, we provide an understanding of PINN failure modes. We show how failure modes of PINN should be properly understood from an optimization point of view. The PDE problems we have selected are all widely recognized as having failure modes.[4,5,6,7,9] PINN failure mode is a special phenomenon, usually with low training loss and high training error, which is different from typical optimization problems (high loss, high error) and generalization problems (low training loss, high inference loss). The PDE problems we study are highly representative of failure modes.\n\n\n>**[Weakness 3]**: These problems can also be easily solved by traditional numerical methods, and it can not reflect the advantages of PINNs.\n\n**[Response to W3]**: Our goal in this paper is not to show the advantages of PINN, but to address the real-world problems encountered by PINN. In terms of the advantages of PINN, PINN uses neural networks as a general function approximator to unify physical conservation and data supervision into a microscopic framework through automatic discretization, and thus is more flexible, mesh-saving, and easy to integrate compared to traditional numerical methods in complex geometries, real-time multiple solutions, and physics-data fusion scenarios. Despite these advantages, PINN still has some failure modes, and this paper aims to understand the fundamental reasons for these failure modes. \n\n>**[Question 1]**: Is the training failure caused by the L-BFGS optimizer? Does the same training failure exist for Adam optimizer of PINN?\n\n**[Response to Q1]**: The failure modes of the PINN are not caused by the L-BFGS optimizer, but the hard-to-optimize loss surface. The same training failure exist for Adam as well. In fact, the L-BFGS optimizer has a generally stronger performance on PINN than first-order optimizers such as Adam. The experiments results we have provided in the table above (response to W1) illustrates this.\n\n\n>**[Question 2]**: How does the training challenge for the long time prediction of PINN? This work only predict to time t = 1, how about the long time prediction such as t = 10 , t = 50 ?\n\n**[Response to Q2]**: PINN is agnostic to the length of time, no matter how long training interval t is. Our choice of training interval t = 1 is due to the normalization to the time coordinate, which can be done by a simple linear transformation for the system, and similarly, an inverse transformation can be performed to fit any time coordinate.\n\nThe standard PINN method does not support out-of-domain generalization for the time being, and the standard PINN trained on the interval of time $[0, 1]$ does not support generalization on the interval $[1, +\\infty]$. This generalization outside of the training domain has been studied in a few studies [3], but is beyond the scope of this paper, and we leave it for future work.\n\n>**[Question 3]**: Although the four equations (convection equations, reaction equations, wave equations, and Allen-Cahn) have been widely studied in the PINN community, I don't think they are suitable for bechmarks nowdays. These four equations can be well-solved by PINN's variants, as well as traditional numerical methods. There exists PINN variants and training methods which are superior than the baseline models compared in this paper. Complex PDE problems such as the Navier-Stokes flow with vortex, high dimensional PDEs, inverse PDE problems, etc, should be compared to show the improvements by using high-precision.\n\n**[Response to Q3]**: The object of this paper is PINN failure modes, and the four benchmarks we use in this paper are classical failure modes problems. These questions are also widely used in several other recent papers that study PINN failure modes.[4,5,6,7,9]\n\nThe core goal of this paper is not to reach a new state-of-the-art on benchmarks, but to present an understanding of the particular phenomenon of PINN failure modes. 2-D Navier-Stokes with vortex, Burger's equation, etc., for example, are not PINN failure modes problems, and standard PINNs can achieve good performance with the L-BFGS optimizer under these problems. Therefore, testing on these problems does not reflect the innovation of this paper.\n\nStill, we agree that it would be helpful to test the effectiveness of FP64 training on some complex problems. We present some experimental results in the table below. We find that FP64 still consistently improves the performance of the model.\n\n\n|Problem|Precision|rMAE|rRMSE|\n|---|---|---|---|\n|2D Naiver-Stoke|FP32|13.08|9.08|\n||FP64|0.0397|0.0285|\n|Fluid Dynamics|FP32|0.3659|0.4082|\n||FP64|0.1475|0.1721|\n\nThe PDE problem are defined as following: \n\n2D Naiver-Stoke: \n\n\n$\\frac{\\partial u}{\\partial t}+\\left(u \\frac{\\partial u}{\\partial x}+v \\frac{\\partial u}{\\partial y}\\right)  =-\\frac{\\partial p}{\\partial x}+0.01\\left(\\frac{\\partial^2 u}{\\partial x^2}+\\frac{\\partial^2 u}{\\partial y^2}\\right)$\n\n$\\frac{\\partial v}{\\partial t}+\\left(u \\frac{\\partial v}{\\partial x}+v \\frac{\\partial v}{\\partial y}\\right)  =-\\frac{\\partial p}{\\partial y}+0.01\\left(\\frac{\\partial^2 v}{\\partial x^2}+\\frac{\\partial^2 v}{\\partial y^2}\\right) $\n\n$\n\\frac{\\partial u}{\\partial x}+\\frac{\\partial v}{\\partial y}  =0$\n\nwhere $u,v$ denotes the velocity on the x and y-axis respectively, and $p$ denotes the pressure field.\n\nFluid Dynamics:\n\n$\\frac{\\partial w}{\\partial t}+\\left(u \\frac{\\partial w}{\\partial x}+v \\frac{\\partial w}{\\partial y}\\right)  =\\frac{1}{\\operatorname{Re}}\\left(\\frac{\\partial^2 w}{\\partial x^2}+\\frac{\\partial^2 w}{\\partial y^2}\\right), \\quad(t, x, y) \\in[0, T] \\times \\Omega $\n\n$ \\frac{\\partial u}{\\partial x}+\\frac{\\partial v}{\\partial y}  =0, \\quad(t, x, y) \\in[0, T] \\times \\Omega $ \n\n$ w(0, x, y)  =w_0(x, y),(x, y) \\in \\Omega$\n\nwhere $u,v$ denotes the velocity  of the fluid on the x and y-axis respectively, $w$ is the vorticity, $T=10$ and $\\Omega = [0,2\\pi]^2$. $Re = 100$ is Reynolds number, $w_0$ is initial condition.\n\nIn addition, we have some concerns about the generalizability of the methods in the paper [8], and the methods they propose do not essentially solve the failure modes. For example, for the convection problem, a periodic prior is introduced to the solution of the equations in [8], but considering that the analytical solution of this problem can be as simple as $u(x,t) = sin(x-\\beta t)$, it is clear that this is not generalizable, as well as that this prior assumption of periodicity is too strong.\n\n[1] Rathore P, Lei W, Frangella Z, et al. Challenges in training pinns: A loss landscape perspective. ICML, 2024.\n\n[2] Urbán J F, Stefanou P, Pons J A. Unveiling the optimization process of physics informed neural networks: How accurate and competitive can PINNs be? Journal of Computational Physics, 2025.\n\n[3] Bonfanti A, Santana R, Ellero M, et al. On the generalization of pinns outside the training domain and the hyperparameters influencing it. Neural Computing and Applications, 2024.\n\n[4] Zhao Z, Ding X, Prakash B A. Pinnsformer: A transformer-based framework for physics-informed neural networks. ICLR, 2024.\n\n[5] Wu H, Luo H, Ma Y, et al. Ropinn: Region optimized physics-informed neural networks. NeurIPS, 2024.\n\n[6] Wu H, Ma Y, Zhou H, et al. Propinn: Demystifying propagation failures in physics-informed neural networks[J]. arXiv preprint arXiv:2502.00803, 2025.\n\n[7] Xu C, Liu D, Hu Y, et al. Sub-sequential physics-informed learning with state space model. ICML, 2025.\n\n[8] Wang S, Sankaran S, Wang H, et al. An expert's guide to training physics-informed neural networks[J]. arXiv preprint arXiv:2308.08468, 2023.\n\n[9] Krishnapriyan A, Gholami A, Zhe S, et al. Characterizing possible failure modes in physics-informed neural networks. NeurIPS, 2021."}]

Author rebuttal:

Meta-review: This paper argues that common “failure modes” in Physics-Informed Neural Networks (PINNs) stem not from local minima but from insufficient precision. With FP32, L-BFGS prematurely convergences, causing spurious failure, while FP64 removes these stalls and enables reliable PDE solving. The authors identify a three-stage training dynamic (un-converged, failure, success) governed by arithmetic precision, reframing failures as precision-induced and highlighting the need for high-precision computation.

The reviewers consistently find the paper well-motivated and easy to follow. They highlight the novelty of the loss landscape analysis, the practical simplicity of the proposed solution, and the thorough evaluation showing that failures can be resolved without architectural changes. However, they also raise concerns. The approach is seen as somewhat rudimentary, with insufficient analysis of why FP64 works, no quantitative assessment of its computational cost, and limited exploration of alternative precision strategies. The experiments are viewed as narrow in scope—restricted to simple PDEs, L-BFGS optimization, and a single benchmark—leaving questions about generality, applicability to harder PDEs, and effectiveness with other optimizers like Adam. Issues with clarity of writing and figures are also noted.

Based on the strengths and weaknesses discussed, the AC believes the strengths outweigh the weaknesses and recommends acceptance, while strongly encourages the authors to incorporate the rebuttal into the final version of the paper.
Decision: Accept (poster)