Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work studies two specific (yet essential) PDE learning cases in the PINN framework. Notably, the work proposes two main conclusions to explain the difficult training in PINN networks: 
- the loss landscape varies greatly with the PDE coefficient values (and works well with only low diffusion/conduction coefficient), 
- the loss landscape (at the end of training) varies greatly with the physics informed penalty coefficient. 

The main explanation to such difficulties lies in the condition number associated to PINN penalty. The work proposes two solutions: a warm-up, and a sequence to sequence approach.
main_review: First, I highlight that the paper addresses a significant problem of sciML community.
The presented work is well-written and straightforward. The presented arguments are sound and can be leveraged by the community to address the evoked problems.

Concerning the proposed solutions. The investigation of Section 5.1 is interesting but only superficial, and therefore cannot lead to a solid conclusion. Also, the presentation of the solution 5.2 is in my opinion unclear.

Two main flows appears in the paper:
1) The causes of the flows in learning PINNs are well identified however it is difficult to state whether the evoked flows were taken into consideration when designing the solutions. 
2) The part on the sequence-to-sequence learning is very poorly described and not clear, despite being a possible solution to the described problems even though experimental results are not compelling.


My questions are: 
1) About the warm states: how come such warm states perform better ? What is the distribution of the weights at the end of training vs in the beginning or vs a regular initialization ?
Can the gradients of the parameters with respect to the loss be investigated in such a case ?
2) A simple approach to the problem would be to gradually increase the PINN penalty, progressively enforcing the constraint. Can such a simple approach solve the condition problem ?
3) Despite the vanishing/exploding gradients in RNN, the NLP community successfully leveraged simple optimization tricks such as gradient clipping ? Can the authors comment on the possible applicability of such optimization strategy ?
4) Can’t interior domain points for supervision help solving the ill-condition of the learning ?


____
### post rebuttal
____

The novel experimental results shows the advantage of the proposition to solve the highlighted issue. 
I have decided to raise my score to 6.

limitations_and_societal_impact: see above.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper the authors inspect the popular Physics Informed Neural Network (PINN) set of models and characterize their behavior (specifically their failure modes) in the context of two popular PDE systems, namely diffusion and convection. In each case, the authors demonstrate that PINNs learn effective representations in simple regimes while failing to learn representations of the PDE systems in more complex regimes. In addition, authors also demonstrate that it is not the inability of the neural network to fit the functions (due to some potential inability to represent non-trivial functions), but rather the way the PINNs incorporate the PDEs into the learning pipeline that is the reason for the failure. Finally, the authors also propose casting the PINN problem as a sequential prediction task (i.e., predicting all collocation points per time period) as opposed to tasking the neural network with simultaneously learning the full spatial and temporal domain. 

main_review: Overall the paper is clearly written and appropriately organized. But for a few minor typos, there are not many problems with the text or narrative structure.

**Positives:**
1. In this paper, authors tackle an important question of characterizing the performance of the popular PINN model [1], in the context of two canonical PDE systems, namely the convection and diffusion PDEs and do indeed demonstrate failure modes of PINNs.


2. Specifically, authors work in the context of 1D PDEs and showcase that for increasing values of equation coefficients (i. \beta i.e., convection coefficient in the case of the convection equation; ii. \nu i.e., viscosity in the case of the diffusion equation), PINN behavior degrades. Figures 1,3 in the paper demonstrate how the PINN prediction errors scale with increase in value of the equation coefficients (i.e., convection coefficient and viscosity respectively).



3. Further, the authors in their experimental setup section clearly outline a rigorous evaluation procedure wherein results have been calculated based on multiple runs of a PINN model optimized with L-BFGS and executed using different random seeds. 

limitations_and_societal_impact: 1. The proposed results although clearly demonstrative of certain failure modes in PINNs, offer an incomplete characterization and the solutions proposed (specifically the sequence-to-sequence approach) to alleviate the failure models is not fully detailed and demonstrated for sophisticated higher order PDEs (e.g., second order coupled PDEs like Navier-Stokes equation).

2. The claim that the reason for PINN based failure is due to the “subtle problems”  introduced through the PDE based soft constraint of L_F is not fully characterized or explained. What are the set of “subtle problems” that are being discussed? One that is definitely demonstrated is that the loss-landscape gets more complicated. However, other problems could have to do with the interplay between the loss terms which aren’t discussed.

    Speaking specifically to the point of the “interplay” between loss terms at different values of equation coefficients, it is imperative to demonstrate the influence of each of these loss terms on PINN learning dynamics to fully contextualize the failure. For example: at different values of $\beta$ (or $\nu$), how do the gradient distributions change w.r.t the gradient distributions of the other terms in the loss? 

4. The model re-formulation in 3.1.2 and 3.2.2 is unclear. For example, in the case of 3.1.2, are the $L_B$ , $L_U$ terms replaced by those in equation 9 and the value of the forcing term `q` replaced with the new value on line `178`? If so, wouldn’t $\beta$ although absent in the analytical solution due to detailed simplifications), still have an influence on the PINN results?
Specifically (assuming the above point of $\beta$’s influence on the PINN gradient holds true), it should be inspected how the gradient dynamics with increasing $\beta$ (and correspondingly for increasing $\nu$ in section 3.2.2) influence the learning especially at the initial training epochs of the network. Have these dynamics been inspected and if they have, what are the learnings?

5. Has the addition of standard regularization terms (`L1, L2`, in conjunction with current PINN losses) been inspected and if so does that have a positive effect on the learned representation? 


6. There has been some previous work inspecting the effect of various physics based regularizers in PINN [2] and certain adaptive learning rate schemes have been proposed and shown to be effective. It is important to contextualize the current set of results in the context of this previously published work to fully characterize the failure modes of PINNs. 

**References:**

[1]  Raissi M, Perdikaris P, Karniadakis GE. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics. 2019 Feb 1;378:686-707.

[2] Wang S, Teng Y, Perdikaris P. Understanding and mitigating gradient pathologies in physics-informed neural networks. arXiv preprint arXiv:2001.04536. 2020 Jan 13.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors provide a detailed empirical analysis of the numerical difficulties encountered when training so-called "physics informed neural networks" (PINNs), a recent class of models which incorporate physical (often differential) equations into the loss function of a neural network. The authors perform a systematic analysis on two relatively simple but common use-cases in scientific literature, convection and diffusion equations, and demonstrate the PINNs fail to adequately learn the underlying dynamics of the system even in simple cases. They show that these failures are due to the effect of the convection/diffusion coefficients and forcing terms on the conditioning of the dynamical system, i.e. larger coefficients or forcings tend to make the problem increasingly ill-conditioned, thereby impacting characteristics of the loss function. The authors then propose two possible workarounds for these issues; improved initialization via a warm start or preconditioning and re-framing the problem as a sequence-to-sequence learning task.
main_review: Overall, the paper is very well written, the experiments are thorough and well designed, and the results are both interesting and of great practical value for researchers and practitioners in scientific machine learning. The theoretical analysis of the Lagrange optimization problem, while relatively informal, provides valuable insight to readers who are less familiar with the field of non-convex optimization and should encourage researchers to think carefully about how arbitrary regularization terms affect the underlying optimization problem in PINNs and other similar methods.

Ultimately, the results presented by the authors might not be terribly surprising to those with a background in scientific/numerical computing, where the problem of ill-conditioned differential operators in numerical integration and optimization has been grappled with for decades. However, it's important to confirm intuition and expectation with rigorous experimentation, which is exactly what the authors have done here.

In summary, this work provides valuable experimental confirmation and analysis of pathologies in PINNs and will, I think, be a valuable contribution to the field. The paper appears pretty much ready to publish as-is, but I will make a few suggestions below.

1. I missed a discussion on discretization strategies, both for the temporal and spatial derivatives. It is briefly mentioned in the supplementary materials that a uniform spatial grid spacing is used for convection (?), but that's all I can find. This is somewhat interesting since uniform grid spacing is typically the easiest case, so if the PINN can't even do that correctly, then it would be reasonable to expect it to fail on non-uniform grids as well. However, I would think that the resolution of the discretization would affect the numerical stability of the loss term, as finer meshes would result in more accurate gradients (assuming finite differencing is used) but also potentially more floating point error. Ideally, it would be nice to see an experiment demonstrating the sensitivity of the results presented here to discretization choices, but a brief discussion would also suffice, I think.


2. It would be nice to see a few more details on training, e.g. the number of training samples (collocation points?), discretization details (as mentioned previously), the convergence criterion, etc.


3. The results in figure D.1 are a bit puzzling to me. The authors claim "As the regularization parameter is increased, the loss landscape becomes increasingly more complex and harder to optimize (see the z-axis scale)" but this supposed increase in complexity is not apparent (at least to me) in the plots. The value of lambda that produced the lowest error values ($\lambda = 0.01$) has a very similar loss landscape plot to the worst value ($\lambda = 10.0$). In the latter case, the loss values are of course much higher, but that's to be expected as the value of $\lambda$ itself partially determines the scale of the loss term. It would be more informative to see a breakdown of how each loss term (initial condition, boundary conditions, and the unscaled differential equation) changes in response to the value of $\lambda$. I don't think the raw loss value is very informative.

4. Minor stylistic suggestion: The qualifier "Interestingly," is used, in my opnion, a bit too frequently throughout the text.

5. I was a bit confused to find empty sections in the appendix and had to go to the supplementary material to find the figures/tables. I'm not sure if this is a standard formatting choice of NeurIPS or not, but I would suggest at least adding notes directing the reader to the supplementary materials, or removing the section headers from the main document altogether.
limitations_and_societal_impact: Limitations and direction for future work are clearly discussed. I don't see any major societal impact issues, outside of perhaps concerns about potential misuse of PINNs in scientific research. However, the authors' work should, if anything, help ameliorate this.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper studies the behavior of Physics Informed NNs (PINNs) on two common PDEs of physical relevance. It observes that PINNs fail to learn a good solution under standard training regime when the convection coefficient or viscosity coefficient is high, even when the closed-form solution does not depend on the constants.

The authors then seek to explain this behavior from an optimization standpoint. They plot the loss landscape along the two dominant hessian eigenvectors. They hypothesize that the optimization problem is ill conditioned due to the regularization term in the optimization objective that enforces the pde constraint.

They also run experiments to argue that sequence to sequence learning might work better for these ill conditioned problems.
main_review: I think the paper tackles an important problem in PINNs, but it could go deeper in its investigations.

- When diagnosing whether the loss landscape is ill conditioned, would it be possible to get more information about the spectrum of the hessian? It would be valuable to know bounds on the condition number, or some other investigations on the eigenvalue spectrum.
- The authors give a heuristic scaling law for the condition number. Could you give a bit more detail on the derivation? Would it be possible to empirically verify this? You could compute the condition number for the operator discretized on a finer and finer grid.
- The authors mention the condition number of the PDE operator, but isn't what we care about the condition number wrt the NN parameters? During optimization, we change the NN parameters and hope that doesn't lead to large changes in output. It would be interesting to know how the conditioning changes depending on NN architecture.
- In the warm start experiments, from my understanding you only run these on problems where the solution does not depend on the parameter. In that case, wouldn't the weights not change when you move to a larger coefficient problem, since you warm start from the solution? It would be interesting to see warm start experiments on problems where the solution does depend on the coefficients.
- The sequence to sequence solution at the end is mentioned, but the results seem mixed. It seems to work much better for diffusion vs convection. Do you have any insight on why this could be the case?

I am borderline in this paper, currently leaning towards a rejection, but open to changing my score if most of my comments above are addressed.
limitations_and_societal_impact: Limitations mentioned above.
review_title: null

Meta-review: This paper addresses an interesting and important topic, and performs a careful and well motivated analysis, producing very practical lessons. There was active discussion of this paper -- 3 out of 4 reviewers raised their scores as a result, and all 4 reviewers vote for acceptance. I am therefore recommending that this paper be accepted.
Decision: Accept (Poster)