INTEGRAL PINNS FOR HYPERBOLIC CONSERVATION LAWS

Authors: manvendra p rajvanshi
david i ketcheson
Venue: ICLR_cc_2024_Workshop_AI4DiffEqtnsInSci
Published: 2024-02-11 12:25:53

Abstract: Traditional physics-informed neural networks (PINNs) are trained based on differential equations and thus have difficulty capturing shock discontinuities in weak solutions of hyperbolic PDEs, since the differential equation doesn’t apply at the discontinuity. We propose Integral PINNs (IPINNs), which are trained based on the integral form of the conservation law, which holds at both continuous and discontinuous points of the solution. We use neural nets to model the integrals of the solution instead of the solution itself. We apply IPINNs to systems of hyperbolic conservation laws and show that they are much better at capturing the correct location and speed of shocks, compared to traditional PINNs. We also present a heuristic approach for detecting shock locations.

Main paper: 
 introduction: 
 Many natural phenomena are governed by conservation laws that take the form of hyperbolic partial differential equations (PDEs) (LeVeque, 2002) . A challenging and ubiquitous feature of such PDEs is the formation of discontinuities, or shock waves, even when starting from smooth initial data. Detection and handling of shocks is the key challenge in simulation of such systems. When a shock appears, the differential form description (which relies on continuity) breaks down and one instead looks for "weak solutions" based on the corresponding integral conservation law (Osher, 1984) . Any computational method needs dedicated tools for handling these sharp or discontinuous data, usually in the form of Riemann solvers and slope or flux limiters (Toro, 2013; LeVeque, 2002) .With the recent rise of machine learning technologies (deep learning (Goodfellow et al., 2016) in particular), there has been a lot of interest in using deep neural nets for scientific simulations, leading to the development of physics-informed neural networks (PINNs) (Raissi et al., 2019) . PINNs are more difficult to train than the vanilla supervised learning problems (Krishnapriyan et al., 2021) . PINNs loss is minimally composed of two parts: (a) A supervised loss at boundary(+initial) conditions and (b) Differential equation loss (PINN loss). Because the PINN loss relies on differentiation, continuously differentiable activation functions are preferred over the non-smooth ones that are popular in traditional supervised learning. This also suggests that PINNs might have difficulty solving PDEs with discontinuous solutions. Indeed it has been shown (Abreu & Florindo, 2021; Mojgani et al., 2023; Fuks & Tchelepi, 2020; Patel et al., 2022; Mao et al., 2020) that standard PINNs fail to capture shocks or fail to converge even for perhaps the simplest 1D hyperbolic conservation law, the inviscid Burgers equation:EQUATIONThe application of PINNs to this problem has been studied in many papers and various strategies have been proposed to improve their performance (Mojgani et al., 2023; Abreu & Florindo, 2021; Patel et al., 2022) . Similar issues have been observed in the application of PINNs to other hyperbolic systems, like convection-dominated flow (Krishnapriyan et al., 2021) , the Buckley-Leverett equation (Fuks & Tchelepi, 2020; Abreu & Florindo, 2021) , and compressible gas dynamics (Patel et al., 2022; Mao et al., 2020) . A number of case studies with proposed solutions are listed in (Mojgani et al., 2023 , Table 1 ). One of the popular remedies, which was also common in the early days of gridbased numerical solvers, is to add a very small (artificial) viscosity to the PDE (Fuks & Tchelepi, 2020; Abreu & Florindo, 2021; Patel et al., 2022) . There exist a wide range of proposed solutions which vary in level of formulation or implementation ranging from reformulation of the problem to be optimized to adaptive sampling and from architectural considerations to enforcing additional constraints in the loss function.The very definition of weak (discontinuous) solutions of hyperbolic PDEs is based on an integral conservation law, so it is natural to consider enforcing the integral form of the conservation law in order to capture discontinuities. Here we explore this idea by proposing a loss function based on the integral form, and we find that this significantly enhances the training of PINNs. In order to isolate and study the effect of this new approach to the loss function, we do not invoke any of the other strategies that have been proposed for dealing with discontinuities; many of those could be applied in addition and might lead to further improvements.We note that our work is similar in spirit to that of Patel et al. (2022) , as both works are based on the integral form of the conservation law. But in realization the approaches are very different, as Patel et al. (2022) use discretization of spacetime with a loss function which uses a numerical integration method. As will be clear from next section, we do not require these. Also, we use NN to model integral of the solution instead of solution itself and we do not need any domaindecomposition, making our approach conceptually different from other conservation-based PINNs (Patel et al., 2022; Jagtap et al., 2020; Hansen et al., 2023) . In next section we outline the idea and then present case studies in further sections. 
traditional pinns and shock-capturing ipinns: 
 We consider systems of hyperbolic conservation laws of formEQUATIONU (x, t) : R × R → R m is a set of conserved quantites (e.g. mass, momentum, energy) and F is the corresponding flux function. For example, in the inviscid Burgers equation (1) above, F (u) = 1 2 u 2 . The traditional PINNs approach is to apply a neural network ansatz for U i.e.EQUATIONwhere N θ is a neural network with weights θ. A Physics informed loss function is imposed on N θ :L PI = 1 n d n d j=1 ∂N θ (t j , x j ) ∂t + F (N θ (t j , x j )) x 2 (4)in addition to supervised loss on boundary/initial points (which we collectively denote by using subscript b for boundary).L B = 1 n b n b j=1 (N θ (t j , x j ) -U j ) 2 (5)where U j := U (t j , x j ) are values of function that we know from boundary/initial conditions. The total loss is:L = L PI + L B (6)Henceforth we will refer to the approach just described as traditional PINN. As discussed already, many improvements to traditional PINNs have been proposed already for hyperbolic PDEs, such as weighted loss or additional losses for improving conservation properties. But here we focus on evaluating a specific new formulation of the loss function.Weak (i.e. discontinuous) solutions of the differential conservation law equation 2 must satisfy the integral conservation law (obtained by integrating equation 2 over an arbitrary spatial interval)EQUATIONThis relation holds for arbitrary intervals [x a , x b ]. We leverage this fact by letting neural net N θ represent the integral of U rather than U itself:N θ (t, x) ≈ x l U (t, x ′ )(dx ′ ) (8)where l is some reference point and can be conveniently taken as the left or right boundary of the spatial domain. Then the IPINN loss becomes:L IP = 1 n d n d j=1 ∂N θ (t j , x jb ) -N θ (t j , x ja ) ∂t -(F ((N θ ) x )| xa -F ((N θ ) x )| x b ) 2 (9)where (N θ ) x = ∂N θ ∂x and each training sample j has three components x b , x a , t i.e. each training point is composed of two different points from space with a time coordinate. So instead of a pointwise equation this is effectively a conservation equation over randomly-chosen intervals. The values of U that are required for calculating fluxes at end points are calculated by taking the x-derivative of neural network. This is combined with the standard boundary loss function equation 5 to give a total loss function L = L IP + L B .A key challenge for hyperbolic PDEs with shocks is to detect the location of shocks. Even in the case where one is not looking for the neural-net based solution over the entire domain, it would be useful to have a (neural-net) model that informs us about the location of shocks in spacetime. This could further facilitate dynamic domain decomposition for either classical numerical solvers (similar to Beck et al. (2020) ) or further training of domain-wise/domain-decomposition based neural networks (Shukla et al., 2021) .Here we look at the prospect of detecting shock locations based on the IPINN that we have trained. For simplicity, we will work with the example of 1-d Burgers of subsection 3.1. We notice that there are two indicators that can be used to detect the location of shocks. Both exploit the fact that there are equalities/inequalities that are satisfied in region of smoothly varying solutions but break down at shocks. The first indicator is based on the entropy inequality. Hyperbolic conservation law systems typically possess a so-called entropy-flux pair (LeVeque, 2002) (η, q) such that µ(x, t) := η t + q x (10) vanishes except in the vicinity of a shock, where this quantity can be negative. For Burgers equation, one such entropy-flux pair is (Patel et al., 2022) :EQUATION)Given a trained IPINN, we can use the set of points where |µ| exceeds some threshold as an indication of where shocks may appear.Another indicator of shock location is based simply on violation of the differential equation ( 2), and we discuss this in appendix A.1. 
test cases: 
 In this section, we present experiments and comparison between IPINNs and traditional PINNs. We use solutions from the finite volume (FV) solver Clawpack (Clawpack Development Team, 2020) as reference solutions. Besides the examples in this section, we give an additional example in appendix A.2.We start with the inviscid Burgers equation ( 1), with smooth initial data:u(0, x) = 2 sin(3x) + cos(2x) -1.0, if -π < x < π 0, otherwisewhere x ∈ [-10, 10], t ∈ [0, 6] with constant (homogeneous Dirichlet) boundary conditions at x = ±10. Multiple shocks form after a short time. This example is even more challenging than, say, a Riemann problem, because of the transition from a smooth solution to a discontinuous one.Results are shown in Figure 1 , where it can be seen that a traditional PINN completely fails to capture the shocks, while the IPINN captures both the amplitude and location of each moving shock fairly accurately, albeit still with some oscillations near two of the shocks. We also plot in pink the locations where |µ| > 1/2 to show where this would indicate the presence of shocks, and we see that it coincides almost perfectly with the actual shock locations. The shallow water equationsEQUATIONare used to model water waves whose wavelength is long relative to the overall water depth, such as tsunamis in the deep ocean (Lannes, 2013) . Here h is the water depth and u is the horizontal water velocity, while g represents the force of gravity. We solve the Riemann problem with initial dataEQUATIONThe solution consists of two outward-going shocks, and computed solutions are shown in Figure 2 .An entropy-flux pair for ( 12) is (Ketcheson & Quezada de Luna, 2022) :EQUATIONWe show the final shock locations, as indicated by entropy residual, in the lower-right plot of Figure 2 . In this figure we plot |µ| (where now we take the appropriate entropy function and corresponding flux for the SWEs) for the two types of PINNs that we consider along with the profile of h from Clawpack to show the discontinuity location as calculated by FV methods in Clawpack. We see again that IPINNs accurately capture the moving shock locations.  
conclusion, challenges and further development: 
 We have shown that putting the neural ansatz on the integral of the solution and using it in conservation form for PINN training enhances the performance of PINNs in learning solutions with moving shocks. In doing this we have not used any extra regularizations or modifications that have been proposed in literature. Those could be applied in combination with the IPINN approach to obtain possible further improvements. We note that IPINNs require extra autograd operations and can be slightly slower than traditional PINN implementations, but this is often also the case with other proposed modifications of PINNs.We are currently working on generalizing this approach to 2D and 3D problems, as well as performing a cost-benefit analysis in terms of extra computational costs when scaling to systems of equations in 2D or 3D. The preliminary results in this work suggest that the proposed formalism might be more useful in scenarios where one expects to get strong moving shocks. The differential form 2 of conservation laws break down where shocks are formed, so we can use "extreme peaks" of this differential form violation as an indicator of shock location. We define differential form violation (DF V ) as:EQUATIONNote that this is just the traditional Physics-informed loss. For a converged model, this is supposed hold everywhere but at shocks. We plot this quantity in figure 3 for the inviscid Burgers case at time corresponding to figure 1 . Here we consider the challenging case of non-convex flux function:f (u) = u 2 u 2 + (1 -u) 2 (17)with piecewise constant initial condition of u(0, x) = 1 for x ≤ 0 and u(0, x) = 0 otherwise. One particular feature of this case that makes it complicated is the fact that the solution consists of a piecewise constant region, then a continuous rarefaction and a moving discontinuity. In the cases of Riemann problems where the evolution transforms piecewise constant initial data into piecewise constant data at any later time, one might use piecewise defined activation functions like relu and get very good results, but this would certainly fail for cases like Buckley-Leverett. Our experiments with Buckley-Leverett show that it is indeed tough to handle this complexity with simple PINNs, but the proposed IPINNs give much better results than traditional PINNs as shown in figure 4 .  
 
 Appendix:  


Decision: The reviewers highlight the importance of the proposed method empirically for solving hyperbolic conservation laws with shocks, which can be a failure mode of the original PINN method. I vote for acceptance but agree with the other works, e.g., cPINNs, cvPINNs and ProbConserv [Hansen et. al, "Learning Physical Models that Can Respect Conservation Laws", ICML 2023] that also proposes to use the integral form of the conservation law, should be compared to or at least referenced. Note that there is a minor typo in Eqn. 7, where $x_b$ is missing.

Reviews: review: title: title: Good Paper, Accept
main_review: main_review: I congratulate the authors on getting this research to this stage. 

The research is good quality and clearly describe the problem and propose the solution. The work presented is known to be original (in my knowledge) and marks an significant improvement to use IPINNs for hyperbolic conservation laws. 

The text is written in clear manner and easily understandable. There are some minor inconsistencies in the text , eg. neural net and nets are used interchangeably, Line 4 use the word "such" and rephrasing is needed here. This is not an exhaustive list and therefore I request authors to do a quick proof reading.
title: title: Effective use of integral form in PINNS, clear accept
main_review: main_review: # Summary
This paper introduces integral physics-informed neural networks (IPINNs), which differ from traditional PINNs by training a neural network to model the integral of the solution rather than the solution itself. This is in contrast to other PINN papers that attempt to incorporate integral conservation laws by numerically integrating the sub-regions [1] or modelling flux on interfaces between sub-regions [2].

This approach is targeted to hyperbolic PDEs, where traditional PINNs are known to break down because the solution is discontinuous near "shock" points. This paper shows that the IPINNs capture the solution profile better than traditional PINNS on two challenging problems (Inviscid Burgers' and Shallow Water Equations).

# Review
Overall, this work warrants acceptance to the workshop. The modification to PINNs to incorporate the integral form is elegant in its simplicity. More importantly, the authors demonstrate that their method is effective on challenging hyperbolic PDEs where PINNs are not.

## Originality
This is not the first work to integrate the integral form of conservation laws into PINNs. However, this work differs from previous approaches by having the neural network learn the integral of the solution over the state domain, rather than the solution itself. The original solution is then recovered by auto-differentiation of the neural network. To my knowledge, this seems original, but there may have been something I have missed in the deluge of PINNs-related literature.

## Significance
Unlike many PINNs-related papers, this work is targeted at difficult, hyperbolic PDEs that pose challenges for both PINNs and traditional numerical methods. By taking advantage of knowledge of the PDE (hyperbolic with conservation law), the performance of PINNs can be greatly improved. This illustrates that traditional PINNs are not a "plug and play" approach that can solve any PDE with the same framework.

## Quality
The method is relatively straightforward as written. The authors change the loss to be on the integral of $U$ rather than $U$ itself. However, this makes a large difference in the performance of the network in empirical settings (Figures 1 and 2). The authors pick challenging examples that are a failure mode of traditional PINNs, and aim to solve an important downstream task (shock detection).

While not necessarily required for a workshop paper, I would expect in a full paper to see an empirical comparison to the referenced competing "conservative" PINNS approaches such as cvPINNs [1] and cPINNs [2]. My hunch is that these other methods will break down in the hyperbolic setting, but it would be good to validate that hunch empirically.

## Clarity
The method and paper are clear and easy to follow. However, Figure 1 (left panel) and Figure 2 are quite difficult to read. It is difficult to visually distinguish the solutions. I suggest breaking out each competing method into a separate facet that is compared with just the reference (Clawpack).

Also, I have a minor stylistic comment. It is confusing to have the neural network be referred to as $N$ and $N_\theta$. The use of capital "N" is typically reserved for a number (e.g. number of samples). This is especially confusing when in Equation 8 the lowercase $n_d$ refers to the dimension size.

[1] Patel, Ravi G., Indu Manickam, Nathaniel A. Trask, Mitchell A. Wood, Myoungkyu Lee, Ignacio Tomas, and Eric C. Cyr. “Thermodynamically Consistent Physics-Informed Neural Networks for Hyperbolic Systems.” Journal of Computational Physics 449 (January 15, 2022): 110754. https://doi.org/10.1016/j.jcp.2021.110754.

[2] Jagtap, Ameya D., Ehsan Kharazmi, and George Em Karniadakis. “Conservative Physics-Informed Neural Networks on Discrete Domains for Conservation Laws: Applications to Forward and Inverse Problems.” Computer Methods in Applied Mechanics and Engineering 365 (June 15, 2020): 113028. https://doi.org/10.1016/j.cma.2020.113028.
title: title: INTEGRAL PINNS FOR HYPERBOLIC CONSERVATION LAWS
main_review: main_review: The paper introduces Integral Physics-Informed Neural Networks (IPINNs) as an innovative approach to overcoming the limitations of traditional Physics-Informed Neural Networks (PINNs) in handling shock discontinuities in weak solutions of hyperbolic partial differential equations (PDEs). The novelty of IPINNs lies in their training methodology, which is based on the integral form of conservation laws. This method is advantageous because it remains valid across both continuous and discontinuous solution points, thereby facilitating more accurate modeling of shocks in hyperbolic conservation law systems. 
The critique centers on the perceived lack of novelty in the proposed Integral Physics-Informed Neural Networks (IPINNs), noting their similarity to previous methods developed by authors Patel and Jagtap. To constructively address this concern, it is crucial to highlight any distinct features, improvements, or specific applications of IPINNs that differentiate them from the earlier approaches. This includes detailing any methodological enhancements, computational efficiencies, or broader applicability to a range of hyperbolic PDEs. Direct comparison in terms of performance metrics, accuracy in shock capturing, and computational resources required could further elucidate the advancements IPINNs offer over the previous methods. Without clear differentiation or demonstrated superiority, the contribution of the paper to the field may indeed appear limited.