Title: Lie Point Symmetry and Physics-Informed Networks
Venue: NeurIPS 2023
Accepted: yes
Keywords: PDE

Lie point symmetry

Symmetry

Neural PDE solver

PINNs
Abstract: Symmetries have been leveraged to improve the generalization of neural networks through different mechanisms from data augmentation to equivariant architectures. However, despite their potential, their integration into neural solvers for partial differential equations (PDEs) remains largely unexplored. We explore the integration of PDE symmetries, known as Lie point symmetries, in a major family of neural solvers known as physics-informed neural networks (PINNs). We propose a loss function that informs the network about Lie point symmetries in the same way that PINN models try to enforce the underlying PDE through a loss function. Intuitively, our symmetry loss ensures that the infinitesimal generators of the Lie group conserve the PDE solutions.. Effectively, this means that once the network learns a solution, it also learns the neighbouring solutions generated by Lie point symmetries.
Empirical evaluations indicate that the inductive bias introduced by the Lie point symmetries of the PDEs greatly boosts the sample efficiency of PINNs.

Lie Point Symmetry and Physics Informed Networks

Abstract

Symmetries have been leveraged to improve the generalization of neural networks through different mechanisms from data augmentation to equivariant architectures. However, despite their potential, their integration into neural solvers for partial differential equations (PDEs) remains largely unexplored. We explore the integration of PDE symmetries, known as Lie point symmetries, in a major family of neural solvers known as physics-informed neural networks (PINNs). We propose a loss function that informs the network about Lie point symmetries in the same way that PINN models try to enforce the underlying PDE through a loss function. Intuitively, our symmetry loss ensures that the infinitesimal generators of the Lie group conserve the PDE solutions. Effectively, this means that once the network learns a solution, it also learns the neighbouring solutions generated by Lie point symmetries. Empirical evaluations indicate that the inductive bias introduced by the Lie point symmetries of the PDEs greatly boosts the sample efficiency of PINNs.

1 Introduction

In recent years, deep learning has accelerated data-driven approaches to science and engineering. A prominent example is the role of deep learning in solving partial differential equations (PDEs), which are ubiquitous in many scientific disciplines.

This is mainly driven by the fact that traditional handcrafted numerical solvers can be prohibitively expensive, and thus, learning to solve PDEs has the potential to significantly impact various areas of science, ranging from quantum chemistry to biology to climate science to cosmology, [Wang et al., 2020a, Kochkov et al., 2021, Kashinath et al., 2021, Gupta and Brandstetter, 2022, Nguyen et al., 2023, Bi et al., 2022]. Like any other machine learning problem, learning to solve PDEs can benefit from inductive biases to boost sample efficiency and generalization capabilities. As such, the PDE itself and its respective Lie point symmetries, i.e. joint symmetries of coordinate and field transformations, are two natural inductive biases for constructing efficient neural PDE solvers.

For parametric PDEs, Physics-Informed Neural Networks (PINNs) have emerged as an efficient new learning paradigm. PINNs constrain the solution network to satisfy the underlying PDE via the PINN loss terms [Raissi et al., 2019], which comprise

80

0.6

0.4

0.2

0.0

0.2

0.4

0.6

u(x)

0.8

u(x)

0

10

15

20

SZ

x

a residual loss, as well as loss terms for initial and boundary points. As such, PINNs can be seen as a data-free learning approach, utilizing available physics information to guide neural network training. Lie point symmetries are uniquely special in that it is possible to characterize the full set of permissible transformations for each PDE, and thus naturally offer additional physics information which piggy-backs the data-free learning objective. Lie point symmetries have been introduced to deep learning by Brandstetter et al. [2022a] for Lie point symmetry data augmentation which places a firm mathematical footing on data augmentation pipelines for neural PDE solvers. Further, Lie point symmetries showed promising results when used in extending the framework to self-supervised learning for neural PDE solvers [Mialon et al., 2023]. In this work, for the first time we integrate symmetries into PINNs and demonstrate the effectiveness of this symmetry regularization on generalization capabilities.

Lie point symmetries of a PDE, by definition, map a solution to a solution, preserving the PDE. Lie point symmetry transformations act on both independent ( e.g. , time and space), and dependent (PDE solution) fields and, by extension, on all partial derivatives. To enforce a Lie point symmetry, we first need to prolong a symmetry transformation to find how it transforms the partial derivatives of the respective PDE. We show how to do this using automatic differentiation, which enables us to transform the entire PDE for each infinitesimal generator of each Lie point symmetry. 'Imposing a symmetry' means that by transforming the PDE via a symmetry transformation, the PDE is required to be preserved. In practical terms, Lie point symmetries should be orthogonal to the PDE gradient. Each Lie Point symmetry results in one such orthogonality constraint, which we enforce using a penalty term within the PINN framework. We can enforce these on the same points where the PINN loss is imposed. Importantly, our Lie Point symmetry regularization is complementary to the PINN loss: while the PDE loss regularizes the neural network to satisfy the PDE at select points, the Lie point symmetry loss regularizes the neural network such that infinitesimal changes in different directions continue to satisfy the underlying PDE.

2 Background

2.1 Partial Differential Equations

Partial differential equations (PDEs) are used to mathematically describe the dynamics of various physical systems. In a PDE, the evolution of a function that involves several variables is described in terms of local updates expressed by partial derivatives. Since temporal PDEs are especially prevalent

in physical sciences, we consider PDEs of the following general form:

where u ( t, x ) ∈ R D u is the solution to the PDE, t denotes the time, and x is a vector of possibly multiple spatial coordinates. Ω ⊂ R D is the domain and D x [ . ] is a non-linear differential operator . f ( x ) is known as the initial condition function and g ( t, x ) describes the boundary conditions. In the following, we sometimes explicitly separate the time from other independent variables and sometimes for convenience group them together.

Example 1 (Heat Equation) . The one-dimensional heat equation describes the heat conduction in a one-dimensional rod, with viscosity coefficient ν . Its differential operator is given by D x [ u ] = -νu xx .

Using Neural Networks for Solving PDEs. For many systems, obtaining an analytical solution to the PDE is impossible; hence, numerical methods are traditionally used to obtain approximate solutions. Numerical solvers, such as finite element methods (FEM) or finite difference methods (FDM) rely on discretizing the space T × Ω [Quarteroni, 2009]. The topology of the space has to be taken into account when constructing the mesh, and the resolution of the discretization will affect the accuracy of the predicted solutions. Additionally, these solvers are often computationally expensive, especially for complex dynamics, and each time the initial or boundary conditions of the PDE change the solver must be rerun. These considerations and constraints make designing numerical solvers difficult, and scientists often need to handcraft a specific solver for an application [Quarteroni, 2009].

Given the recent successes of neural networks, especially in dealing with large datasets, using deep learning to solve PDEs has become a promising direction. The idea is to learn a function or, more generally, a functional that bypasses the numerical computation and produces the solution to the PDE in a single shot or by progression through time. Broadly, there are two approaches to solving PDEs with neural networks: neural operator methods which approximate the solution operator that maps between solutions of the underlying PDEs, and direct methods which learn the underlying solution function. The most prominent representative of the latter are PINNs.

PINNs. In contrast to neural operator methods [Lu et al., 2021, Li et al., 2021a] where the main idea is to generalize neural networks to obtain mappings between function space, PINNs directly learn the solution function of the underlying PDE. Consequently instead of relying on large training sets,

PINNs operate as a surrogate model for the PDE solution, trained directly with the PDE equation itself. The simplicity of the PINN idea has made it the subject of many follow-up improvements; see Krishnapriyan et al. [2021], Wang et al. [2020b, 2022, 2023]

In PINNs, the PDE solution u ( t, x ) of Eq. (1) is a neural network u θ ( t, x ) with parameters θ . The loss function is then compromised of two parts:

The first term is the physics-informed objective, ensuring that the function learned by the neural network satisfies the PDE Eq. (1). Let r ( t, x ) denote the residual error in agreement with the PDE equation:

where the derivatives of the solution network u θ are calculated using automatic differentiation [Raissi et al., 2019]. This penalty is then imposed on a finite set of points ( t, x ) 1: N r which are sampled from inside the domain [0 , T ] × Ω to obtain the PDE loss :

The second term in Eq. (2), is a supervised loss which ensures that the function learned by the neural network satisfies the initial and boundary conditions of the problem - that is

where ( x 0 ) 1: N s ∈ Ω are N s samples at which the initial condition function f is sampled. ( t b , x b ) 1: N b are N b points sampled on the boundary (from [0 , T ] × ∂ Ω ).

There have also been multiple models that combine the operator learning approach with the physicsinformed loss. For example, in Li et al. [2021b], the NO approach is combined with the PINN loss. Another example is Wang et al. [2021], which we describe further in Section 3.1.

2.2 Symmetries

Groups. Symmetries are essentially transformations of the object that leave an aspect of it invariant: for example, the nature of an object does not change if we translate or rotate it. In mathematics, symmetries of an object are described by abstract objects known as groups . More concretely, a group is a set G and a group operation · : G × G → G satisfying: associativity, the existence of an identity element, and the presence of an inverse element for every element in the set.

Lie Groups. Groups that are also differentiable manifolds are known as Lie groups , and are important in the study of continuous symmetries. In Lie groups, in addition to satisfying the three properties, the group operation · and its inverse are smooth maps. Each Lie group has an associated Lie algebra which is its tangent space, as a vector space, at identity. 3 Intuitively, Lie algebras describe the smooth transformations of the Lie groups in the limit of infinitesimal transformations. The elements of the Lie algebra are vectors describing the direction of the infinitesimal symmetry transformation.

One Parameter Subgroups. The Lie group, G , can be multi-dimensional and complex. When the n -dimensional group G is simply connected , it is often represented in terms of a series of n one-parameter transformations , g = g 1 ( ϵ 1 ) g 2 ( ϵ 2 ) . . . g n ( ϵ n ) , where g i : R →G , i = 1 , . . . , n and such that g i ( ϵ ) g i ( δ ) = g i ( ϵ + δ ) . The ϵ 's are the real parameters of the transformation, and each g i is a continuous group homomorphism ( i.e. , a smooth, group-structured map) from this parameter to the symmetry group.

2.3 Symmetries of Partial Differential Equations

In this section, we will provide the mathematical background of symmetries of differential equations. We will mostly follow the exposition presented in Olver [1986] and refer the reader to this original text for a more in-depth treatment of this topic.

In the context of differential equations, symmetries are transformations that map a solution of the PDE to another solution. For example, in Fig. 1, we can see how the solutions of a simple harmonic oscillator can be obtained via symmetry transformations of a given solution, where the symmetry group is SL (3) . Consider the PDE ∆ , involving p independent variables x = ( x 1 , . . . , x p ) ∈ X and q dependent variables u = ( u 1 , . . . , u q ) ∈ U . The solutions to the PDE will be of the form u i = f i ( x 1 , . . . , x p ) for i = 1 , . . . , q , for x ∈ Ω , where Ω ⊂ X is the domain of the function f. The symmetry group G , of ∆ , is the local group of transformations on an open subset of the space of dependent and independent variables, M ⊂ X × U , transforming solutions of ∆ to other solutions.

Prolongations. To formalize this abstract definition of PDE symmetries, Lie proposed viewing ∆ as a concrete geometric object and introduced the concept of prolongation [Olver, 1986]. The idea is to prolong the space of independent and dependent variables, X × U , to a space that also represents the partial derivatives involved in the PDE. More concretely, we have the following definition:

Definition 1. The n-th order prolongation (or n-th order jet space ) of X × U is denoted as X × U ( n ) = X × U 1 × · · · × U n , whose coordinates represent the independent and dependent variables as well as all the partial derivatives of the dependent variables up to order n.

Equivalently we have the notion of prolongation of u as u ( n ) = ( u x , u xx , . . . , u n x ) where u i x is all the unique i th derivatives of u , for i = 1 , . . . , n . For example, if x = ( x, y ) , then u xx = ( ∂ xx u , ∂ xy u , ∂ yy u ) .

Using this notion of prolongation, we can represent a PDE as an algebraic equation, ∆( x , u ( n ) ) = 0 , where ∆ is the map that determines the PDE, i.e. , ∆ : X × U ( n ) → R . In other words, the PDE tells us where the map ∆ vanishes on X × U ( n ) .

For example, for the heat equation described in 1, ∆ is given by:

The graph of all prolonged solutions is the set S ∆ ⊂ X × U ( n ) , and is defined as S ∆ = { ( x , u ( n ) ) : ∆( x , u ( n ) ) = 0 } . In this new notation, we can say that u ( x ) is a solution of the PDE if Γ ( n ) u = { ( x , pr ( n ) u ( x )) } ⊂ S ∆ . where pr ( n ) u ( x ) : X → U n , is a vector-valued function whose entries represent all derivatives of u wrt x up to order n .

Prolongations of the Infinitesimal Generators. Let v be the vector field on the subspace M ⊂ X × U with corresponding one-parameter subgroup exp( ϵ v ) . In other words, the vector field v is the infinitesimal generator of the one-parameter subgroup. Intuitively, this vector field describes the infinitesimal transformations of the group to the independent and dependent variables, and we can write it as:

where ξ i ( x , u ) and ϕ α ( x , u ) are coordinate-dependent coefficients. To study how symmetries transform a solution to another solution, we need to know how they transform the partial derivatives and, therefore, the jet space X × U ( n ) .

A symmetry transformation of the independent ( x ) and dependent ( u ) variables will also induce transformations in the partial derivatives u x , u xx , . . . . The prolongation of the infinitesimal generator , pr ( n ) v is a generalization of the generator v which describes these induced transformations in these partial derivatives.

This prolongation will be defined on the jet-space X × U ( n ) and it is given by:

where we have used the notation J = ( i 1 , . . . , i k ) for the multi-indices, with 0 ≤ i k ≤ p and 0 ≤ k ≤ n and u α J = ∂ k u α ∂x i 1 ...∂x i k . Calculating ϕ ( J ) α , the coefficients of ∂ u α J , can be done using the prolongation formula, which involves the total derivative operator D (see Olver [1986] for a derivation):

The upshot is that we can mechanically calculate the prolonged vector field using partial derivatives of u , which are, in turn, produced by automatic differentiation. In practice, prolonged vector fields are implemented as vector-valued functions (or functionals) of x and u . Therefore, the implementation of this mechanical process is generic and can be applied to any PDE. See the Appendix A for examples.

Lie Point Symmetries of PDEs. We can now define the prolongation of the action of G :

Definition 2. For symmetry group G acting on M , the prolongation of action of G on the open subset M ⊂ X × U is the induced action on M ( n ) = M × U ( n ) which transforms derivatives of a solution, u = f ( x ) , into corresponding derivatives of another solution, u ′ = f ′ ( x ′ ) . We can write this as:

Using the definition above, we can provide a new criterion for G being a symmetry group of ∆ , under a mild assumption on the PDE equation. 4

Theorem 2.1. G is the symmetry group of the n -th order PDE ∆( x , u n ) , if G acts on M , and its prolongation leaves the solution set S ∆ invariant: pr ( n ) g · ( x , u ( n ) ) ∈ S ∆ , ∀ g ∈ G .

Finally, we can express the symmetry condition in terms of the infinitesimal generators v of G :

Theorem 2.2 (Infinitesimal Criterion) . G is a symmetry group of the PDE ∆( x , u n ) if for every infinitesimal generator v of G , we have that

Example 2 (A Symmetry of the Heat Equation) . As an illustrative example, we can consider the heat Eq. (5) and will show that the following vector field generates a symmetry group for this PDE: v = 2 νt∂ x -xu∂ u . We need to find the first prolongation ϕ ( t ) and the second prolongation ϕ ( xx ) , where ϕ = -xu . Using the prolongation formula given in Eq. (7) , we get:

Clearly pr (2) v [∆] = 0 when ∆ = 0 , hence v is a symmetry of the heat equation.

3 Methods

3.1 Solving PDEs with Different Initial/Boundary Conditions with PINNs

Wang et al. [2021] combines the approach introduced in Lu et al. [2021] with the PINN loss to solve PDEs with different initial or boundary conditions without requiring retraining of the model as the original PINN model does. We will also use this proposed framework to examine the effect of enforcing the symmetry condition of the PDE on the model.

Recall that we want to model the operator O : A → U , where A is the space of initial condition functions and U is the space of PDE solutions. Our model consists of two neural networks: e θ 1 embeds the initial condition function, and g θ 2 embeds the independent variables, [ t, x ] ∈ R p .

In particular, to embed the initial condition function f ( x ) = u (0 , x ) ∈ R p , it is sampled at fixed points { x 1 , . . . , x n } , and the concatenated values are fed to f θ 1

The final prediction is the inner product of these embedding vectors:

where θ = ( θ 1 , θ 2 ) . In Algorithm 1, we use the notation u θ to denote the operator O θ and use ( x l, u l ) 1: N l to include both boundary and initial condition samples.

While we acknowledge recent architectural improvements to DeepONets [Krishnapriyan et al., 2021], since our goal is to showcase the effectiveness of symmetries, we deploy MLPs for both networks.

3.2 Imposing the Symmetry Criterion

To further inform PINNs about the symmetries of the PDE, we use an additional loss term L sym . Conveniently, this loss sometimes also contains the PDE loss of Eq. (2).

Recall that the infinitesimal criterion of Theorem 2.2 requires that by acting on a solution ( x , u n ) in the jet space using the prolonged infinitesimal generator pr ( n ) v , the PDE equation should remain satisfied. In simple terms, our symmetry loss encourages the orthogonality of pr ( n ) v and the gradient of ∆ - in other words, infinitesimal symmetry transformations are encouraged to fall on the level-sets of ∆ , maintaining ∆ = 0 . Next, we elaborate on this procedure.

Assume that the Lie algebra of the symmetry group of the n -th order PDE, ∆ , is spanned by K independent vector fields, { v 1 , . . . , v K } , where each v k is defined as in Eq. (6). As noted earlier, for each v k , we can obtain their prolongations using automatic differentiation and create a vector of the corresponding coefficients:

This is a vector of infinitesimal transformations in the jet space. 5

We also use the notation J ∆ for the gradient of ∆ wrt all independent and dependent variables: J ∆ = ( ∂ ∆ ∂x i , ∂ ∆ ∂ u α J ) .

The symmetry loss encourages the orthogonality of each of K prolonged vector fields and the gradient vector above on points inside the domain:

An alternative is to minimize the absolute value of cosine similarity. We found both of these to work well in practice.

Therefore, the total loss we use to train the two networks consists of the PINN loss introduced in Eq. (2) and the symmetry loss: L = α L PDE + β L data -fit + γ L sym , where α , β and γ are hyperparameters. However, as we see through examples, one or more symmetries of a PDE often simplify to a constant times the L PDE , removing the need to separate treatment of the PDE loss. Algorithm 1 summarizes our training algorithm.

Algorithm 1 PINN with Lie Point Symmetry

4 Experiments

4.1 Heat Equation

First, we study the effectiveness of imposing the symmetry constraint on the heat equation, described in Eq. (5). This equation is a simple linear PDE with a rich symmetry group.

Symmetries. The following 6 -dimensional lie algebra spans the symmetry group of the heat equation:

For example, the infinitesimal generator v 1 corresponds to space translation, and v 5 to Galilean boost. We refer the reader to Olver [1986] for more details on the derivation.

Data. We generate simulated solutions, which we use to test the models' performance. We use Ω = [0 , L ] = [0 , 2 π ] , discretized uniformly into 256 points and assume periodic spatial boundaries. We also use [0 , T ] = [0 , 16] , discretized into 100 points. The viscosity coefficient is set to ν = 0 . 01 .

GroundTruth

10

1. 2

14

16

Prediction(NoSymmetry)

Prediction(Symmetry)

t =0.00

t=4.85

t =12.12

N,=2000

X

N, =5000

X

10000

[=

u(t,x)

u(t,x)

u(t,x)

Ground Truth

Prediction(NoSymmetry)

Prediction(Symmetry)

(a)

(b)

Similar to Brandstetter et al. [2022b,a] and Bar-Sinai et al. [2019], we represent the initial condition functions by truncated Fourier series with coefficients A k , l k , ϕ k sampled randomly, and K = 10 :

These functions are sampled at N s = 200 fixed points, which are used as input to e θ 1 in Eq. (9). We also sample a total of N l = 300 points (including the 200 points sampled at t = 0 ), used to impose the data-fit loss, L data -fit . We also randomly sample 100 and 300 of these initial conditions and use them for the validation and test datasets respectively.

Training and Experiments. The main objective of our experiments is to confirm the hypothesis that training with symmetry loss helps improve the model's prediction capability in a low-data regime. Therefore, we train the model with and without symmetry and evaluate the model's predictions on the test dataset as we increase the number of samples inside the domain, N r . To illustrate the effectiveness of the symmetries in a low-data regime, we use N f = 100 different initial conditions and test the performance as we increase N r from 500 to 2000 and 10000 . We refer to Appendix C for details on the architectures and hyperparameters.

Results. In Table 1, we can see a comparison between the performance of the two models on the test dataset of unobserved initial conditions as N r increases. We note that when trained with few samples, the model trained with symmetry loss, L sym performs significantly better than the baseline model. Fig. 2, also illustrates this point as it shows the performance of both models on a single instance from the test dataset. We note that the improvement in prediction results in the model where symmetry loss is enforced is especially significant at larger values of time, t .

We want to highlight an important detail: by using the infinitesimal criterion for enforcing symmetries, not all symmetries of the PDE will help improve the training. There are instances when the gradient of ∆ along the vector field is trivially zero, and in other instances, 'we obtain c ∆ where c is a constant. In the case of the heat equation, only the symmetry transformations from vector fields v 5 and v 6 provide useful training signals. This means that in our experiments, we can simply eliminate the PDE loss, L PDE , and only use the symmetry loss, L sym , in addition to the supervised loss.

(a)

(b)

GroundTruth

Prediction(Symmetry)

Prediction(NoSymmetry)

t=0.00

t = 0.75

t=1.88

2.0

b

1.5

4

0.5

1.0

X

0.0

2

0.5

1.0

0

0

0

1

0.5

0.0

0.5

1.5

u(t,x)

u(t,x)

u(t,x)

Q

6.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

1.5

2.0

0.0

0.5

1.0

1.5

2.0

t

t

Ground Truth

---Prediction (No Symmetry)

---Prediction(Symmetry)

4.2 Burgers' Equation

The second PDE we analyze is Burgers' Eq. (14), which combines diffusion (with thermal diffusivity ν ) and non-linear advection (wave motion). The nonlinearity of this equation makes it more complex, resulting in shock formation.

Symmetries. Burgers' equation in the form described in Eq. (14) has a symmetry group spanned by the following 5 -dimensional vector space.

However, only the last generator v 5 results in a useful training signal. The first three generators give L sym = 0 and v 4 gives L sym = c ∆ = c L PDE , for a constant c . As with the heat equation experiment, we can eliminate the PDE loss and only use symmetry and supervised losses for training.

Data. The data used to evaluate the model is obtained using the Fourier Spectral method with periodic spatial boundaries. Initial conditions are obtained similarly to the heat equation experiment, described in Eq. (13). We use ν = 0 . 1 as the diffusion coefficient. The domain is [0 , L ] = [0 , 2 π ] and [0 , T ] = [0 , 2 . 475] discretized uniformly into 256 and 100 points respectively.

Training and Experiments. For Burgers' equation, we train the model on datasets of N f = 500 initial conditions and N r = 5000 , 25000 and 100000 samples. We found that for L sym , cosine similarity works better in this case. The models' architectures are similar to those used for the heat equation described in Appendix C.

Results. Table 2 shows the average mean-squared errors on the test dataset for the two models as N r increases. We can see that, even with one symmetry group useful for training, the model trained with L sym performs better. The predictions on a single instance of the test dataset can also be seen in Section 4.2. Again, we see that the symmetry loss especially improves the model's performance for larger values of t . See Appendix B for additional plots of the prediction results. We also note that the high standard deviations in Table 2 are because, compared to the heat equation, the behaviour of the solution (specifically shock formation) varies a lot based on the initial conditions.

Conclusion and Limitations

Lie groups and continuous symmetries are historically rooted in the study of differential equations, yet to this day, their application to Neural PDE solvers has been limited. Our work presents the

foundations for leveraging Lie point symmetry in a large family of Neural PDE solvers that do not require access to accurate simulations. Using available machinery of automatic differentiation, we show that local symmetry constraints can improve PDE solutions found using PINN models.

The method we propose to leverage local symmetries has some limitations: 1) while the Lie point symmetries of important PDEs are well-known, in general, one needs to analytically derive them for a given PDE to use our approach; 2) as we mentioned in Section 4, not all symmetries of the equation will necessarily be useful for constraining the PINN. Fortunately, the usefulness of symmetries is obvious from the corresponding infinitesimal criterion, and one could limit the symmetry loss to useful symmetries; 3) while symmetries can significantly improve performance, based on our empirical observations (see Table 1), one could achieve a similar effect with PINN by increasing the sample size. These limitations motivate our future direction, which builds on our current understanding, to impose symmetry constraints through equivariant architectures.

Acknowledgments

The authors would like to thank Prakash Panangaden for insightful discussions on symmetries and Lie Group theory. We also thank Oumar Kaba for his helpful feedback on this manuscript. This research is in part supported by Canada CIFAR AI Chair and Microsoft Research. Computational resources are provided by Mila and Digital Research Alliance of Canada.

References

Yohai Bar-Sinai, Stephan Hoyer, Jason Hickey, and Michael P. Brenner. Learning data-driven discretizations for partial differential equations. Proceedings of the National Academy of Sciences , 116(31):15344-15349, 2019. doi: 10.1073/pnas.1814058116.

Kaifeng Bi, Lingxi Xie, Hengheng Zhang, Xin Chen, Xiaotao Gu, and Qi Tian. Pangu-weather: A 3d high-resolution model for fast and accurate global weather forecast. arXiv preprint arXiv:2211.02556 , 2022.

Johannes Brandstetter, Max Welling, and Daniel E. Worrall. Lie point symmetry data augmentation for neural PDE solvers. In Proceedings of the 39th International Conference on Machine Learning , volume 162, pages 2241-2256. PMLR, 2022a.

Johannes Brandstetter, Daniel E. Worrall, and Max Welling. Message passing neural PDE solvers. In The 10th International Conference on Learning Representations . OpenReview.net, 2022b.

Jayesh K Gupta and Johannes Brandstetter. Towards multi-spatiotemporal-scale generalized pde modeling. arXiv preprint arXiv:2209.15616 , 2022.

Kashinath, M. Mustafa, A. Albert, J-L. Wu, C. Jiang, S. Esmaeilzadeh, K. Azizzadenesheli, R. Wang, A. Chattopadhyay, A. Singh, A. Manepalli, D. Chirila, R. Yu, R. Walters, B. White, H. Xiao, H. A. Tchelepi, P. Marcus, A. Anandkumar, P. Hassanzadeh, and Prabhat. Physicsinformed machine learning: case studies for weather and climate modelling. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences , 379(2194): 20200093, February 2021. doi: 10.1098/rsta.2020.0093.

Dmitrii Kochkov, Jamie A. Smith, Ayya Alieva, Qing Wang, Michael P. Brenner, and Stephan Hoyer. Machine learning - accelerated computational fluid dynamics. Proceedings of the National Academy of Sciences , 118(21), 2021. doi: 10.1073/pnas.2101784118.

Aditi S. Krishnapriyan, Amir Gholami, Shandian Zhe, Robert M. Kirby, and Michael W. Mahoney. Characterizing possible failure modes in physics-informed neural networks. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021 , pages 26548-26560, 2021.

Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. In Proceedings of the 9th International Conference on Learning Representations , 2021a.

Zongyi Li, Hongkai Zheng, Nikola B. Kovachki, David Jin, Haoxuan Chen, Burigede Liu, Kamyar Azizzadenesheli, and Anima Anandkumar. Physics-informed neural operator for learning partial differential equations. CoRR , abs/2111.03794, 2021b. URL https://arxiv.org/abs/2111. 03794 .

Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nature Machine Intelligence , 3:218-229, 2021.

Grégoire Mialon, Quentin Garrido, Hannah Lawrence, Danyal Rehman, Yann LeCun, and Bobak T Kiani. Self-supervised learning with lie symmetries for partial differential equations. arXiv preprint arXiv:2307.05432 , 2023.

Tung Nguyen, Johannes Brandstetter, Ashish Kapoor, Jayesh K Gupta, and Aditya Grover. Climax: A foundation model for weather and climate. arXiv preprint arXiv:2301.10343 , 2023.

Peter J. Olver. Applications of Lie Groups to Differential Equations . Springer, 1986.

Alfio Quarteroni. Numerical Models for Differential Problems , volume 2. Springer, 2009.

Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics , 378:686-707, 2019.

Rui Wang, Karthik Kashinath, Mustafa Mustafa, Adrian Albert, and Rose Yu. Towards physicsinformed deep learning for turbulent flow prediction, 2020a.

Sifan Wang, Yujun Teng, and Paris Perdikaris. Understanding and mitigating gradient pathologies in physics-informed neural networks, 2020b.

Sifan Wang, Hanwen Wang, and Paris Perdikaris. Learning the solution operator of parametric partial differential equations with physics-informed deeponets. Science Advances , 7(40), 2021.

Sifan Wang, Shyam Sankaran, and Paris Perdikaris. Respecting causality is all you need for training physics-informed neural networks, 2022.

Sifan Wang, Shyam Sankaran, Hanwen Wang, and Paris Perdikaris. An expert's guide to training physics-informed neural networks. arXiv preprint arXiv:2308.08468 , 2023.

A Illustrative Examples

Example 3 (Obtaining the Prolongation of SO (2) ) . We can consider If X × U = R × R and the infinitesimal generator of the 2-dimensional rotation group, SO (2) :

In this 2-dimensional case, the calculation of the prolonged generator is simple:

We will work through another example of obtaining the prolongation of an infinitesimal generator of the heat equation:

Example 4 (Obtaining the Prolongation of an Infinitesimal Generator) . As an example, we will consider X × U = R 2 × R and the following infinitesimal generator, which is a symmetry of the heat equation:

where x, t denote the independent variables, u is the dependent variable and ν is a positive constant. By the prolongation formula, Eq. (7) , the first prolongation in t is given by:

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

u(x)

1.00

i(x)

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

As a final illustrative example of the symmetry criterion, we will follow Olver's example below:

Example 5. As an illustrative example of the infinitesimal criterion, we can consider a simple DE:

We can easily see that SO (2) is a symmetry group of this differential equation, using the prolongation of the generator we calculated in Example 3:

Therefore:

Since ∆ u x = 0 when ∆ = 0 , we can conclude that SO (2) is indeed a symmetry group of the equation. In fact, we can see that it transforms solutions of this differential equation to other solutions in Fig. 4.

B Implementation and Training Details

We model the two networks, g θ 1 and e θ 2 in Eq. (9) with MLPs consisting of 7 hidden layers of width 100 . This choice was based on the previous research using PINN and DeepONets for solving Burgers' equation [Wang et al., 2021]. We used elu activation as differentiable activations are required for the PDE loss. The output of the embedding vectors from both networks is 100 dimensional. We used ADAM optimizer with learning rate of 0 . 001 for the training and performed early stopping using the validation dataset.

For both the Heat equation and Burgers' equation experiments, we perform hyper-parameter tuning on the coefficients of the loss terms from the set [0 . 1 , . . . , 1 , . . . , 10 , . . . , 100 , . . . 200] . This is done separately for the baseline model and the model trained with symmetry loss, L sym , as we varied the number of samples, N r . We found that similar to PINNs, the model is sensitive to the weight given to the supervised loss for the initial conditions vs the symmetry/PINN loss. The specific coefficient values for the models trained with and without symmetry loss for the heat equation are in Table 3 and Table 4.

.

.

We also note that for Burgers' equation, we found that cosine similarity for L sym works better than the dot product. The results reported in Section 4 use cosine-similarity.

We will make the data and the code available on GitHub.

C Additional Results

In the figure below, we can see the behaviour of the two models, trained with and without symmetry loss for Burgers' equation, as we increase the number of training samples. It can be seen that, as expected, in the model trained with L sym performs significantly better with low samples inside the domain. The corresponding mean-squared errors are reported in Table 2.

To show the effect of the number of symmetry groups of a PDE in the performance of the model trained with the symmetry loss, we analyzed the performance on (average MSE over the test dataset) as we increased the number of infinitesimal generators that were used to calculate L sym for the Heat equation. The two tables below show this effect for two different models that are trained with N r = 500 and N r = 10000 unique points sampled from inside the grid, respectively. As it can be seen, including both of the useful symmetries of the Heat equation leads to the best performance compared to the models that are trained with 1 or 0 infinitesimal generators. Furthermore, as shown in previous results, we can see that the model's overall performance in all 3 cases increases as we

Ground Truth

0.0

0.5

1.0

1.5

2.0

Prediction(nosymmetry)

Prediction(symmetry)

t = 0.00

t = 0.75

t = 1.88

N=5000

N=25000

N=100000

X

0.0

u(t, x)

0

u(t, x)

0

1

0.5

0.0

u(t, x)

0.5

Exact

--Prediction(symmetry)

Prediction(nosymmetry)

(a)

(b)

sample more points inside the grid and that the effect of the symmetry loss is more pronounced in the low-data regime. We also tried training models with and without a symmetry loss for the

Heat equation using a modified MLP architecture, as suggested in Wang et al. [2020b]. We found that, unexpectedly, this architecture was performing worse than a simple MLP architecture in our experiments. However, as it can be seen in Table 6, the model trained with the symmetry loss still performs better than the one trained with just the PINN loss.

Peer reviews:
comments: SUMMARY:
This study proposes a new loss function for PINNs that imposes the symmetry of PDE.

STRENGTHS:
PINN loss comes from the equation, but the proposed loss comes from the property of the equation. The idea is insightful.

WEAKNESSES:
The additional loss increases the computational cost. With the same computational budget, one can increase the number of evaluation points (N_r?) instead of introducing the proposed loss. The comparison might not be fair.

The proposed method was only evaluated with the heat equation and Burgers' equation, which are very simple PDEs. Is the symmetry found in more practical and complicated PDEs? Then, is the proposed method useful?

The symbols are not unified. In (7), N_l or N_r? Just after (13), N_l=300 is used for data-fit, but the number of points for data-fit is N_0 in (8).

In Table 1, "1000" might be wrong.

QUESTIONS:
With the same computational cost, is the proposed method superior to the vanilla PINN?

Is the symmetry found in more practical and complicated PDEs? Then, is the proposed method useful?

LIMITATIONS:
The proposed method is limited to PDEs with known symmetry. The generality is unknown.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, a method for finding solutions to differential equations that represent physical phenomena using neural networks is considered. In particular, a method that takes symmetry into account is proposed. Specifically, the authors consider an infinitesimal generator that represents the symmetry of the equation, and constrain the prolongation of the solutions to be orthogonal to the isosurface of the solution expressed in Jet space. This is expected to improve the accuracy of the solutions.\n\nSTRENGTHS:\nAs far as I know, the use of the symmetry of the model as an infinitesimal generator, rather than in the form of conservation laws, is certainly new. This enables the use of symmetry for equations that are not derived from the variational principle. The proposed method would be reliable in the sense that the method has a theoretical basis. In addition, the paper is clearly written.\n\nWEAKNESSES:\nThe computation and implementation of symmetries and the constraints on the jet space based on them are considered to be quite difficult. On the other hand, the improvement confirmed by the numerical experimentation is not significant, so the effect is limited compared to the difficulty of implementation.\n\nQUESTIONS:\n(1) Can the computation of symmetry be automated by using computational algebraic software (e.g., Mathematica, Maple, Sigular)? \n(2) Although the improvement in numerical experiments is not significant for the cases considered in this paper, can it be effective for systems with many symmetries and conservation laws, such as integrable systems?\n(3) Perhaps I am misunderstanding something, but is it not sufficient to simply add some additional constraints? For example, does introducing the higher-order derivative of the loss function as additional cost functions (i.e. Sobolev learning) have a similar effect as the proposed method? I suppose that this can be used as an additional constraint because If the loss function is zero for all x and t, then the derivative of the loss function with respect to x and t should also be zero. In this way, it seems to me, it would be easy to create additional constraints without the laborious symmetry computations.\n\nLIMITATIONS:\nNo potential negative societal impact is expected.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work proposes a generic method to incorporate Lie point symmetry into physics-informed neural networks (PINNs) by augmenting loss function. The method leverages automatic differentiation as other PINNs do because the condition for symmetries is written using differentials. The authors demonstrated the model with symmetry gained better performance than one without symmetry.\n\nSTRENGTHS:\n* The work is based on an established mathematical theory found in Olver (1986).\n* Once the Lie point symmetries are obtained, it seems quite simple to incorporate them, thanks to automatic differentiation and the PINNs framework.\n\nWEAKNESSES:\n* The experiments look insufficient. They compared with and without symmetry but no other state-of-the-art PINN-based models. It makes the contribution of the work look limited because most of the mathematical contributions directly come from Olver (1986), and the results from the symmetry model are not good enough (in particular, Figure 2). They claimed that \"our goal is to showcase the effectiveness of using symmetries\" (lines 230-231). However, the positive effect of the symmetries is well-known to the community (e.g., Wang et al. 2021a).\n* The superiority of incorporating symmetries through loss function is unclear. I suggest the authors include data augmentation methods (e.g., Brandstetter et al. 2022a) and equivariant models (e.g., Wang et al. 2021a).\n* The authors claimed, \"Our work presents the foundations for leveraging Lie point symmetry in a large family of Neural PDE solvers that do not require access to accurate simulations\" (lines 316-318). However, another part says, \"The data for Burgers' equation is obtained using the Fourier Spectral method\" (line 299). It makes the reviewer confused about whether the method uses simulation data as supervisory signals.\n* The contribution of the paper is not clear enough. Is it a new theorem proved, an undiscovered observation found, a state-of-the-art performance, or a novel problem setting? The reviewer recommends the authors clearly state the contribution of the work and show the supporting facts.\n* The mathematical presentation lacks correctness and clarity. The reviewer found no clear definition of \"Lie point symmetry.\" The definition of $u^{(n)}$ lacks $u$ (see Olver (1986) p.97). It may be by mistake; however, this is an essential part of the theory. Thus the reviewer recommends the authors carefully re-check the manuscript for more mathematical correctness. For instance, if there is no $u$ in $u^{(n)}$, one cannot express the advection term of the Burgers' equation in the jet space.\n\nMinor points:\n* Please add an explanation about $N_s$.\n* The paper uses \"viscosity\" for the heat equation, which seems uncommon. It could be called the diffusion coefficient.\n* In the first equation of Equation (1), $t$ should be defined in an open set (see, e.g., Jürgen Jost \"Partial Differential Equations ThirdEdition\" (2012) Chapter 1). That's why we need the initial condition.\n\nQUESTIONS:\n* How long do the training and prediction take, respectively?\n* What is the meaning of \"orthogonality\" in the paper, e.g. \"our symmetry loss encourages the orthogonality of $\\mathrm{pr}^{(n)}v$ and the gradient of $\\Delta$\" (line 237)? The condition contains PDE itself, so the reviewer guesses the use of the terminology could require some explanation.\n\nLIMITATIONS:\nThe limitations are stated sufficiently in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper proposes to enhance Physics-Informed Neural Networks (PINNs) by incorporating local Lie-point symmetry into them. It is achieved by introducing an additional symmetry loss term, which requires analytic computation of the PDE’s symmetries.
This loss term is designed to encourage orthogonality between the PDE equation and the different symmetry transformations.
Evaluation is performed on 1D heat and Burgers equations.They show that the incorporation of symmetry gives better performance, especially when the number of sampled points is low.
Being unfamiliar with Lie theory, I found the theoretical to be quite challenging.
Overall, I found the paper interesting, but a bit short on the experimental side.

STRENGTHS:
- The proposed approach is theoretically motivated, and leads to good performance compared to vanilla PINN
- It doesn’t need many modifications to the vanilla PINN to implement

WEAKNESSES:
- The proposed method needs to compute the symmetries analytically first, before integrating them into the loss function. Would there be a way to automate this part?
- In the considered examples, the better performance obtained with the additional loss terms could have also been achieved by denser sampling, as has been noted. What

QUESTIONS:
- How does the number of symmetry constraints scale wrt to the dimensionality of the considered PDE?
- Could this symmetry loss term be incorporated to other PINN variants too?

LIMITATIONS:
- The experiments are only showcased on 1D PDEs
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes adding a custom loss function to training physics-informed neural networks (PINNs) to force symmetry requirements when learning solutions to PDEs. A given partial differential equation (PDE) is associated with a Lie group that acts on the space of solutions of the PDE that leaves this space invariant. The custom loss penalizes non-orthogonality between the gradient from the PDE and the symmetry constraint. This forces the updates to the solution to be on the level-set of group invariant solution. The loss is computed using the action of the Lie algebra generators on the jet space of the solutions.  \n\nThe authors show that the additional loss leads to better data efficiency and accuracy in two test case of  Heat Equation and Burgers’ Equation.\n\nSTRENGTHS:\n- The idea of adding symmetry constraints to neural networks PDE solvers is a very interesting contribution to the field.\n- The execution is novel and makes clever use of Lie algebra theory on PDEs.\n- The exact form of the loss function is clearly motivated.\n- The experimental results are compelling and convincing.\n\nWEAKNESSES:\n- The paper would benefit significantly from a more thorough ablation study.  Specifically, an exploration of the effects of incrementally adjusting the relative weighting of the different loss terms may offer crucial insights. It would be nice to analyse the data efficieny in a power low for example, as a function of the loss weights.\n- The discussion surrounding the two types of loss used in this study (cosine and orthogonality) is not fully clear. While the authors do mention these losses, there is insufficient detail regarding their selection,.\n- The mathematical notations of the paper are not clear. Overall, I've found the paper quite hard to parse. For example, the notations $\\text{pr}^{(n)}[\\Delta]$ is not really explained and I could only guess what it means. The Lie algebras are introduced suddently while not referred before. Please re write the introduction of projector using the Lie algebras.\n- There is an absence of any comparison with baseline models using data augmentation techniques or fully equivariant architecture in term of numerical results.\n\nQUESTIONS:\n- See my point on the power law before. I would plot different power laws based on different triplets of loss weights and show the coefficients.\n- Please compare to other baselines (including neural network operators and other PINNs) regarding numerical values. There are better ways to compare methods than eyeballing the graphs in Fig. 2 and Fig. 3. In particular because the errors might be very frame dependent. I would give a numerical error by averaging several runs via several seeds over several frames. In the current form, I am unable to fully appreciate the performance to the method. If this point is addressed correctly, I am open to increasing my mark.\n\nLIMITATIONS:\nLimitations have been well discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}