Lie Point Symmetry and Physics-Informed Networks
Authors: Tara Akhound-Sadegh
Laurence Perreault-Levasseur
Johannes Brandstetter
Max Welling
Siamak Ravanbakhsh
Venue: NeurIPS 2023 poster (2023)
Keywords: PDE
Lie point symmetry
Symmetry
Neural PDE solver
PINNs
TL;DR: We use Lie Point Symmetries of PDEs to Improve Physics Informed Networks.

Abstract: Symmetries have been leveraged to improve the generalization of neural networks through different mechanisms from data augmentation to equivariant architectures. However, despite their potential, their integration into neural solvers for partial differential equations (PDEs) remains largely unexplored. We explore the integration of PDE symmetries, known as Lie point symmetries, in a major family of neural solvers known as physics-informed neural networks (PINNs). We propose a loss function that informs the network about Lie point symmetries in the same way that PINN models try to enforce the underlying PDE through a loss function. Intuitively, our symmetry loss ensures that the infinitesimal generators of the Lie group conserve the PDE solutions.. Effectively, this means that once the network learns a solution, it also learns the neighbouring solutions generated by Lie point symmetries.
Empirical evaluations indicate that the inductive bias introduced by the Lie point symmetries of the PDEs greatly boosts the sample efficiency of PINNs.

Lie Point Symmetry and Physics Informed Networks
Tara Akhound-Sadegh School of Computer Science, McGill University, Mila Quebec Artificial Intelligence Institute, Montreal, Quebec, Canada
Laurence Perreault-Levasseurr
Université de Montréal, Montreal, Quebec, Canada
Ciela Institute, Montreal, Quebec, Canada
Mila Quebec Artificial Intelligence Institute, Montreal, Quebec, Canada Trottier Space Institute, Montreal, Quebec, Canada CCA, Flatiron Institute, New York, USA Perimeter Institute, Waterloo, Ontario, Canada
Johannes Brandstetter
Max Welling University of Amsterdam, Amsterdam, Netherlands
Microsoft Research AI4Science, Amsterdam, Netherlands
Siamak Ravanbakhsh
School of Computer Science, McGill University, Mila Quebec Artificial Intelligence Institute, Montreal, Quebec, Canada
Abstract
Symmetries have been leveraged to improve the generalization of neural networks through different mechanisms from data augmentation to equivariant architectures. However, despite their potential, their integration into neural solvers for partial differential equations (PDEs) remains largely unexplored. We explore the inte- gration of PDE symmetries, known Lie point symmetries, in major family of neural solvers known as physics-infone informed neural networks (PINNs). We propose loss function that informs the network about Lie point symmetries in the way that PINN models try to enforce the underlying PDE through loss function. Intuitively, symmetry loss ensures that the infinitesimal generators of the Lie group conserve the PDE solutions. Effectively, this means that once the network learns solution, it also learns the neighbouring solutions generated by Lie point symmetries. Empirical evaluations indicate that the inductive bias introduced by the Lie point symmetries the PDEs greatly boosts the sample efficiency of PINNs.
Introduction
In recent years, deep learning has accelerated data-driven approaches to science and engineering. A prominent example is the role of deep learning in solving partial differential equations (PDEs), which are ubiquitous many scientific disciplines.
"Corresponding author: akhound CContribation was done when at Microsoft Research AI4Science, Amsterdam.
3khhunddaaddehhmiin aadegh@mila.aa.aaaaaaaaaadaaaaae quebec
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

This is mainly driven by the fact that traditional handcrafted numerical solvers can be prohibitively expensive, and thus, learning to solve PDEs has the potential to significantly impact various areas of science, ranging from quantum chemistry to biology to climate science to cosmology,
18
14-
52-
50-
[Wang et al.) 2020a Kochkov et al. [002]] Kashinath et al., 2021, Gupta and Brand-
stetter 2022 Nguyen et al. 2023] Bi et al.
2022 Like any other machine learning problem, learning to solve PDEs can ben- efit from inductive biases boost sample efficiency and generalization capabilities. As such, the PDE itself and its respective Lie point symmetries, i.e. joint symmetries of coordinate and field transformations, are two natural inductive biases for constructing efficient neural PDE solvers.
All the solutions of the differential equation of harmonic oscillator can be reached via symmetry transfor- mations of single solution. Its Lie-point symmetry group is the special linear group. SL(3), corresponding to eight one one- pere-aereeer subgroups transforming a given solution, iden- tified using different colour in this figure. While, in general, Lic-poing symmetries form more than one orbit that is, we cannot reach all solutions using such transformations. they can significantly narrow down the solution space.
Figure 1:
For parametric PDEs, Physics-Infoe Informed Neural Networks (PINNs) have emerged an efficient new learning paradigm. PINNs constrain the solution network to satisfy the underlying PDE via the PINN loss terms (Raissi et al., 2019), which comprise a residual loss, as well as loss terms for initial and boundary points. As such, PINNs can be seen as a data-free learning approach, utilizing available physics information to guide neural network training. Lie point symmetries are uniquely special in that it is possible to characterize the full set of permis- sible transformations for each PDE, and thus naturally offer additional physics information which piggy-backs the data-free learning objective. Lie point symmetries have been introduced to deep learning by [Brandstetter et al. [022a] for Lie point symmetry data augmentation which places firm mathematical footing on data augmentation pipelines for neural PDE solvers. Further, Lie point sym- metries showed promising results when used in extending the framework to self--uuprvised learning for neural PDE solvers [Mialon et al. 2023). In this work, for the first time we integrate symmetries into PINNs and demonstrate the effectiveness of this symmetry regularization on generalization capabilities.
Lie point symmetries of PDE, by definition, map a solution to a solution, preserving the PDE. Lie point symmetry transformations act on both independent (e.g., time and space), and dependent (PDE solution) fields and, by extension, on all partial derivatives. To enforce Lie point symmetry, we first need to prolong symmetry transformation to find how it transforms the partial derivatives of the respective PDE. We show how to do this using automatic differentiation, which enables us to transform the entire PDE for each infinitesimal generator of each Lie point symmetry. "Imposing symmetry" means that by transforming the PDE via a symmetry transformation, the PDE is required to be preserved. In practical terms, Lie point symmetries should be orthogonal to the PDE gradient. Each Lie Point symmetry results in one such orthogonality constraint, which we enforce using a penalty term within the PINN framework. We can enforce these the same points where the PINN loss is imposed. Importantly, our Lie Point symmetry regularization is complementary to the PINN loss: while the PDE loss regularizes the neural network to satisfy the PDE select points, the Lie point symmetry loss regularizes the neural network such that infinitesimal changes in different directions continue to satisfy the underlying PDE.
2 Background
2.1 Partial Differential Equations
Partial differential equations (PDEs) are used to mathemattcally describe the dynamics of various physical systems. In a PDE, the evolution of function that involves several variables is described in terms of local updates expressed by partial derivatives. Since temporal PDEs especially prevalent

---

in physical sciences, we consider PDEs of the following general form: E [0, T,
u, + Dx[ul 0,
(I)
(00 x) = f(x), (t, x) =t(t,)),
X E 92
x 02, t
[0,
where x(t,x) P is the solution the PDE, t denotes the time, and x a vector of possibly multiple spatial coordinates. f C IR' is the domain and Px[-] is non-linear differential operator. f(x) is known as the initial condition function and g(t, x) describes the boundary conditions. In the following, we sometimes explicitly separate the time from other independent variables and sometimes for convenience group them together.
Example 1 (Heat Equation). The one-dimensional heat equation describes the heat conduction in one-dimensional rod, with viscosity coefficient Its differential operator given by Da. [u] -Vlaa-
Using Neural Networks for Solving PDEs, For many systems, obtaining an analytical solution to the PDE is impossible; hence, numerical methods are traditionally used to obtain approximate solutions. Numerical solvers, such as finite element methods (FEM) or finite difference methods (FDM) rely discretizing the space T ! [Quarteroni] 2009]. The topology of the space has to be taken into account when constructing the mesh, and the resolution of the discretization will affect the accuracy of the predicted solutions. Additionally, these solvers often computationally expensive, especially for complex dynamics, and each time the initial or boundary conditions of the PDE change
the solver must be rerun. These considerations and constraints make designing numerical solvers difficult, and scientists often need to handcraft a specific solver for an application [Quarteroni 2009).
Given the recent successes of neural networks, especially in dealing with large datasets, using deep learning to solve PDEs has become promising direction. The idea is to learn a function or, more generally, a functional that bypasses the numerical computation and produces the solution to the PDE in single shot or by progression through time. Broadly, there are two approaches solving PDEs with neural networks: neural operator methods which approximate the solution operator that maps between solutions of the underlying PDEs, and direct methods which learn the underlying solution function. The most prominent representative of the latter are PINNs.
PINNs. In contrast to neural operator methods et al.
2021 Li al.
2021a where the main
idea is to generalize neural networks to obtain mappings between function space, PINNs directly learn the solution function of the underlying PDE. Consequently instead of relying on large training sets,
PINNs operate as a surrogate model for the PDE solution, trained directly with the PDE equation itself. The simplicity of the PINN idea has made it the subject of many follow-up improvements; see
Krishnapriyan et al. [2021). Wang et al. 2020b 2022 2023
In PINNs, the PDE solution u(t, x) of Eq. I is a neural network us (t, x) with parameters 0. The loss function is then compromised of two parts:
[(0) = LPDE + Ldata-ft
The first term is the physics-informed objective, ensuring that the function learned by the neural network satisfies the PDE Eq. (1. Let r(t, x) denote the residual error in agreement with the PDE equation:
2 at us (t, x) (t,
(t,
where the derivatives of the solution network ug are calculated using automatic differentiation [Rissi ett al. 22019). This penalty is then imposed on finite set of points (f, x)INN, which are sampled from inside the domain [0, T x to obtain the PDE loss:
LPDE
Nr Σ [re(tt,
(3)
The second term in Eq. 2). is supervised loss which ensures that the function learned by the neural network satisfies the initial and boundary conditions of the problem - that is
N, [0, x!) - f(
No
Ldata-fit fit
- 9 D
(4)
N.
No
3

---

where (x") )I.N. E S are N, samples at which the initial condition function f is sampled. (t", x(",x")x 1x.N, are No points sampled on the boundary (from [0. T] x aS2).
There have also been multiple models that combine the operator learning approach with the physics- informed loss. For example, in Li et al. [021b], the NO approach is combined with the PINN loss.
Another example is Waann et al. 2022 which we describe further in Section 3.1
2.2 Symmetries
Groups.
Symmetries essentially transformations of the object that leave aspect of it invariant: for example, the nature of an object does not change we translate or rotate it. In mathematics, symmetries of an object described by abstract objects known as groups. More concretely, group a set G and a group operation : G' x G - G satisfying: associativity, the existence of an identity element, and the presence of inverse element for every element in the set.
Lie Groups. Groups that are also differentiable manifolds are known as Lie groups, and are important in the study of continuous symmetries. In Lie groups, in addition to satisfying the three properties, the group operation and its inverse are smooth maps. Each Lie group has an associated Lie algebru which is its tangent space, describe the smooth transformations of the Lie groups in the limit of infinitesimal transformations. The elements of the Lie algebra are vectors describing the direction of the infinitesimal symmetry transformation.
vector space, at identity.
Intuitively, Lie algehras
One Parameter Subgroups. The Lie group, G, can be multi-dimensional and complex. When the n-dimensional group simply connected, it is often represented in terms of series of one-parameter transformations, g = (e1 (1)))((())) ... (n()),, where 9: R - G, i = 1,.... Ti and such that (6) = gi + 8). The e's are the real parameters of the transformation, and each gi is a continuous group homomorphism (i.e., smooth, group-structurg map) from this parameter to the symmetry group.
2.3 Symmetries of Partial Differential Equations
In this section, we will provide the mathematical background of symmetries of differential equations. We will mostly follow the exposition presented in Olver [9866] and refer the reader to this original text for more in-depth treatment of this topic.
In the context of differential equations, symmetries transformations that map solution of the PDE to another solution. For example, in Fig. [ we can see how the solutions of simple harmonic oscillator can be obtained via symmetry transformations of given solution, where the symmetry group is SL(3). Consider the PDE A, involving p independent variables x = (I)...., Σp) E X and q dependent variables u (u1,-... "y) U. The solutions to the PDE will be of the form -q, for x E S2 where Ω C X is the domain of the function The
u4 = f pp) for = 1....... symmetry group of A, is the local group of transformations
an open subset of the space of dependent and independent variables, M C X x U, transforming solutions of A to other solutions.
Prolongations. To formalize this abstract definition of PDE symmetries, Lie proposed viewing A as concrete geometric object and introduced the concept of prolongation [Olver] 1986). The idea to prolong the space of independent and dependent variables, X U, to space that also represents the partial derivatives involved in the PDE. More concretely, we have the following definition:
Definition 1. The n-th order prolongation (or n-th order jet space) of X x U is denoted as X U(m) X x U1 x ... x Un, whose coordinates represent the independent and dependent variables well all the partial derivatives of the dependent variables up order
Equivalently we have the notion of prolongation of as (n)) = (ux; ulxx..... [nxx where Uix is all the unique fth derivatives of u, for i 1,..., For example, if x = (I, y), then uxx (8z. u, Ozgu, u).
Formally, the Lie algebra vector space equipped with binary operation known as the Lie bracket.

---

Using this notion of prolongation, we can represent a PDE as an algebraic equation, A(x, u(*)) 0,
where A is the map that determines the PDE, i.e., A : X x ((()) - IR. In other words, the PDE tells us where the map A vanishes on X x U(a).
For example, for the heat equation described in 1 A is given by: A((x, The graph of all prolonged solutions is the set SA C X x U(a), and is defined as Sa = {(x, u u()): Axx, ( = 0}. In this new notation, we can say that u(x) is solution of the PDE if r {(x, [(x))] C SA- where pr())(() X -> U is a vector-valued function whose entries represent all derivatives of u wrt x up to order n.
ue Vilee
(5)
Prolongations of the Infinitesimal Generators. Let v be the vector field on the subspace M C X with corresponding one-pan mee-aammeter subgroup exp (cv). In other words, the vector field is the infinitesimal generator of the one perameeteer subgroup. Intuitively, this vector field describes the infinitesimal transformations of the group to the independent and dependent variables, and we can
write it as:
(x u
Da (x, u)
(6)
Ox
Oua
a=1
where <'(x, u) and 6 (x, u) are coordinate-dependée coefficients. To study how symmetries transform solution to another solution, we need to know how they transform the partial derivatives and, therefore, the jet space X x U(m).
A symmetry transformation of the independent (x) and dependent (u) variables will also induce transformations in the partial derivatives Ux: Uyx: The prolongation of the infinitesimal generator, vr is a generalization of the generator v which describes these induced transformations in these partial derivatives.
This prolongation will be defined on the jet-space x (n) and is given by:
a
4
Σ (x,,u)) u)
0a
(2) (x. u)
Ox
Oug
where we have used the notation J (i....., ix) for the multi-indices, with S in S p and
0 < k < n and uř dr'i 8²u" the prolongation formula, involves the total derivative operator (see Olver (1986) for derivation):
00'' Calculating da the coefficients of Oug, can be done using
Oug 3
p E
Oua
D +
where
Qa da
(8)
Fxt
Oxt
The upshot is that we can mechanically calculate the prolonged vector field using partial derivatives of u, which are, in turn, produced by automatic differentiation. In practice, prolonged vector fields are implemented as vector-valued functions (or functionals) of x and u. Therefore, the implementation of this mechanical process is generic and can be applied to any PDE. See the Appendix for examples.
Lie Point Symmetries of PDEs.
We can now define the prolongation of the action of G:
Definition 2. For symmetry group G acting on M, the prolongation of action of 9 on the open subset M C X x U is the induced action on (()) = M x (()) which transforms derivatives of solution, u fxx), into corresponding derivatives of another solution, u' fxx'.. We can write this as:
(m) (x, (") (g x, (r g (")).
Using the definition above, we can provide a new criterion for g being a symmetry group of A, under a mild assumption on the PDE equation fl
"This assumption is that A is of maximal rank. We refer the readers to Olver 1986) for the detinition of this condition. However, we note that this assumption does not pose restriction since for any PDE not satisfying this condition, it is possible to find equivalent PDE which does.
5

---

Theorem 2.1. G is the symmetry group of the n-th order PDE A(x, u"), if g acts on M. and its prolongation leaves the solution set Sa invariant: pr(") g. (x, u()) E Sa. Vg g.
Finally, we can express the symmetry condition in terms of the infinitesimal generators v of g: Theorem 2.2 (Infinitesimal Criterion). G is symmetry group of the PDE Axx,u") u²) if for every infinitesimal generator v of G, we have that
v[[ =0 when -
Example (A Symmetry the Heat Equation). As an illustrative example, we can consider the heal Eq. (5) and will show that the following vector field generates symmetry group for this PDE: V 2vtt, zudw. We need to find the first prolongation Φit) and the second prolongation &f d(zx))
where o
Using the prolongation formula given in Eq. 7 we get:
(1) - -2vug and
g( = 22u - Fuge
Now: pr v () vo((z)) Iu 2/1
(2 IVUza) z(ue vuzu)
Clearly pr(2) v = 0 when A 0, hence is symmetry of the heat equation.
3
Methods
3.1
Solving PDEs with Different Initial/Boundary Conditions with PINNS
Wang et al. 2021 combines the approach introduced Lu et al. [2021] with the PINN loss to solve PDEs with different initial or boundary conditions without requiring retraining of the model as the original PINN model does. We will also use this proposed framework to examine the effect of enforcing the symmetry condition of the PDE the model.
Recall that we want to model the operator O : A - U, where A the space of initial condition functions and U is the space of PDE solutions. Our model consists of two neural networks: es. embeds the initial condition function, and ge, embeds the independent variables, [t. x] E R2.
In particular, to embed the initial condition function f(x) = u(0, x) E IP, it is sampled at fixed points {x1..... X,, ), and the concatenated values are fed to fo.
The final prediction the inner product of these embedding vectors:
(x)
eT, (u(0, xI), u(0, x..) go, (x, f)
where 0 = (01, In Algorithm D we use the notation no to denote the operator Og and use (xl, u') I.N, to include both boundary and initial condition samples.
While we acknowledge recent architectural improvements to DeepONets (Krishnapriyan et al. 2021]. since our goal is to showcase the effectiveness of symmetries, we deploy MLPs for both networks.
3.2 Imposing the Symmetry Criterion
To further inform PINNs about the symmetries of the PDE, we use an additional loss term Laym- Conveniently, this loss sometimes also contains the PDE loss of Eq.
Recall that the infinitesimal criterion of Theorem 2.2 requires that by acting on a solution (x, u²) in the jet space using the prolonged infinitesimal generator (r))),, the PDE equation should remain satisfied. In simple terms, our symmetry loss encourages the orthogonality (r and the gradient of A - in other words, infinitesimal symmetry transformations are encouraged to fall on the level-sets of A, maintaining A Next, we elaborate on this procedure.
Assume that the Lie algebra of the symmetry group of the n-th order PDE, A, is spanned by K independent vector fields, fvi...., Vx J. where each vù is defined as in Eq. (6). As noted earlier, for each VA. we can obtain their prolongations using automatic differentiation and create a vector of the corresponding coefficients:
coef vw)) [&) ... 5th og. ......, oy (00 Za
(0
)*]
(10)
This is vector of infinitesimal transformations in the jet space
F
Using coef in the equation above is to differentiate the abstract definition of Eq. 6 and vector of its coefficients.

---

We also use the notation J/ for the gradient of A wit all independent and dependent variables:
JA =
0A
0A
Or duy
The symmetry loss encourages the orthogonality of each of K prolonged vector fields and the gradient vector above on points inside the domain:
K
Laym Σ
co
(11)
An alternative is to minimize the absolute value of cosine similarity. We found both of these to work
well in practice.
Therefore, the total loss we use to train the two networks consists of the PINN loss introduced in Eq. 2 and the symmetry loss: L aLPDE + -ittta-fit + -aymm where a, 3 and 7 are hyperparameters. However, as we see through examples, one or more symmetries of a PDE often simplify constant times the CPDE, removing the need to separate treatment of the PDE loss. Algorithm summarizes our training algorithm.
Algorithm PINN with Lie Point Symmetry
inputs:
A: the PDE equation of order n,
(VR)1.K infinitesimal generators of symmetries of A
D -= i (x, u)1.N, u1:x.) ,.,," dataset for N, different initial conditions init: initialize parameters of network uo
for iteration do
Sample from D
calculate Laym:
calculate (nn) using automatic differentiation for X1:N,
calculate (or using XI.N,, (u(n))in, auto diff and Eq.
calculate Laym using (or((pr(") vi. and A and equation Eq.
calculate LPDE:
using A, (u(())),N, and equation Eq.
calculate Ldata-fit calculate ú ux(,, uI:N,) for X1:N, calculate L using U1:N, i1:, and equation Eq.
L = aLDDE + _Latta fit +uyy 7
0 +- 0 v
end for return Ui
Experiments
4.1 Heat Equation
First, we study the effectiveness of imposing the symmetry constraint on the heat equation, described in Eq. (5). This equation is simple linear PDE with rich symmetry group.
Symmetries. The following 6-dimensional lie algebra spans the symmetry group the heat equation:
Vi 0n V3 = O, V2 = 0 V1 = rd2 + 2t0, Vi = 4vtro, - 4vt2 - (x² + 2vttuud
V5 2v10, xuδ,
(12)
For example, the infinitesimal generator v1 corresponds to space translation, and Us to Galilean boost. We refer the reader to Olver (986) for more details the derivation.
Data.
We generate simulated solutions, which we use to test the models' performance. We use
0, f.] [0, 2m). discretized uniformly into 256 points and assume periodic spatial boundaries. We also use [0, T] 16), discretized into 100 points. The viscosity coefficient is set to 0.01.
7

---

Ground Truth
Prediction (No Symmetry)
Prediction (Symmetry)
t
4
1212.12
N, 2000
N, 1000 N, = 5000
ottid
uftx)
uft.)
Truth
Prediction Symmetry)
Prediction (Symmetry)
(a)
(b)
Figure 2: The effect of training the PDE solver for the heat equation with and without the symmetry loss for one of the PDEs in the test dataset, (a) shows the ground truth solution and the predictions of the two models as the number of samples inside the domain increases from 500 2000 and 1000.(b) shows the corresponding predictions and the ground truth solution different slices.
Similar to Brandstetter al. 2022b a and Bar-Sinai 2019), we represent the initial condition functions by truncated Fourier series with coefficients Ag, 4 Φ% sampled randomly, and K = 10:
K
u(t = 0, f(x)
Σ
A (2TT22//// σx)
(13)
These functions are sampled at N₂ = 200 fixed points, which are used as input to eø, Eq. ( also sample total of Ni 300 points (including the 200 points sampled at 0), used to impose the data-fit fit loss, Ldata-fit- We also randomly sample 100 and 300 of these initial conditions and use them for the validation and test datasets respectively.
Training and Experiments. The main objective of our experiments to confirm the hypothesis that training with symmetry loss helps improve the model's prediction capability in low-data regime. Therefore, we train the model with and without symmetry and evaluate the model's predictions the test dataset effectiveness the symmetries a low-data regime, we use N, 100 different initial conditions and test the performance as we increase N, from 500 2000 and 10000. We refer Appendix C for details on the architectures and hyperparameters.
increase the number of samples inside the domain, N,. To illustrate the
Results. In Table we can see a comparison between the performance of the two models on test dataset of unobserved initial conditions as N, increases. We note that when trained with few samples, the model trained with symmetry loss, Laym performs significantly better than the baseline model. Fig. 2 also illustrates this point it shows the performance of both models instance from the test dataset. We note that the improvement in prediction results in the model where symmetry loss is enforced is especially significant at larger values of time, t.
single
We want to highlight an important detail: by using the infinitesimal criterion for enforcing symmetries, not all symmetries of the PDE will help improve the training. There are instances when the gradient of A along the vector field trivially zero, and in other instances, 'we obtain eλ where is a constant. In the case of the heat equation, only the symmetry transformations from vector fields and ve provide useful training signals. This means that our experiments, we can simply eliminate the PDE loss. LPDE, and only the symmetry loss, Layms in addition to the supervised loss.

---

Table 1: The average test set mean- squared error for the Heat equation.
Table 2: The average test set mean- mean--suared error for Burgers' equation.
Number Points
No Symmetry
Symmetry
Number of Points (N,)
Symmetry
Symmetry
(N,)
500
1.12 0.58
0.30 + 0.15
5000
0.041 0.042
0.034
0.039
2000
0.36 1 0.19
0.24 L 0.14
25000
0.030 0.038
0.017
0.020
10000
0.22 1 0.14
0.21 1 0.13
100000
0.018 0.022
0.013 : 0.020
Ground Truth
Preciction (Symmetry) Prediction (No Symmetn)
0.75
1.88
Ground Truth Prodiction Symmetyy ...-Predccion (Bymmatry)
(b)
Figure 3: (a) predictions of the network trained with a total of 100000 points inside the domain (500 initial conditions and only 200 points for each PDE), with and without the symmetry loss, for Burgers' equation on data. (b) corresponding predictions different time slices showing that the model trained with symmetry loss performs better, especially at larger t.
4.2 Burgers' Equation
The second PDE we analyze is Burgers' Eq. (1), which combines diffusion (with thermal diffusivity v) and non-linear advection (wave motion). The nonlinearity of this equation makes it more complex, resulting in shock formation.
=
Vila
(14)
Symmetries. Burgers' equation in the form described in Eq. 14 has symmetry group spanned by the following 5-dimensional vector space.
V1 0₂ V3 102- On V2 = 0 V4 = xd,, + 21dl - v...
V5 tx02 120, (x -.
(15)
However, only the last generator v, results in a useful training signal. The first three generators give Luym 0 and Va gives faym cλ = cLPDg, for a constant As with the heat equation experiment, we can eliminate the PDE loss and only use symmetry and supervised losses for training.
Data. The data used to evaluate the model is obtained using the Fourier Spectral method with periodic spatial boundaries. Initial conditions are obtained similarly to the heat equation experiment, described in Eq. (3). We use = 0.1 as the diffusion coefficient. The domain is [0, L] = [0, 2x] and [0. T J0, 2.475] discretized uniformly into 256 and 100 points respectively.
Training and Experiments. For Burgers' equation, we train the model on datasets of N, 500 initial conditions and N, = 5000, 25000 and 100000 samples. We found that for Laym cosine similarity works better in this case. The models' architectures are similar to those used for the heat equation described in Appendix
Results. Table 2 shows the average mean-squared errors on the test dataset for the two models as N, increases. We can see that, even with one symmetry group useful for training, the model trained with Laym performs better. The predictions
single instance of the test dataset Section 4.2 Again, we see that the symmetry loss especially improves the model's 's performance for larger values of t. See Appendix B for additional plots of the prediction results. We also note that the high standard deviations in Table are because, compared to the heat equation, the hehaviour of the solution (specifically shock formation) varies lot based the initial conditions.
also be seen in
Conclusion and Limitations
Lie groups and continuous symmetries are historically rooted in the study of differential equations, yet to this day, their application to Neural PDE solvers has been limited. Our work presents the

---

foundations for leveraging Lie point symmetry in a large family of Neural PDE solvers that do not require access to accurate simulations. Using available machinery of automatic differentiation, we show that local symmetry constraints can improve PDE solutions found using PINN models.
The method we propose to leverage local symmetries has some limitations: 1) while the Lie point symmetries of important PDEs are well-known, in general, one needs to analytically derive them for a given PDE to use our approach: 2) as we mentioned in Section not all symmetries of the equation will necessarily be useful for constraining the PINN. Fortunately, the usefulness of symmetries is obvious from the corresponding infinitesimal criterion, and one could limit the symmetry loss to useful symmetries; 3) while symmetries can significantly improve performance, based on our empirical observations (see Table D.. one could achieve similar effect with PINN by increasing the sample size. These limitations motivate our future direction, which builds on our current understanding, to impose symmetry constraints through equivariant architectures.
Acknowledgments
The authors would like to thank Prakash Panangaden for insightful discussions on symmetries and Lie Group theory. We also thank Oumar Kaba for his helpful feedback this manuscript. This research is part supported by Canada CIFAR Al Chair and Microsoft Research. Computational resources provided by Mila and Digital Research Alliance of Canada.
References
Yohai Bar-Sinai, Stephan Hoyer, Jason Hickey, and Michael P. Brenner. Learning data-driven discretizations for partial differential equations. Proceedings of the National Academy of Sciences, 116631):1534-11554, 2019. doi: 10.1073/pmas. 1811...1111 188105811 16.
Kaifeng Bi, Lingxi Xie, Hengheng Zhang, Xin Chen, Xiaotao Gu, and Qi Tian. A 3d high-resolution model for fast and accurate global weather forecast. arXiv preprint arXiv:2211.02553 2022.
Johannes Brandstetter, Max Welling, and Daniel E. Worrall. Lie point symmetry data augmentation for neural PDE solvers. In Proceedings of the 39th International Conference on Machine Learning, volume 162, pages 2241-2256. PMLR, 2022a.
Johannes Brandstetter, Daniel E. Worrall, and Max Welling. Message passing neural PDE solvers. In The 10th International Conference on Learning Representations. OpenReview.net, 2022b.
Jayesh K Gupta and Johannes Brandstetter. Towards muliispaatocempprra-ssceel generalized pde modeling, arXiv preprint arXiv:2209.1561 2022.
K. Kashinath, M. Mustafa, A. Albert, J-L. Wu, C. Jiang, S. Esmaeilzadeh, K. Azizzadeneshelli R. Wang, A. Chattopadhyay, A. Singh, A. Manepalli, D. Chirila, R. Yu, R. Walters, B. White, H. Xiao, H. A. Tchelepi, P. Marcus, A. Anandkumar, P. Hassanzadeh, and Prabhat. Physics- informed machine learning: case studies for weather and climate modelling. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 379(2194): 20200093, February 2021. doi: 10. 10088/9sta22020000000
Dmitrii Kochkov, Jamie A. Smith, Ayya Alieva, Qing Wang. Michael P. Brenner, and Stephan Hoyer. Machine learning accelerated computational fluid dynamics. Proceedings of the National Academy of Sciences, 118(21), 2021. doi: 10. 210077/81
Aditi S. Krishnapriyan, Amir Gholami, Shandian Zhe, Robert M. Kirby, and Michael W. Mahoney, Characterizing possible failure modes in physics-infommn neural networks. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, pages (6548-265666 2021.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. In Proceedings of the 9th International Conference on Learning Representations, 202
10

---

Zongyi Li. Hongkai Zheng, Nikola B. Kovachki, David Jin, Haoxuan Chen, Burigede Liu, Kamyar Azizzadenesheli, and Anima Anandkumar. Physics-informet informed neural operator for learning partial differential equations. CoRR, abs/2111.03794, 2021b. URL httph:ttp///// pectps://arxiv.ooo wrwp:ii.oor/abba. abs/ 2111
03794
Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nature Machine Intelligence, 2-218-229 2021.
Grégoire Mialon, Quentin Garrido, Hannah Lawrence, Danyal Rehman, Yann LeCun, and Bobak Kiani. Self-supervised learning with lie symmetries for partial differential equations. arXiv preprint arXiv:2307.05432, 2023.
Tung Nguyen, Johannes Brandstetter, Ashish Kapoor, Jayesh K Gupta, and Aditya Grover. Climax: foundation model for weather and climate. arXiv preprint arXiv:2301.10343 2023.
Peter J. Olver. Applications of Lie Groups to Differential Equations. Springer, 1986.
Alfio Quarteroni. Numerical Models for Differential Problems, volume 2. Springer, 2009.
Maziar Raissi, Paris Perdikaris, and George Karniadakis. Physics-informed neural networks: A
deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686-707, 2019.
Rui Wang, Karthik Kashinath, Mustafa Mustafa, Adrian Albert, and Rose Yu. Towards physics- informed deep learning for turbulent flow prediction, 2020a.
Sifan Wang. Yujun Teng, and Paris Perdikaris. Understanding and mitigating gradient pathologies in physics-informed neural networks, 2020b.
Sifan Wang, Hanwen Wang, and Paris Perdikaris. Learning the solution operator of parametric partial
differential equations with physics- informed deeponets. Science Advances, 7(40), 2021.
Sifan Wang. Shyam Sankaran, and Paris Perdikaris. Respecting causality is all you need for training physics-informed neural networks, 2022.
Sifan Wang, Shyam Sankaran, Hanwen Wang, and Paris Perdikaris. An expert's guide to training physics- neural networks. arXiv preprint arXiv:2308. 08468, 2023.

---

A Illustrative Examples
Example (Obtaining the Prolongation of SO(2)). We consider If X infinitesimal generator of the 2-dimensional rotation group, SO(2):
U R R and the
SSO(2) [(x, u)0, + Φ(z, u)O,
-u, + xd.
In this 2-dimensional case, the calculation of the prolonged generator is simple:
Φf()) = (2 - {uz) Eurr
D2 mux) Wuyr
(1 uzz + uuzz) uugz
ux
Therefore:
pr VSo(2) -ud xdw + (1 (1+u2)00..
We will work through another example of obtaining the prolongation of an infinitesimal generator of the heat equation:
Example 4 (Obtaining the Prolongation of Infinitesimal Generator). As an example, will consider X x U IR2 x R and the following infinitesimal generator. which is symmetry of the heat equation:
(1),,t + (z t, uja + 6(2,t,u)0.
-vvt0= xud,
where æ, & denote the independent variables, u the dependent variable and v a positive constant. By the prolongation formula, Eq.
7 the first prolongation in t is given by:
Φ² Dz( E E24+) + Eiu + ExWet
D2( 10(--z1 zu 22tuz) + 2vtuzt
21/U..
Figure 4: Various solutions of the PDE A(x, u, Ux) (u x u) + a + obtained via symmetry
transformation (rotation) of know solution (in red).
As final illustrative example of the symmetry criterion, we will follow Olver's example below: Example As an illustrative example of the infinitesimal criterion, we can consider a simple DE: ( u, u.) (u x))u, + u
We can easily that S0(2) is symmetry group of this differential equation, using the prolongation of the generator we calculated in Example
(1
-uA, + FA, (1 A -uu(1 uz) x(1 + ux) + (1 ++.)+)
u.)(u x)
u,A A
12

---

Since Aug O when A O, we can conclude that SO(2) is indeed symmetry group of the equation. In fact, we can see that it transforms solutions of this differential equation other solutions in Fig.
B Implementation and Training Details
We model the two networks, 90, and co2 in Eq. (9) with MLPs consisting of hidden layers of width 100. This choice was based on the previous research using PINN and DeepONets for solving Burgers' equation [Wang et al. 202T]. We used clu activation as differentiable activations are required for the PDE loss. The output of the embedding vectors from both networks 100 dimensional. We used ADAM optimizer with learning rate of 0.001 for the training and performed early stopping using the validation dataset.
For both the Heat equation and Burgers' equation experiments, we perform hyper- parameter tuming on the coefficients of the loss terms from the set [O.1..... .......... 10..... 100,...0....... 200]. This is done separately for the baseline model and the model trained with symmetry loss, L-ym. as we varied the number of samples, Ny. We found that similar PINNs, the model is sensitive to the weight given to the supervised loss for the initial conditions vs the symmetry/PINN loss. The specific coefficient
values for the models trained with and without symmetry loss for the heat equation are Table 3 and T
Table The loss coefficients the model trained with symmetry (i.e. L BLeym + Ld for the heat equation various number of unique points sampled inside the grid for training, N₂
loss coefficient
N, = 500
N,
2000
N = 10000
B
100
80
40
20
20
20
Table he loss coefficients for the model trained without the symmetry loss (i.c. C. = CCCINNN + dfataa-Bt for the heat equation for various numher of unique points sampled inside the grid for training. N,.
loss coefficient
N, 500
N,
2000
N, 10000
Qx
150
150
130
7
20
20
20
We also note that for Burgers' equation, we found that cosine similarity for Laym works better than the dot product. The results reported in Section 4 use cosine-similarity
will make the data and the code available on GitHub.
C Additional Results
In the figure below, we can see the behaviour of two models, trained with and without symmetry loss for Burgers' equation, as we increase the number of training samples. It can be seen that, as expected. in the model trained with Caym performs significantly better with low samples inside the domain. The corresponding mean-squared errors are reported in Table 2
To show the effect of the number of symmetry groups of a PDE in the performance of the model trained with the symmetry loss, we analyzed the performance on (average MSE over the test dataset) as we increased the number of infinitesimal generators that were used calculate Caym for the Heat equation. The two tables below show this effect for two different models that are trained with N, = 500 and N, 10000 unique points sampled from inside the grid, respectively. As it can be seen, including both of the useful symmetries of the Heat equation leads to the best performance compared to the models that are trained with or infinitesimal generators. Furthermore, as shown in previous results, we can see that the model's overall performance in all 3 cases increases
13

---

Gmunnd Thuh
Prediction tho symmetry!
Predction (symmetry)
1.000
1*18
10-00006
20 05 1.0
06
03
symmetry)
(a)
Figure The effect training PDE solver the Burgers' equation and without the symmetry loss for one of the PDEs in the test dataset. (a) shows the ground truth solution and the predictions of the two models as the number of samples inside the domain increases from 5000 to 25000 and 100000.(b) shows the corresponding predictions and the ground truth solution different time slices.
sample more points inside the grid and that the effect of the symmetry loss is more pronounced in the low-data regime. We also tried training models with and without symmetry loss for the
Table 5: The average test set mean mean-squared squared error for the Heat equation as function increasing the number of infinitesimal generators used to compute the symmetry loss. The MSE is reported for models trained with different numbers of unique points sampled from inside the grid. It can be seen that including both of the useful symmetries of the Heat equation leads the best performance compared the models that trained with 0 infinitesimal generators.
Number of Symmetries (K)
MSE when N, 500
MSE when N, 10000
0
0.73 0.38
0.26 0.21
1
0.61 + 0.33
0.25 + 0.17
2
0.44 + 0.26
0.20 + 0.12
Heat equation using modified MLP architecture, suggested in Wang al. [020b] We found that, unexpectedly, this architecture was performing worse than a simple MLP architecture in our experiments. However, it can be seen in Table 6 the model trained with the symmetry loss still performs better than the one trained with just the PINN loss.
Table The average test set mean -eun--quared error of unique points sampled inside the grid. The model architecture is modified MI., as suggested in Wang et al, [2020b]. It can be seen that the model trained with the symmetry loss performs betters than that trained without, especially in the low-data regime.
Heat equation as function increasing the number
Number of Points (N,)
No Symmetry
Symmetry
500
1.15 + 0.754
0.859 + 0.689
2000
0.974 + 0.667
0.520 + 0.488
10000
0.391 + 0.577
0.357 + 0.363
14


Reviews:
[{"review_id": "3x1QV9XZZg", "reviewer": "Reviewer_vAL8", "summary": "The paper proposes adding a custom loss function to training physics-informed neural networks (PINNs) to force symmetry requirements when learning solutions to PDEs. A given partial differential equation (PDE) is associated with a Lie group that acts on the space of solutions of the PDE that leaves this space invariant. The custom loss penalizes non-orthogonality between the gradient from the PDE and the symmetry constraint. This forces the updates to the solution to be on the level-set of group invariant solution. The loss is computed using the action of the Lie algebra generators on the jet space of the solutions.  \n\nThe authors show that the additional loss leads to better data efficiency and accuracy in two test case of  Heat Equation and Burgers’ Equation.", "questions": "- See my point on the power law before. I would plot different power laws based on different triplets of loss weights and show the coefficients.\n- Please compare to other baselines (including neural network operators and other PINNs) regarding numerical values. There are better ways to compare methods than eyeballing the graphs in Fig. 2 and Fig. 3. In particular because the errors might be very frame dependent. I would give a numerical error by averaging several runs via several seeds over several frames. In the current form, I am unable to fully appreciate the performance to the method. If this point is addressed correctly, I am open to increasing my mark.", "limitations": "Limitations have been well discussed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The idea of adding symmetry constraints to neural networks PDE solvers is a very interesting contribution to the field.\n- The execution is novel and makes clever use of Lie algebra theory on PDEs.\n- The exact form of the loss function is clearly motivated.\n- The experimental results are compelling and convincing. ", "weaknesses": "- The paper would benefit significantly from a more thorough ablation study.  Specifically, an exploration of the effects of incrementally adjusting the relative weighting of the different loss terms may offer crucial insights. It would be nice to analyse the data efficieny in a power low for example, as a function of the loss weights.\n- The discussion surrounding the two types of loss used in this study (cosine and orthogonality) is not fully clear. While the authors do mention these losses, there is insufficient detail regarding their selection,.\n- The mathematical notations of the paper are not clear. Overall, I've found the paper quite hard to parse. For example, the notations $\\text{pr}^{(n)}[\\Delta]$ is not really explained and I could only guess what it means. The Lie algebras are introduced suddently while not referred before. Please re write the introduction of projector using the Lie algebras.\n- There is an absence of any comparison with baseline models using data augmentation techniques or fully equivariant architecture in term of numerical results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback. Below we respond to the questions and concerns raised in the review. \n\n**R**:  The paper would benefit significantly from a more thorough ablation study. Specifically, an exploration of the effects of incrementally adjusting the relative weighting of the different loss terms may offer crucial insights. It would be nice to analyze the data efficiency in a power low, for example, as a function of the loss weights.\n\n**A**:  We agree with the reviewer that more experiments can always be more helpful. We will elaborate on the following empirical observation that suggests the proposed ablation on loss terms may not add much: We observe that for different problems using the same ratio for PINN and symmetry loss performs well, and further adjustment leads to little change in performance. In contrast, the initial condition matching loss is sensitive (as also seen in prior works on PINN), and therefore we treat it as a hyper-parameter. We believe the suggested ablation will reflect these findings, but we are open to running the experiments if the reviewer still finds it useful. \n\n**R**: There is an absence of any comparison with baseline models using data augmentation techniques or fully equivariant architecture in terms of numerical results.\n\n**A**: Since PINN models are not trained on any dataset, one cannot compare them to data augmentation. This is in contrast to neural operator methods, where data augmentation makes sense. The closest one could get to data augmentation is to add points to PINN loss corresponding to symmetry transformations. But since the value of the dependent variable is unobserved, it cannot be transformed, and the whole scheme simply involves increasing the points for the PINN loss. We do have such experiments. Additionally, to the best of our knowledge, there is no equivariant architecture for PINN model, in part due to the form of the action of the Lie point symmetries on the PDE.\n\n\n**R**: The mathematical notations of the paper are not clear. Overall, I've found the paper quite hard to parse. For example, the notation is not really explained, and I could only guess what it means. The Lie algebras are introduced suddently while not referred before. Please re write the introduction of projector using the Lie algebras.\n\n**A**: We will clarify the specific examples pointed out by the reviewer. The topic is mathematically challenging, and we use a significant portion of the background section introducing many ideas with a running example.  \n\n**R**: Please compare to other baselines (including neural network operators and other PINNs) \n\n**A**: Due to access to ground truth data during training, Neural operators generally perform better than PINNs. Since we are improving PINN using symmetry loss, the vanilla PINN model seems like the right baseline. Our objective here is to show that the PINN model at large can be improved through the use of symmetries. To further strengthen the point, we performed new experiments comparing the variant of [X] with and without symmetry. Here again, we see the positive effect of symmetry loss, although we do not see an improvement in general performance compared to vanilla MLP.\n\n**R**: regarding numerical values. [...] In particular because the errors might be very frame dependent. I would give a numerical error by averaging several runs via several seeds over several frames. [...] If this point is addressed correctly, I am open to increasing my mark.\n\n**A**: Results in Tables 1 and 2 are consistent with the reviewer’s suggestions. They are produced by averaging the error over multiple initial conditions, which is the main source of variance. The addition of multiple seeds for the same initial condition does not significantly change the results. Images are only provided for qualitative comparison."}, {"review_id": "HdYY60zOAk", "reviewer": "Reviewer_qo6e", "summary": "This paper proposes to enhance Physics-Informed Neural Networks (PINNs) by incorporating local Lie-point symmetry into them. It is achieved by introducing an additional symmetry loss term, which requires analytic computation of the PDE’s symmetries.\nThis loss term is designed to encourage orthogonality between the PDE equation and the different symmetry transformations.\nEvaluation is performed on 1D heat and Burgers equations.They show that the incorporation of symmetry gives better performance, especially when the number of sampled points is low.\nBeing unfamiliar with Lie theory, I found the theoretical to be quite challenging.\nOverall, I found the paper interesting, but a bit short on the experimental side.", "questions": "- How does the number of symmetry constraints scale wrt to the dimensionality of the considered PDE?\n- Could this symmetry loss term be incorporated to other PINN variants too?", "limitations": "- The experiments are only showcased on 1D PDEs", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The proposed approach is theoretically motivated, and leads to good performance compared to vanilla PINN\n- It doesn’t need many modifications to the vanilla PINN to implement", "weaknesses": "- The proposed method needs to compute the symmetries analytically first, before integrating them into the loss function. Would there be a way to automate this part?\n- In the considered examples, the better performance obtained with the additional loss terms could have also been achieved by denser sampling, as has been noted. What ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback! Below we respond to their questions and comments. \n\n**R**: The proposed method needs to compute the symmetries analytically first, before integrating them into the loss function. Would there be a way to automate this part?\n\n**A**: Yes, computational algebraic software can be used to calculate symmetry groups for a given PDE. Symbolic programs in MACSYMA, REDUCE, MAPLE, and Mathematica have been developed to find the determining equations for the infinitesimal generators. For example, the document (not ours) in the link below is a guide for one such package.\n(https://docs.google.com/viewer?url=https%3A%2F%2Flibrary.wolfram.com%2Finfocenter%2FID%2F4231%2FYaLie.ps%3Ffile_id%3D3408)\n\n**R**: In the considered examples, the better performance obtained with the additional loss terms could have also been achieved by denser sampling, as has been noted. What\n\n**A**: We agree that increasing sample size, similar to equivariance/symmetry, leads to better generalization. We show the effect of both symmetry and increased sample size for both heat and Burgers' equations. Please note that the question was incomplete.\n\n\n**R**: How does the number of symmetry constraints scale wrt to the dimensionality of the considered PDE?\n\n**A**: The number of symmetry constraints corresponds to the number of infinitesimal generators of the symmetry group of the PDE. While this number depends on the PDE equation itself, one generally expects that in higher dimensional PDE, the existence of certain symmetries (e.g., Euclidean symmetry) leads to an increased number of these infinitesimal generators.\n\n\n**R**: Could this symmetry loss term be incorporated to other PINN variants too?\n\n**A**: Yes, this loss term can be used in any PINN model. As an example, we have added an experiment for the Heat equation with a modified MLP model introduced in [1]. We also tried the causal training suggested in [2]. However, in the operator learning setting that we were trying (i.e. using DeepONet architecture to handle different initial conditions), we did not get improvement to the results as only the models trained with a small $\\epsilon$ in the PDE equation ( slope of temporal weights) led to reasonable predictions.\n\n\n\n[1] Sifan Wang, Yujun Teng, and Paris Perdikaris. Understanding and mitigating gradient pathologies in physics-informed neural networks, 2020b.\n\n[2] Sifan Wang, Shyam Sankaran, & Paris Perdikaris. (2022). Respecting causality is all you need for training physics-informed neural networks."}, {"review_id": "bRSgSYMwK8", "reviewer": "Reviewer_dief", "summary": "The work proposes a generic method to incorporate Lie point symmetry into physics-informed neural networks (PINNs) by augmenting loss function. The method leverages automatic differentiation as other PINNs do because the condition for symmetries is written using differentials. The authors demonstrated the model with symmetry gained better performance than one without symmetry.", "questions": "* How long do the training and prediction take, respectively?\n* What is the meaning of \"orthogonality\" in the paper, e.g. \"our symmetry loss encourages the orthogonality of $\\mathrm{pr}^{(n)}v$ and the gradient of $\\Delta$\" (line 237)? The condition contains PDE itself, so the reviewer guesses the use of the terminology could require some explanation.", "limitations": "The limitations are stated sufficiently in the paper.", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "* The work is based on an established mathematical theory found in Olver (1986).\n* Once the Lie point symmetries are obtained, it seems quite simple to incorporate them, thanks to automatic differentiation and the PINNs framework.", "weaknesses": "* The experiments look insufficient. They compared with and without symmetry but no other state-of-the-art PINN-based models. It makes the contribution of the work look limited because most of the mathematical contributions directly come from Olver (1986), and the results from the symmetry model are not good enough (in particular, Figure 2). They claimed that \"our goal is to showcase the effectiveness of using symmetries\" (lines 230-231). However, the positive effect of the symmetries is well-known to the community (e.g., Wang et al. 2021a).\n* The superiority of incorporating symmetries through loss function is unclear. I suggest the authors include data augmentation methods (e.g., Brandstetter et al. 2022a) and equivariant models (e.g., Wang et al. 2021a).\n* The authors claimed, \"Our work presents the foundations for leveraging Lie point symmetry in a large family of Neural PDE solvers that do not require access to accurate simulations\" (lines 316-318). However, another part says, \"The data for Burgers' equation is obtained using the Fourier Spectral method\" (line 299). It makes the reviewer confused about whether the method uses simulation data as supervisory signals.\n* The contribution of the paper is not clear enough. Is it a new theorem proved, an undiscovered observation found, a state-of-the-art performance, or a novel problem setting? The reviewer recommends the authors clearly state the contribution of the work and show the supporting facts.\n* The mathematical presentation lacks correctness and clarity. The reviewer found no clear definition of \"Lie point symmetry.\" The definition of $u^{(n)}$ lacks $u$ (see Olver (1986) p.97). It may be by mistake; however, this is an essential part of the theory. Thus the reviewer recommends the authors carefully re-check the manuscript for more mathematical correctness. For instance, if there is no $u$ in $u^{(n)}$, one cannot express the advection term of the Burgers' equation in the jet space.\n\nMinor points:\n* Please add an explanation about $N_s$.\n* The paper uses \"viscosity\" for the heat equation, which seems uncommon. It could be called the diffusion coefficient.\n* In the first equation of Equation (1), $t$ should be defined in an open set (see, e.g., Jürgen Jost \"Partial Differential Equations ThirdEdition\" (2012) Chapter 1). That's why we need the initial condition.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback! Below we respond to individual questions and comments. \n\n**R**: … It makes the contribution of the work look limited because most of the mathematical contributions directly come from Olver (1986), and the results from the symmetry model are not good enough (in particular, Figure 2). They claimed that \"our goal is to showcase the effectiveness of using symmetries\" (lines 230-231). However, the positive effect of the symmetries is well-known to the community (e.g., Wang et al. 2021a).\n\n**A**: While the theory of Lie point symmetry in PDE’s is an established area, our novel contribution is the methodology for incorporating this symmetry and, in particular, the design of the symmetry loss for PINNs. To our knowledge, our’s is the first work showing the effectiveness of using Lie point symmetries for PINN (Wang et al.22a, while quite relevant, is not concerned with PINNs).\n\n**R**: The superiority of incorporating symmetries through loss function is unclear. I suggest the authors include data augmentation methods (e.g., Brandstetter et al. 2022a) and equivariant models (e.g., Wang et al. 2021a).\n\n**A**: Data augmentation is not possible for PINN models as they are trained directly with the PDE equation – i.e., there is no ground truth training data in PINN to augment. Therefore, one could simply increase or “augment” the number of sampling points, and our experiments compare the effect of sample size and symmetry. To the best of our knowledge, there is no “equivariant model” for PINN. In general, we expect comparison to other neural solvers that are not PINN to significantly favour those solvers that rely on the underlying true PDE for learning. However, they also suffer from the same issue.\n\n**R**: The authors claimed, \"Our work presents the foundations for leveraging Lie point symmetry in a large family of Neural PDE solvers that do not require access to accurate simulations\" (lines 316-318). However, another part says, \"The data for Burgers' equation is obtained using the Fourier Spectral method\" (line 299). It makes the reviewer confused about whether the method uses simulation data as supervisory signals.\n\n**A**: Note that Tte data (solution) is generated only for evaluation purposes, not for training. For evaluation, we report the MSE with respect to the ground truth solution. We will make this point clear in the revision. \n\n**R**: The contribution of the paper is not clear enough. Is it a new theorem proved, an undiscovered observation found, a state-of-the-art performance, or a novel problem setting? The reviewer recommends the authors clearly state the contribution of the work and show the supporting facts.\n\n**A**: Main contributions are the following; we will make them explicit in the revision:\nMethodology to calculate the group action on the jet space of the PDE using automatic differentiation.\nEquation for a symmetry loss that enforces arbitrary Lie-point symmetry of the PDE in PINN models.\nDemonstrate the effectiveness of symmetry relative to increasing the number of sampling points in PINNs.\n\n**R**: Please add an explanation about N_s\n\n**A**: Sorry for this confusion in the notation. We will change N_s to N_0 to be consistent with the notation originally introduced. \n\n**R**: The paper uses \"viscosity\" for the heat equation, which seems uncommon. It could be called the diffusion coefficient.\n\n**A**: Yes, we will change this. \n\n**R**: In the first equation of Equation (1), t should be defined in an open set\n\n**A**: Thanks for pointing out this typo. It will be fixed in the revision. \n"}, {"review_id": "5DnOBzieS1", "reviewer": "Reviewer_5AKh", "summary": "In this paper, a method for finding solutions to differential equations that represent physical phenomena using neural networks is considered. In particular, a method that takes symmetry into account is proposed. Specifically, the authors consider an infinitesimal generator that represents the symmetry of the equation, and constrain the prolongation of the solutions to be orthogonal to the isosurface of the solution expressed in Jet space. This is expected to improve the accuracy of the solutions.", "questions": "(1) Can the computation of symmetry be automated by using computational algebraic software (e.g., Mathematica, Maple, Sigular)? \n(2) Although the improvement in numerical experiments is not significant for the cases considered in this paper, can it be effective for systems with many symmetries and conservation laws, such as integrable systems?\n(3) Perhaps I am misunderstanding something, but is it not sufficient to simply add some additional constraints? For example, does introducing the higher-order derivative of the loss function as additional cost functions (i.e. Sobolev learning) have a similar effect as the proposed method? I suppose that this can be used as an additional constraint because If the loss function is zero for all x and t, then the derivative of the loss function with respect to x and t should also be zero. In this way, it seems to me, it would be easy to create additional constraints without the laborious symmetry computations.", "limitations": "No potential negative societal impact is expected.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "As far as I know, the use of the symmetry of the model as an infinitesimal generator, rather than in the form of conservation laws, is certainly new. This enables the use of symmetry for equations that are not derived from the variational principle. The proposed method would be reliable in the sense that the method has a theoretical basis. In addition, the paper is clearly written.", "weaknesses": "The computation and implementation of symmetries and the constraints on the jet space based on them are considered to be quite difficult. On the other hand, the improvement confirmed by the numerical experimentation is not significant, so the effect is limited compared to the difficulty of implementation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful comments. Below we respond to their individual questions and comments.\n\n**R**: Can the computation of symmetry be automated by using computational algebraic software (e.g., Mathematica, Maple, Sigular)? \n**A**: Yes, computational algebraic software can be used to calculate symmetry groups for a given PDE. Symbolic programs in MACSYMA, REDUCE, MAPLE, and Mathematica have been developed to find the equations for the infinitesimal generators. For example, the document  (not ours) in the link below is a guide for one such package. We thank the referee for pointing out the fact that we had not mentioned this, and we will clarify the draft.\n(https://docs.google.com/viewer?url=https%3A%2F%2Flibrary.wolfram.com%2Finfocenter%2FID%2F4231%2FYaLie.ps%3Ffile_id%3D3408)\n\n**R**: Although the improvement in numerical experiments is not significant for the cases considered in this paper, can it be effective for systems with many symmetries and conservation laws, such as integrable systems?\n\n**A**: Interesting point! Our hypothesis is that symmetry loss will lead to more improvement for a system with many symmetries. We have added an experiment (see the attached PDF) where we track the improvement due to symmetry loss as we use more symmetries for the heat equation. This experiment shows how increasing the number of infinitesimal generators used in calculating the symmetry loss leads to improved performance. This hypothesis is also confirmed when comparing heat to Burgers’ since we can see that the greater number of symmetries for the heat equation leads to greater improvement in the results. \n\n**R**: Is it not sufficient to simply add some additional constraints (e.g. higher order derivatives of the loss)?\n\n**A**: Additional constraints on higher derivatives of the PINN loss are very different from constraints on derivatives produced through symmetry loss. The former is changing the PINN loss (rather than the solution) for example making the PINN loss smoother, and we’re not sure how it will affect the performance. However, the symmetry loss is explicitly encouraging that certain infinitesimal transformations of the solution found by the network remain solutions to the PDE. We will add a discussion of these points to the revision."}, {"review_id": "ggJAfFQyAB", "reviewer": "Reviewer_KJvK", "summary": "This study proposes a new loss function for PINNs that imposes the symmetry of PDE.\n", "questions": "With the same computational cost, is the proposed method superior to the vanilla PINN?\n\nIs the symmetry found in more practical and complicated PDEs? Then, is the proposed method useful?\n", "limitations": "The proposed method is limited to PDEs with known symmetry. The generality is unknown.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "PINN loss comes from the equation, but the proposed loss comes from the property of the equation. The idea is insightful.\n", "weaknesses": "The additional loss increases the computational cost. With the same computational budget, one can increase the number of evaluation points (N_r?) instead of introducing the proposed loss. The comparison might not be fair.\n\nThe proposed method was only evaluated with the heat equation and Burgers' equation, which are very simple PDEs. Is the symmetry found in more practical and complicated PDEs? Then, is the proposed method useful?\n\nThe symbols are not unified. In (7), N_l or N_r? Just after (13), N_l=300 is used for data-fit, but the number of points for data-fit is N_0 in (8).\n\nIn Table 1, \"1000\" might be wrong.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their useful feedback! Below we answer their questions and comments.\n\n**R**: The additional loss increases the computational cost. With the same computational budget, one can increase the number of evaluation points ($N_r$?) \n\n**A**: This is true. We have included experiments comparing the effect of the number of evaluation points to that of symmetry loss.\n\n**R**: The proposed method was only evaluated with the heat equation and Burgers' equation, which are very simple PDEs. Is the symmetry found in more practical and complicated PDEs? Then, is the proposed method useful?\n\n**A**: While symmetries are present in more complicated PDEs and generally, one may expect more symmetries in higher dimensions, we had a hard time adapting existing boundary-conditioned PINN models to produce a reasonable performance in these settings so that we can then improve their results using symmetry loss. To see the effect of these symmetry constraints on larger and more complex problems, we believe the symmetry loss should be combined with further innovations in PINNs that can enable its application to high-dimensional PDEs.\n\n**R**: The symbols are not unified. In (7), $N_l$ or $N_r$? Just after (13), $N_l=300$ is used for data-fit, but the number of points for data-fit is $N_0$ in (8).\n\n**A**: Thanks for pointing out the typos! In (7) it should be $N_r$. $N_l$ is used to refer to samples from both initial and boundary conditions ($N_l = N_0 + N_b$). We will make the notation more consistent and clear in the revision. \n\n**R**: In Table 1, \"1000\" might be wrong.\n\n**A**: Yes, thanks! It should be 10000. "}]

Author rebuttal:
We thank the reviewers for their insightful and constructive feedback! We are happy to see that they found the idea to be motivated (R-qo6e, R-vAL8), insightful (R-KJvK) and novel (R-vAL8, R-5AKh), the empirical results to be compelling (R-vAL8), and the presentations of the paper to be clear (R-5AKh). Below we address individual questions and concerns raised in the reviews. 

One common concern was in using alternative baselines. In the attached PDF, Table 4 presents new results comparing the effect of symmetry for different sample sizes using another architecture for PINN that uses a gating mechanism [1]. We do not observe any significant improvements over the vanilla MLP with or without symmetry loss. However, we do observe consistent improvement with the addition of symmetry loss to the PINN loss, especially in a low data regime. We also experimented with Causal PINN [2], but were did not see any improvements with or without symmetry loss over vanilla PINN loss. Another suggested baseline is data augmentation; Since PINN model is not trained on any dataset, we cannot perform data augmentation. One could simply increase the number of training points, and we do have experiments comparing the effect of symmetry loss with that of increasing data points (note that we cannot even “transform” the training points to perform augmentation since the symmetry also acts on the unknown dependent variable.) Yet another family of suggested baselines is Neural Operator-based methods. We expect Neural Operators to generally outperform PINNs, due to their advantage of training on exact solutions. Note that all other symmetry-based neural solvers cited by the reviewers (and in our paper), for example [3], are either Neural Operators or otherwise require exact solutions for training; therefore, they are not comparable to our PINN-based approach. We are unaware of any symmetry-based improvements for PINN. 
 
Another common question was regarding the effect of the number of PDE symmetries on the effectiveness of symmetry loss. Table 3 reports the result of a new ablation, in which we increase the number of symmetries of the same equation used in the symmetry loss. We observe consistent improvement as we incorporate more symmetries.  

Some reviewers have raised concerns about the technical difficulty of the material, as it affects readability. We appreciate this difficulty, and our solution to make the content palatable to ML community is to make the paper as self-contained as possible by providing a comprehensive (less than 3 pages) background on Lie point symmetries and using simple running examples and figures. However, some level of difficulty remains inevitable due to the technical nature of the topic. 

Finally, a common question is about the automatic derivation of symmetries from the PDE. This is indeed possible – computational algebraic software can calculate symmetry groups for a given PDE. Symbolic programs in MACSYMA, REDUCE, MAPLE, and Mathematica have been developed to find the equations for the infinitesimal generators.

We will clarify these points and include the new results in the revision. 


[1] Sifan Wang, Yujun Teng, and Paris Perdikaris. Understanding and mitigating gradient pathologies in physics-informed neural networks, 2020b.

[2] Sifan Wang, Shyam Sankaran, & Paris Perdikaris. (2022). Respecting causality is all you need for training physics-informed neural networks.

[3] Rui Wang, Robin Walters, and Rose Yu. Incorporating symmetry into deep dynamics models for improved generalization, 2021a.

Meta-review: During the initial phase of reviewing, the submission received mixed scores.  Some of the concerns raised related to the clarity of presentation, which have been addressed by the rebuttal. One of the referees comments on the novelty of the work as the symmetries are already known. However, the incorporation of the symmetries in the training of PINNs is a sufficiently novel contribution. There still remain concerns about limited experimental validation. The submission would indeed be strengthened significantly by taking into account the suggestions of the referees. However, there is sufficient merit in the submission to be accepted.
Decision: Accept (poster)