SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where  $\theta = (\theta_{1},\theta_{2})$ . In Algorithm 1, we use the notation  $u_{\theta}$  to denote the operator  $\mathcal{O}_{\theta}$  and use  $(\mathbf{x}l,\mathbf{u}^{l})_{1:N_{l}}$  to include both boundary and initial condition samples.
While we acknowledge recent architectural improvements to DeepONets [Krishnapriyan et al., 2021], since our goal is to showcase the effectiveness of symmetries, we deploy MLPs for both networks.
3.2 Imposing the Symmetry Criterion
To further inform PINNs about the symmetries of the PDE, we use an additional loss term  $\mathcal{L}_{\mathrm{sym}}$ . Conveniently, this loss sometimes also contains the PDE loss of Eq. (2).
Recall that the infinitesimal criterion of Theorem 2.2 requires that by acting on a solution  $(\mathbf{x},\mathbf{u}^n)$  in the jet space using the prolonged infinitesimal generator  $\mathrm{pr}^{(n)}\mathbf{v}$ , the PDE equation should remain satisfied. In simple terms, our symmetry loss encourages the orthogonality of  $\mathrm{pr}^{(n)}\mathbf{v}$  and the gradient of  $\Delta$  - in other words, infinitesimal symmetry transformations are encouraged to fall on the level-sets of  $\Delta$ , maintaining  $\Delta = 0$ . Next, we elaborate on this procedure.
Assume that the Lie algebra of the symmetry group of the  $n$ -th order PDE,  $\Delta$ , is spanned by  $K$  independent vector fields,  $\{\mathbf{v}_1,\dots ,\mathbf{v}_K\}$ , where each  $\mathbf{v}_k$  is defined as in Eq. (6). As noted earlier, for each  $\mathbf{v}_k$ , we can obtain their prolongations using automatic differentiation and create a vector of the corresponding coefficients:
$$
\operatorname {c o e f} \left(\Pr^ {(n)} \mathbf {v} _ {k}\right) = \left[ \xi_ {0} ^ {k}, \dots \xi_ {p} ^ {k}, \phi_ {0} ^ {k}, \dots , \phi_ {q} ^ {k}, \left(\phi_ {0} ^ {x _ {0}}\right) ^ {k}, \dots , \left(\phi_ {q} ^ {x ^ {1}}, \dots x ^ {p}\right) ^ {k} \right]. \tag {10}
$$
This is a vector of infinitesimal transformations in the jet space.
Using coef in the equation above is to differentiate the abstract definition of Eq. (6) and a vector of its coefficients.
We also use the notation  $J_{\Delta}$  for the gradient of  $\Delta$  wrt all independent and dependent variables:  $J_{\Delta} = \left(\frac{\partial \Delta}{\partial x^i}, \frac{\partial \Delta}{\partial \mathbf{u}_J^\alpha}\right)$ .
The symmetry loss encourages the orthogonality of each of  $K$  prolonged vector fields and the gradient vector above on points inside the domain:
$$
\mathcal {L} _ {\mathrm {s y m}} = \sum_ {k = 1} ^ {K} \left(J _ {\Delta} ^ {\top} \operatorname {c o e f} \left(\Pr^ {(n)} \mathbf {v} _ {k}\right)\right) ^ {2}. \tag {11}
$$
An alternative is to minimize the absolute value of cosine similarity. We found both of these to work well in practice.
Therefore, the total loss we use to train the two networks consists of the PINN loss introduced in Eq. (2) and the symmetry loss:  $\mathcal{L} = \alpha \mathcal{L}_{\mathrm{PDE}} + \beta \mathcal{L}_{\mathrm{data - fit}} + \gamma \mathcal{L}_{\mathrm{sym}}$ , where  $\alpha$ ,  $\beta$  and  $\gamma$  are hyperparameters. However, as we see through examples, one or more symmetries of a PDE often simplify to a constant times the  $\mathcal{L}_{PDE}$ , removing the need to separate treatment of the PDE loss. Algorithm 1 summarizes our training algorithm.
Algorithm 1 PINN with Lie Point Symmetry
inputs:  $\Delta$  : the PDE equation of order  $n$ ,  $(\mathbf{v}_k)_{1:K}$ : infinitesimal generators of symmetries of  $\Delta$ $\mathcal{D} = \left\{\left(\mathbf{x}_{1:N_r},(\mathbf{x},\mathbf{u})_{1:N_l},\mathbf{u}_{1:N_s}\right)\right\}_{1:N_f}$ : dataset for  $N_{f}$  different initial conditions  
init: initialize parameters  $\theta$  of network  $u_{\theta}$   
for iteration do  
Sample from  $\mathcal{D}$   
calculate  $\mathcal{L}_{\mathrm{sym}}$ :  
calculate  $\mathbf{u}^{(n)}$  using automatic differentiation for  $\mathbf{x}_{1:N_r}$   
calculate  $\operatorname{coef}(\operatorname{pr}^{(n)}\mathbf{v}_k)$ , using  $\mathbf{x}_{1:N_r}$ ,  $(\mathbf{u}^{(n)})_{1:N_r}$ , auto diff and Eq. (7)  
calculate  $\mathcal{L}_{\mathrm{sym}}$  using  $\operatorname{coef}(\operatorname{pr}^{(n)}\mathbf{v}_k)$  and  $\Delta$  and equation Eq. (11)  
calculate  $\mathcal{L}_{\mathrm{PDE}}$ :  
using  $\Delta$ ,  $(\mathbf{u}^{(n)})_{1:N_r}$  and equation Eq. (3):  
calculate  $\mathcal{L}_{\mathrm{data - fit}}$ :  
calculate  $\hat{\mathbf{u}} = \mathbf{u}_{\theta}(\mathbf{x},\mathbf{u}_{1:N_s})$  for  $\mathbf{x}_{1:N_l}$   
calculate  $\mathcal{L}_{\mathrm{data - fit}}$  using  $\mathbf{u}_{1:N_l}$ ,  $\hat{\mathbf{u}}_{1:N_l}$  and equation Eq. (4)  
 $\mathcal{L} = \alpha \mathcal{L}_{\mathrm{PDE}} + \beta \mathcal{L}_{\mathrm{data - fit}} + \gamma \mathcal{L}_{\mathrm{sym}}$ $\theta \gets \theta - \nabla_{\theta} \mathcal{L}$   
end for  
return  $u_{\theta}$
4.1 Heat Equation
First, we study the effectiveness of imposing the symmetry constraint on the heat equation, described in Eq. (5). This equation is a simple linear PDE with a rich symmetry group.
Symmetries. The following 6-dimensional lie algebra spans the symmetry group of the heat equation:
$$
\mathbf {v} _ {1} = \partial_ {x} \quad \mathbf {v} _ {3} = \partial_ {u} \quad \mathbf {v} _ {5} = 2 \nu t \partial_ {x} - x u \partial_ {u} \tag {12}
$$
$$
\mathbf {v} _ {2} = \partial_ {t} \quad \mathbf {v} _ {4} = x \partial_ {x} + 2 t \partial_ {t} \quad \mathbf {v} _ {6} = 4 \nu t x \partial_ {x} - 4 \nu t ^ {2} \partial_ {t} - (x ^ {2} + 2 \nu t) u \partial_ {u}.
$$
For example, the infinitesimal generator  $v_{1}$  corresponds to space translation, and  $v_{5}$  to Galilean boost. We refer the reader to Olver [1986] for more details on the derivation.
Data. We generate simulated solutions, which we use to test the models' performance. We use  $\Omega = [0,L] = [0,2\pi]$ , discretized uniformly into 256 points and assume periodic spatial boundaries. We also use  $[0,T] = [0,16]$ , discretized into 100 points. The viscosity coefficient is set to  $\nu = 0.01$ .
Figure 2: The effect of training the PDE solver for the heat equation with and without the symmetry loss for one of the PDEs in the test dataset. (a) shows the ground truth solution and the predictions of the two models as the number of samples inside the domain increases from 500 to 2000 and 1000.(b) shows the corresponding predictions and the ground truth solution at different time slices.
Similar to Brandstetter et al. [2022b,a] and Bar-Sinai et al. [2019], we represent the initial condition functions by truncated Fourier series with coefficients  $A_{k},l_{k},\phi_{k}$  sampled randomly, and  $K = 10$  ..
$$
u (t = 0, x) = f (x) = \sum_ {i = 1} ^ {K} A _ {k} \sin \left(2 \pi l _ {k} x / L + \phi_ {k}\right). \tag {13}
$$
These functions are sampled at  $N_{s} = 200$  fixed points, which are used as input to  $e_{\theta_1}$  in Eq. (9). We also sample a total of  $N_{l} = 300$  points (including the 200 points sampled at  $t = 0$ ), used to impose the data-fit loss,  $\mathcal{L}_{\mathrm{data - fit}}$ . We also randomly sample 100 and 300 of these initial conditions and use them for the validation and test datasets respectively.
Training and Experiments. The main objective of our experiments is to confirm the hypothesis that training with symmetry loss helps improve the model's prediction capability in a low-data regime. Therefore, we train the model with and without symmetry and evaluate the model's predictions on the test dataset as we increase the number of samples inside the domain,  $N_r$ . To illustrate the effectiveness of the symmetries in a low-data regime, we use  $N_f = 100$  different initial conditions and test the performance as we increase  $N_r$  from 500 to 2000 and 10000. We refer to Appendix C for details on the architectures and hyperparameters.
Results. In Table 1, we can see a comparison between the performance of the two models on the test dataset of unobserved initial conditions as  $N_{r}$  increases. We note that when trained with few samples, the model trained with symmetry loss,  $\mathcal{L}_{\mathrm{sym}}$  performs significantly better than the baseline model. Fig. 2, also illustrates this point as it shows the performance of both models on a single instance from the test dataset. We note that the improvement in prediction results in the model where symmetry loss is enforced is especially significant at larger values of time,  $t$ .
We want to highlight an important detail: by using the infinitesimal criterion for enforcing symmetries, not all symmetries of the PDE will help improve the training. There are instances when the gradient of  $\Delta$  along the vector field is trivially zero, and in other instances, 'we obtain  $c\Delta$  where  $c$  is a constant. In the case of the heat equation, only the symmetry transformations from vector fields  $v_{5}$  and  $v_{6}$  provide useful training signals. This means that in our experiments, we can simply eliminate the PDE loss,  $\mathcal{L}_{\mathrm{PDE}}$ , and only use the symmetry loss,  $\mathcal{L}_{\mathrm{sym}}$ , in addition to the supervised loss.
Table 1: The average test set mean-squared error for the Heat equation.
Number of Points (Nr)	No Symmetry	Symmetry
500	1.12 ± 0.58	0.30 ± 0.15
2000	0.36 ± 0.19	0.24 ± 0.14
10000	0.22 ± 0.14	0.21 ± 0.13
Table 2: The average test set mean-squared error for Burgers' equation.
Number of Points (Nr)	No Symmetry	Symmetry
5000	0.041 ± 0.042	0.034 ± 0.039
25000	0.030 ± 0.038	0.017 ± 0.020
100000	0.018 ± 0.022	0.013 ± 0.020
Figure 3: (a) predictions of the network trained with a total of 100000 points inside the domain (500 initial conditions and only 200 points for each PDE), with and without the symmetry loss, for Burgers' equation on a test data. (b) corresponding predictions at different time slices showing that the model trained with symmetry loss performs better, especially at larger  $t$ .
4.2 Burgers' Equation
The second PDE we analyze is Burgers' Eq. (14), which combines diffusion (with thermal diffusivity  $\nu$ ) and non-linear advection (wave motion). The nonlinearity of this equation makes it more complex, resulting in shock formation.
$$
u _ {t} = \nu u _ {x x} - u u _ {x} \tag {14}
$$
Symmetries. Burgers' equation in the form described in Eq. (14) has a symmetry group spanned by the following 5-dimensional vector space.
$$
\mathbf {v} _ {1} = \partial_ {x} \quad \mathbf {v} _ {3} = t \partial_ {x} + \partial_ {u} \quad \mathbf {v} _ {5} = t x \partial_ {x} + t ^ {2} \partial_ {t} - (x - t u) \partial_ {u} \tag {15}
$$
$$
\mathbf {v} _ {2} = \partial_ {t} \quad \mathbf {v} _ {4} = x \partial_ {x} + 2 t \partial t - u \partial_ {u}
$$
However, only the last generator  $\mathbf{v}_5$  results in a useful training signal. The first three generators give  $\mathcal{L}_{\mathrm{sym}} = 0$  and  $\mathbf{v}_4$  gives  $\mathcal{L}_{\mathrm{sym}} = c\Delta = c\mathcal{L}_{\mathrm{PDE}}$ , for a constant  $c$ . As with the heat equation experiment, we can eliminate the PDE loss and only use symmetry and supervised losses for training.
Data. The data used to evaluate the model is obtained using the Fourier Spectral method with periodic spatial boundaries. Initial conditions are obtained similarly to the heat equation experiment, described in Eq. (13). We use  $\nu = 0.1$  as the diffusion coefficient. The domain is  $[0,L] = [0,2\pi]$  and  $[0,T] = [0,2.475]$  discretized uniformly into 256 and 100 points respectively.
Training and Experiments. For Burgers' equation, we train the model on datasets of  $N_{f} = 500$  initial conditions and  $N_{r} = 5000$ , 25000 and 100000 samples. We found that for  $\mathcal{L}_{\mathrm{sym}}$ , cosine similarity works better in this case. The models' architectures are similar to those used for the heat equation described in Appendix C.
Results. Table 2 shows the average mean-squared errors on the test dataset for the two models as  $N_r$  increases. We can see that, even with one symmetry group useful for training, the model trained with  $\mathcal{L}_{\mathrm{sym}}$  performs better. The predictions on a single instance of the test dataset can also be seen in Section 4.2. Again, we see that the symmetry loss especially improves the model's performance for larger values of  $t$ . See Appendix B for additional plots of the prediction results. We also note that the high standard deviations in Table 2 are because, compared to the heat equation, the behaviour of the solution (specifically shock formation) varies a lot based on the initial conditions.
Conclusion and Limitations
Lie groups and continuous symmetries are historically rooted in the study of differential equations, yet to this day, their application to Neural PDE solvers has been limited. Our work presents the
foundations for leveraging Lie point symmetry in a large family of Neural PDE solvers that do not require access to accurate simulations. Using available machinery of automatic differentiation, we show that local symmetry constraints can improve PDE solutions found using PINN models.
The method we propose to leverage local symmetries has some limitations: 1) while the Lie point symmetries of important PDEs are well-known, in general, one needs to analytically derive them for a given PDE to use our approach; 2) as we mentioned in Section 4, not all symmetries of the equation will necessarily be useful for constraining the PINN. Fortunately, the usefulness of symmetries is obvious from the corresponding infinitesimal criterion, and one could limit the symmetry loss to useful symmetries; 3) while symmetries can significantly improve performance, based on our empirical observations (see Table 1), one could achieve a similar effect with PINN by increasing the sample size. These limitations motivate our future direction, which builds on our current understanding, to impose symmetry constraints through equivariant architectures.
The authors would like to thank Prakash Panangaden for insightful discussions on symmetries and Lie Group theory. We also thank Oumar Kaba for his helpful feedback on this manuscript. This research is in part supported by Canada CIFAR AI Chair and Microsoft Research. Computational resources are provided by Mila and Digital Research Alliance of Canada.
Yohai Bar-Sinai, Stephan Hoyer, Jason Hickey, and Michael P. Brenner. Learning data-driven discretizations for partial differential equations. Proceedings of the National Academy of Sciences, 116(31):15344-15349, 2019. doi: 10.1073/pnas.1814058116.
Kaifeng Bi, Lingxi Xie, Hengheng Zhang, Xin Chen, Xiaotao Gu, and Qi Tian. Pangu-weather: A 3d high-resolution model for fast and accurate global weather forecast. arXiv preprint arXiv:2211.02556, 2022.
Johannes Brandstetter, Max Welling, and Daniel E. Worrall. Lie point symmetry data augmentation for neural PDE solvers. In Proceedings of the 39th International Conference on Machine Learning, volume 162, pages 2241-2256. PMLR, 2022a.
Johannes Brandstetter, Daniel E. Worrall, and Max Welling. Message passing neural PDE solvers. In The 10th International Conference on Learning Representations. OpenReview.net, 2022b.
Jayesh K Gupta and Johannes Brandstetter. Towards multi-spatiotemporal-scale generalized pde modeling. arXiv preprint arXiv:2209.15616, 2022.
K. Kashinath, M. Mustafa, A. Albert, J-L. Wu, C. Jiang, S. Esmaeilzadeh, K. Azizzadenesheli, R. Wang, A. Chattopadhyay, A. Singh, A. Manepalli, D. Chirila, R. Yu, R. Walters, B. White, H. Xiao, H. A. Tchelepi, P. Marcus, A. Anandkumar, P. Hassanzadeh, and Prabhat. Physics-informed machine learning: case studies for weather and climate modelling. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 379(2194): 20200093, February 2021. doi: 10.1098/rsta.2020.0093.
Dmitrii Kochkov, Jamie A. Smith, Ayya Alieva, Qing Wang, Michael P. Brenner, and Stephan Hoyer. Machine learning - accelerated computational fluid dynamics. Proceedings of the National Academy of Sciences, 118(21), 2021. doi: 10.1073/pnas.2101784118.
Aditi S. Krishnapriyan, Amir Gholami, Shandian Zhe, Robert M. Kirby, and Michael W. Mahoney. Characterizing possible failure modes in physics-informed neural networks. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, pages 26548-26560, 2021.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. In Proceedings of the 9th International Conference on Learning Representations, 2021a.
Zongyi Li, Hongkai Zheng, Nikola B. Kovachki, David Jin, Haoxuan Chen, Burigede Liu, Kamyar Azizzadenesheli, and Anima Anandkumar. Physics-informed neural operator for learning partial differential equations. CoRR, abs/2111.03794, 2021b. URL https://arxiv.org/abs/2111.03794.
Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nature Machine Intelligence, 3:218-229, 2021.
Grégoire Mialon, Quentin Garrido, Hannah Lawrence, Danyal Rehman, Yann LeCun, and Bobak T Kiani. Self-supervised learning with lie symmetries for partial differential equations. arXiv preprint arXiv:2307.05432, 2023.
Tung Nguyen, Johannes Brandstetter, Ashish Kapoor, Jayesh K Gupta, and Aditya Grover. Climax: A foundation model for weather and climate. arXiv preprint arXiv:2301.10343, 2023.
Peter J. Olver. Applications of Lie Groups to Differential Equations. Springer, 1986.
Alfio Quarteroni. Numerical Models for Differential Problems, volume 2. Springer, 2009.
Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686-707, 2019.
Rui Wang, Karthik Kashinath, Mustafa Mustafa, Adrian Albert, and Rose Yu. Towards physics-informed deep learning for turbulent flow prediction, 2020a.
Sifan Wang, Yujun Teng, and Paris Perdikaris. Understanding and mitigating gradient pathologies in physics-informed neural networks, 2020b.
Sifan Wang, Hanwen Wang, and Paris Perdikaris. Learning the solution operator of parametric partial differential equations with physics-informed deeponets. Science Advances, 7(40), 2021.
Sifan Wang, Shyam Sankaran, and Paris Perdikaris. Respecting causality is all you need for training physics-informed neural networks, 2022.
Sifan Wang, Shyam Sankaran, Hanwen Wang, and Paris Perdikaris. An expert's guide to training physics-informed neural networks. arXiv preprint arXiv:2308.08468, 2023.
A Illustrative Examples
Example 3 (Obtaining the Prolongation of  $SO(2)$ ). We can consider If  $X \times U = \mathbb{R} \times \mathbb{R}$  and the infinitesimal generator of the 2-dimensional rotation group,  $SO(2)$ :
$$
\begin{array}{l} \mathbf {v} _ {S O (2)} = \xi (x, u) \partial_ {x} + \phi (x, u) \partial_ {u} \\ = - u \partial_ {x} + x \partial_ {u} \\ \end{array}
$$
In this 2-dimensional case, the calculation of the prolonged generator is simple:
$$
\begin{array}{l} \phi^ {(x)} = D _ {x} (\phi - \xi u _ {x}) + \xi u _ {x x} \\ = D _ {x} \left(x + u u _ {x}\right) - u u _ {x x} \\ = \left(1 + u _ {x} ^ {2} + u u _ {x x}\right) - u u _ {x x} \\ = 1 + u _ {x} ^ {2} \\ \end{array}
$$
$$
\operatorname * {p r} ^ {(1)} \mathbf {v} _ {S O (2)} = - u \partial_ {x} + x \partial_ {u} + (1 + u _ {x} ^ {2}) \partial_ {u _ {x}}
$$
We will work through another example of obtaining the prolongation of an infinitesimal generator of the heat equation:
Example 4 (Obtaining the Prolongation of an Infinitesimal Generator). As an example, we will consider  $X \times U = \mathbb{R}^2 \times \mathbb{R}$  and the following infinitesimal generator, which is a symmetry of the heat equation:
$$
\begin{array}{l} \mathbf {v} = \xi_ {1} (x, t, u) \partial_ {x} + \xi_ {2} (x, t, u) \partial_ {t} + \phi (x, t, u) \partial_ {u} \\ = 2 \nu t \partial_ {x} - x u \partial_ {u} \\ \end{array}
$$
where  $x, t$  denote the independent variables,  $u$  is the dependent variable and  $\nu$  is a positive constant. By the prolongation formula, Eq. (7), the first prolongation in  $t$  is given by:
$$
\begin{array}{l} \phi^ {t} = D _ {t} \left(\phi - \xi_ {1} u _ {x} - \xi_ {2} u _ {t}\right) + \xi_ {1} u _ {x t} + \xi_ {2} u _ {t t} \\ = D _ {t} (- x u - 2 \nu t u _ {x}) + 2 \nu t u _ {x t} \\ = - x u _ {t} - 2 \nu u _ {x} \\ \end{array}
$$
Figure 4: Various solutions of the PDE  $\Delta (x,u,u_x) = (u - x)u_x + u + x = 0$  obtained via symmetry transformation (rotation) of a know solution (in red).
As a final illustrative example of the symmetry criterion, we will follow Olver's example below:
Example 5. As an illustrative example of the infinitesimal criterion, we can consider a simple DE:
$$
\Delta (x, u, u _ {x}) = (u - x) u _ {x} + u + x = 0
$$
We can easily see that  $SO(2)$  is a symmetry group of this differential equation, using the prolongation of the generator we calculated in Example 3:
$$
\begin{array}{l} \operatorname {p r} ^ {(1)} \mathbf {v} [ \Delta ] = - u \Delta_ {x} + x \Delta_ {u} + (1 + u _ {x} ^ {2}) \Delta_ {u _ {x}} \\ = - u \left(1 - u _ {x}\right) + x \left(1 + u _ {x}\right) + \left(1 + u _ {x} ^ {2}\right) (u - x) \\ = u _ {x} \Delta \\ \end{array}
$$
Since  $\Delta u_{x} = 0$  when  $\Delta = 0$ , we can conclude that  $SO(2)$  is indeed a symmetry group of the equation. In fact, we can see that it transforms solutions of this differential equation to other solutions in Fig. 4.
B Implementation and Training Details
We model the two networks,  $g_{\theta_1}$  and  $e_{\theta_2}$  in Eq. (9) with MLPs consisting of 7 hidden layers of width 100. This choice was based on the previous research using PINN and DeepONets for solving Burgers' equation [Wang et al., 2021]. We used elu activation as differentiable activations are required for the PDE loss. The output of the embedding vectors from both networks is 100 dimensional. We used ADAM optimizer with learning rate of 0.001 for the training and performed early stopping using the validation dataset.
For both the Heat equation and Burgers' equation experiments, we perform hyper-parameter tuning on the coefficients of the loss terms from the set  $[0.1,\dots ,1,\dots ,10,\dots ,100,\dots 200]$ . This is done separately for the baseline model and the model trained with symmetry loss,  $\mathcal{L}_{\mathrm{sym}}$ , as we varied the number of samples,  $N_{r}$ . We found that similar to PINNs, the model is sensitive to the weight given to the supervised loss for the initial conditions vs the symmetry/PINN loss. The specific coefficient values for the models trained with and without symmetry loss for the heat equation are in Table 3 and Table 4.
Table 3: The loss coefficients for the model trained with the symmetry loss (i.e.  $\mathcal{L} = \beta \mathcal{L}_{\mathrm{sym}} + \gamma \mathcal{L}_{\mathrm{data - fit}}$  for the heat equation for various number of unique points sampled inside the grid for training,  $N_{r}$
loss coefficient	Nr=500	Nr=2000	Nr=10000
β	100	80	40
γ	20	20	20
Table 4: he loss coefficients for the model trained without the symmetry loss (i.e.  $\mathcal{L} = \alpha \mathcal{L}_{\mathrm{PINN}} + \gamma \mathcal{L}_{\mathrm{data - fit}}$  for the heat equation for various number of unique points sampled inside the grid for training,  $N_{r}$
loss coefficient	Nr=500	Nr=2000	Nr=10000
α	150	150	130
γ	20	20	20
We also note that for Burgers' equation, we found that cosine similarity for  $\mathcal{L}_{\mathrm{sym}}$  works better than the dot product. The results reported in Section 4 use cosine-similarity.
We will make the data and the code available on GitHub.
In the figure below, we can see the behaviour of the two models, trained with and without symmetry loss for Burgers' equation, as we increase the number of training samples. It can be seen that, as expected, in the model trained with  $\mathcal{L}_{\mathrm{sym}}$  performs significantly better with low samples inside the domain. The corresponding mean-squared errors are reported in Table 2.