Self-Paced Learning  Enhanced Physics-informed Neural Networks for Solving Partial Differential Equations
Authors: Xiaoting Han
Venue: Submitted to ICLR 2023 (2023)

Abstract: There is a hit discussion on solving partial differential equation by neural network. The famous PINN (physics-informed neural networks) has drawn worldwide attention since it was put forward. Despite its success in solving nonlinear partial differential equation, the difficulty in converging and the inefficiency in training process are definitely huge concerns. Normally, data for PINN is randomly chosen for a given distribution. Additionally, it's fitted to a model in a meaningless way. Curriculum Learning is a learning strategy that trains a model from easy samples to hard ones, which represents the meaningful human learning order. Self-paced Learning (SPL) is one of the significant branches of Automatic Curriculum Learning, which takes example-wise the training loss as Difficulty Measurer. SPL is an efficient strategy in enhancing the convergence rate of numerous models. In this paper, we propose a novel SPL-PINN learning framework, with SPL to accelerate the convergence progress of PINN. We demonstrate the effectiveness of SPL-PINN in a typical parabolic equation and Burgers equation. 

Under review as a conference paper at ICLR 2023
SELF-PACED LEARNING ENHANCED PHYSICS- INFORMED NEURAL NETWORKS FOR SOLVING PAR- TIAL DIFFERENTIAL EQUATIONS
Anonymous authors Paper under doublc-blind review
ABSTRACT
There is hit discussion on solving partial differential equation by neural network. The famous PINN (physics-informag informed neural networks) has drawn worldwide at- tention since was put forward. Despite its success solving nonlinear partial differential equation, the difficulty in converging and the inefficiency in training process are definitely huge concerns. Normally, data for PINN is randomly chosen for given distribution. Additionally, it's fitted to model in meaningless way. Curriculum Learning is a learning strategy that trains a model from easy samples hard ones, which represents the meaningful human learning order. Self-paced Learning (SPL) is one of the significant branches Automatic Curriculum Learn- ing, which takes example-wise the training loss Difficulty Measurer. SPL. is an efficient strategy in enhancing the convergence rate of numerous models. this paper, propose novel SPL-PINN learning framework, with SPL celerate the convergence progress of PINN. We demonstrate the effectiveness of SPL-PINN in a typical parabolic equation and Burgers equation.
INTRODUCTION
There is an increasing number of methods for solving partial differential equation. Weinan Yu (0188) proposed a deep learning- based method, the Deep Rize Method, for solving variational prob- lems, which are derived form partial differential equations. [Sirignano & Spiliopoulos/ (2018) pro- posed a mesh-free method, the Deep Galerkin Method, to approximate the solution of partial differential equation by a neural network instead of linear combination of bias functions. (RAT, 2019) aimed to approximate the solution by neural network well as the inner physical informa- tion. There is much profound research related to PINN, the research topics can be filed to three specific problem, the parallel PINN and the acceleration of
fields: the application of PINN PINN. Papers related to PINN abound, suggesting that a great burst of creativity has occured since the emergence of PINN. In this paper, concentrate
how to accelerate the converging the training process. Xiang et al. (2021) proposed a self-adaptive loss function which automatically assigns the weights of different objectives. LIU (2021 is also proposed automatically adjust the weights of different objectives. TThanasutives et al. (202T) treated it as a multi-task learning prob- lem, they proposed adversarial learning generate high-loss samples, similarly distributed the original training distribution. Kim et al. (2020) proposed dynamic pulling method, enabling PINN learn dynamics of the governing equations. Yuu et al.) (2022) assumed that the solution of the given partial differential equation is smooth, so they took the gradient of the PDE residual as part of the loss function Chiu et al. (202I) proposed a CAN-PINN framework, taking the advantage of automatic differentiation and numerical differentiation. increase the accuracy of PINN. Bischof & Kraus (202I) treated it as a multi-objective problem, proposed a novel self-adaptive loss balancing of PINN with random lookback. Generally, the methods aiming at the inefficiency in training pro- cess are treating the weights of different losses. In this paper, we concentrate on the training pattern, fit model form easy samples to hard samples.
in

---

Under review as a conference paper at ICLR 2023
2
RELATED WORK
2.1 PHYSICS-INFORMED NEURAL NETWORK
Physics-informed neural network 2019) successful mesh-free method for solving partial differential equations. Let's start by focusing on general partial differential equation with both initial-value condition and boundary-value condition.
0,u(t,z) (t, = f(t,z) ef (0,T)
u(0, x) = g(x) x xEΩ
(I)
x(),x)= = a((,a) (z, E x (0,7)
Here u(z, t) is the solution of this initial/boundary problem. L is the differential operator the variable r. Ω a subset R PINN approximates u(x, t) by neural network with the physical information instead of traditional finite differential method and finite element method. The loss function can be divided into several parts.
L= L +I LB
(2)
where
LF
Unet (tf xx{) + Unee x.. 2
RF
1.
Unnce (0, x1) - g(a$1)²
(3)
11
j=1
LB
Unet (th,, xxx - htt xx)
In PINN, (LF .off') and =L) randomly sampled from a given distribution. Specially, u(,,z) is derived from a back-propagation neural network by minimizing the proposed loss func-
tion 2
0 argmin L 0
(4)
2.2 SELF-PACED LEARNING
Self-paced Learning(Kunnn Kumar et al. 2010 is one of the significant branches of Automatic Curriculum Learning(Bengee al.] 2009), which devotes readily computable example-wise the training loss as Difficulty Measurer, alleviating the lack of how to efficiently identify easy samples.
Let D y. 1 be the training dataset where x; A' represents the 7th input variable and y; ) represents the correspondingly ith output variable. Given the training data, the parameters 0 of special model are trained by minimizing the defined loss function L. Traditionally, the parameters trained as follows:
min
L L(x, y;; R(0))
(5)
where R(-) a Regularizer. SPL introduces SPL-Regularizer to the traditional learning pattern, thus parameters are trained in a meaningful way.
N us U,L, (2: + g(v;, A)
0 = min
(6)
0
2

---

Under review as a conference paper at ICLR 2023
The value of each is in the interval [0, 1]. À is barrier which determines the number of easy samples that are selected at each training epoch. There are numerous types of Se-Reuularieer and the corresponding close-formed solutions 2" (L, A) such Hard-SP- RequulaeR,,,,,,,zerr Linear- SP-Reqularizer, Segriihnii-SP-RReuaaiiz- Logistic-SP SP- uP-Reuulrr to illustrate the efficiency of our method in section
Sesuuree-S--Rqulaiiiin Dogistic-S-Requlaaae Sequllrarr,,rrr, -atpnomial-SP-Reuaarriiii Wang al. (2022) We choose to Hard-
3
THE SPL-PINN FRAMEWORK
While PINN has shown its remarkably success in solving a variety of forward and inverse partial differential equations, it ignoredd the learning efficiency of the model itself, which resulting in great demand for an effective method to solve this defect. Enormous research output was devoted to this topic, mainly motivated by the different weights of losses. Admittedly, the performance and the weights of losses are inextricably interconnected. However, we try to approach the problem from a different viewpoint. Strategies for the difficulty in converging are an important component of PINN, to satisfy the demand, we propose the SPL-PINN framework, which can effectively alleviate the inefficiency of convergence. Given the training data, instead of pull into PINN an meaningless way, we choose to fit the model from easy one to hard one, which is eminently suitable for human learning process.
3.1 SPL-PINN LOSS FUNCTION
In order accelerate the learning process, determinate the loss function with SPL-Regularizer Samples whether are easy or not are determined by a scalar X. In order to concisely exploit the SPL-PINN structure, we choose use the Hard-SP-Requland the portion of easy samples that are plugged into the model.
and its corresponding st control
Loriginat LF L1 + LB
L
U₁ LzLiiignaat (2 0) g(1;,
N
g(u;, A) -
Σ
4<>
(15 0.
otherwise
The specific details are as follows:
U; means that the tt sample is identified as the easy one, which should be used train the model.
V; = means that the tth sample is identified as the hard one, which should not be used to train the model.
A is threshold, the larger means more samples will be selected.
3.2 OVERVIEW OF SPL-PINN
Normally, the dataset is constructed by randomly sampled from a given distribution. Generally, researchers regard all of the samples have equal contribution to their models. Considering that our deep learning models are finally being used to make effective predictions that solve partial differential equations, especially those equations have physical information, it seams meaningless to some degree to just treat samples equally.
In consideration of human learning habit. which usually starts from simple and general ones then gradually transmits to more complex and specialized ones. For instance, college students normally learn mathematical analysis and algebra the first semester, later they available access real
3

---

Under review as a conference paper at ICLR 2023
and complex analysis, functional analysis. Given the hypothesis that different samples play different role model training process. Motivated by the habit of human learning behavior, we propose a novel SPL-PINN framework, which seamlessly integrates PINN and Self-paced Learning.
We consider the partial differential equation with both inital- value condition and boundary-valuee condition that proposed in section 2.1 For the 7th sample, firstly compute its loss function 4.- Secondly, identifying whether is an easy sample by 2i (4, 2). Subdataset is constructed by selceted samples. Thirdly, model parameters 0 are renewed by minimizing the loss function L. Finally, after converging in this subdataset, update the threshold X. The complete procedures are as follows:
Algorithm SPL-PINN
Input: Input dataset [(1 xf ) { and ((R )=))== pace parameter o
Output: Model parameter 0
I: Initialize 0, A;
2: while not converged do
3: while not converged do
4: Compute of (4, A)
5: Update argmin L
6: end while
7:
<-
8:
end while
4 EXPERIMENTS
To illustrate the efficiency of our method in the converging process, we apply SPL-PINN to a typical parabolic equation and Burgers equation. The dataset is constructed by randomly sampled from the uniform distribution.
4.1 HEAT EQUATION
The heat equation, also known as diffusion equation, is of typical parabolic equations. It is widely studied topic and foundational partial diffusion equation, which describes the evolution in time of the density of some quantity such heat.
Ou Pu
=0
(x, E (0,1) x (0,1)
Ōt
Or
(8)
u(I, 0) sin (xx)
x E (0,1)
u(0, t) u t) = 0 EE (0,1)
The problem stated above has its unique solution:
u(x, 1) = sin(m
We choose the training dataset {(t x {)) {x and {(t fr))) which ny n fg 2000. For fixed iteration, the profermance of SPL-PINN is better than traditional PINN. For given training epoch, we compute mean squared error of u(x, t) in test dataset. Mean squared emror of u(æ, !) of given epochs are represented in Table I Moreover, the solution of and the train losses PINN and SPL-PINN can found in hp We use to measure the residual, = LSPL-PINN LPINN. Typically, 8 is almost everywhere negative, which is compelling illustration of the efficiency of our method.
4.2 BURGERS EQUATION
Burgers equation expresses the propagation and reflection of shock wave. Naturally, it plays significant role in many applications such as gas dynamics, fluid mechanics, nonlinear acoustics and on. The equations are as follows:
4

---

Under review as a conference paper at ICLR 2023
Table 1: Mean squared error of u(x; t)
epochs
PINN
SPL-PIIN
10
0.2016
0.1912
100
0.1322
0.1077
200
0.0421
0.0406
300
0.0388
0.0167
Train loss
1.0
opochs
Figure I: the solution u(x, 1) of heat eqution by Figure 2: train losses of PINN and SPL-PINN SPL-PINN
Ou
Ou
Pu
+ u dr dr²
=0 (z,t) E (-1,1) x (0.1)
Ōt
u(0 x) = sin(TT)
E (-1,1)
(10)
ut, -1] 1) = uut, 1) = te (0,1)
0.01
We choose the training dataset {({ _ ) { and {(t ((ff,=))) )
which ny = nJ
ng 2000. Given the training epoch, the mean squared error is presented in Table 2 while the solution solvving by SPL-PINN is presented Figure 3 In addition, the solution of 10 and the
train losses of PINN and SPL-PINN can be found in Obviously, the great portion of is under
100
075
0117
train
0.75
-1.5
Figure 3: the solution u(æ, t) of Burders eqution by SPL-PINN
Figure 4: train losses of PINN and SPL-PINN
5

---

Under review as a conference paper at ICLR 2023
Table 2: Mean squared error of the initial condition for given epochs SPL-PINN(initia)
epochs
PINN(initial)
10
0.5592
0.5499
100
0.2564
0.2464
200
0.2363
0.2519
250
0.2290
0.2245
Table Mean squared error of the boundary conditions for given epochs
epochs
(a = 1)
SPI-PINN(C: = 1)
PINN( = 1) SPI.PPINN(:
10
0.1451
0.1317
0.1344
0.1274
100
0.0528
0,0339
0.0525
0.0458
200
0.0331
0.0416
0.0397
0.0408
250
0.0388
0.0237
0.0416
0.0205
5
CONCLUSION
Recently, PINN has gained high attention. There many researchers devote their efforts bellishing it. Admittedly, The difficulty converging is huge concern. In an attempt to solve the inefficiency of convergence, samples plugged into the model in a meaning way, from easy ones to hard ones. The performance in hear equation and Burgers equation illustrate the effectiveness of this new pattern.
propose noval SPL-PINN framewrok. It new training pattern,
6 FUTURE WORK
Normally, the loss function of PINN is combined by three parts: the PDE residual, the initial con- dition residual and the boundary condition residual. Considering the three parts maybe in different magnitude, is it fair to put the equal weights them? In future, plan to propose framework to guarantee the faimess of the weights of different losses.
REFERENCES
Physics-informet neural networks: deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 2019.
dual-dimer method for training physics-constrained neural networks with minimax architecture. Neural Newworks, 136: 12 125, 2021.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning, pp. 41-48. As- sociation for Computing Machinery, 2009.
Rafael Bischof and Michael Kraus. Multi-objective loss balancing for physics-informet deep learn- ing. CoRR, abs/21 10.09813. 13, 2021.
Pao-Hsiung Chiu, Jian Cheng Wong, Chinchun Ooi, My Ha Dao, and Yew-Soon Ong. CAN-PINN: A fast physics-informer neural network based on potolee--utommti--nnuee method. CoRR, abs/21 158128 2021.
differentiation
Jungeun Kim, Kookjin Lee, Dongeun Lee, Sheo Yon Jin, and Noseong Park. Dpm: novel training method for physics-informed neural networks in extrapolation, 2020.

---

Under review as a conference paper at ICLR 2023
M. Kumar, Benjamin Packer, and Daphne Koller. Self- learning for latent variable models. In Lafferty. C. Williams, J. Shawe-Taylor, R. Zemel, and A. Culotta (eds.), Advances in Neural Information Processing Systems, volume 23, Curran Associates, Inc., 2010.
Justin Sirignano and Konstantinos Spiliopoulos. Dgm: deep learning algorithm for solving partial differential equations. Journal of Computational Physics, 375:1339-1364, 2018. ISSN 0021- 9991.
Pongpisit Thanasutives, Masayuki Numao, and Ken-ichi Fukui. Adversarial multi-task learning enhanced physics-informed neural networks for solving partial differential equations. In 202/ International Joint Conference on Neural Networks (IJCNN, pp. 1-9, 2021. doi: 10.1109/ 011222887.202995333
Xin Wang, Yudong Chen, and Wenwu Zhu. A survey curriculum learning. IEEE Transactions Pattern Analysis and Machine Intelligence, 44(9):455544576, 2022. doi: 10.1109/TPAML.2211. 3069908.
E. Weinan and B. Yu. The deep ritz method: A deep learning- based numerical algorithm for solving variational problems.. (006-001), 2018.
Zixue Xiang, Wei Peng, Xiaohu Zheng, Xiaoyu Zhao, and Wen Yao. Self-adaptive loss balanced physics-informed neural networks for the incompressible navier-stokes equations, 2021.
Jeremy Yu, Lu Lu, Xuhui Meng, and George Em Karniadakis. Gradient-enhanced physics-informed neural networks for forward and inverse pde problems. Computer Methods in Applied Mechanics and Engineering, 393:1 14823, 2022.


Reviews:
[{"review_id": "wquk72FLAb", "reviewer": "Reviewer_VxbH", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "PINNs have difficulty with convergence. Data is usually randomly sampled according to the authors. This is why the authors propose to use a better sampling strategy.", "summary_of_the_review": "The paper is an incomplete draft", "strength_and_weaknesses": "# Strengths\n\n- good results\n\n#Weaknesses\n\n- badly written\n- limited experiments (only two toy problems)\n- equations unclear and not well explained \n- the \"proposed loss function\" (eq. 4) is completely trivial\n- references are incomplete (no authors are given for the main reference)\n- unclear what lambda is", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "jfcMUrHbnf", "reviewer": "Reviewer_kVD8", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces Curriculum Learning, specifically Self-paced Learning, into the physics-informed neural networks (PINNs) learning, so as to train the model from easy samples to hard ones to improve the convergence. The proposed method simple takes the example-wise training loss as Difficulty Measurer to determine which training sample should be used during the training. The effectiveness of the proposed method is verified in a typical parabolic equation and Burgers equation.", "summary_of_the_review": "This paper seems not to be ready.", "strength_and_weaknesses": "Strength:\n1. The paper focuses on an emerging and important research area, the convergence of PINN.\n2. The idea of introducing Self-paced Learning into PINN training is straightforward and interesting.\n\n\nWeaknesses:\n1. The format of the paper seems not ready for ICLR main conference.\n2. The idea is too naive and the experimental part is too simple and lacks deep analyses.\n3. There are too many typos.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Rg4Ukmg9sw", "reviewer": "Reviewer_mRmU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This manuscript proposes a method that can adaptively scale the loss based on the error of predicting the results of PDE. This manuscript experimented with the method on several classic PDEs, including the heat equation and burger's equation.", "summary_of_the_review": "This submission requires significant work based on the evaluation above.", "strength_and_weaknesses": "Weaknesses:\n- The experiments are too toy to show the true efficacy of the proposed method. Also, the hyperparameters of the experiments are hidden, which makes the result uninterpretable.\n- My main reservation is about the novelty. The proposed method does not contain significant new techniques. In my opinion, I would like to call it a trick instead of a scientific discovery.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LmaVczFmy9", "reviewer": "Reviewer_WN4N", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed to accelerate the training of PINNs by self-paced learning (SPL). A heat equation and Burgers’ equation are used to test the performance of the proposed method.", "summary_of_the_review": "The method proposed in this method has already developed in other papers. The numerical results are not well written, and the method is not well investigated.", "strength_and_weaknesses": "Strengths\n- None.\n\nWeaknesses\n- The paper is not well written.\n- The proposed SPL loss is essentially using a point-wise weight coefficient. Exact the same idea has been proposed in https://arxiv.org/abs/2009.04544 and https://arxiv.org/abs/2001.04860 \n- No sufficient examples.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject