Learning Specialized Activation Functions for Physics-informed Neural Networks
Authors: Honghui Wang
Lu Lu
Shiji Song
Gao Huang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Physics-informed neural network
adaptive activation functions

Abstract: At the heart of network architectures lie the non-linear activation functions, the choice of which affects the model optimization and task performance. In computer vision and natural language processing, the Rectified Linear Unit is widely adopted across different tasks. However, there is no such default choice of activation functions in the context of physics-informed neural networks (PINNs). It is observed that PINNs exhibit high sensitivity to activation functions due to the various characteristics of each physics system, which makes the choice of the suitable activation function for PINNs a critical issue. Existing works usually choose activation functions in an inefficient trial-and-error manner. To address this problem, we propose to search automatically for the optimal activation function when solving different PDEs. This is achieved by learning an adaptive activation function as linear combinations of a set of candidate functions, whose coefficients can be directly optimized by gradient descent. In addition to its efficient optimization, the proposed method enables the discovery of novel activation function and the incorporation with prior knowledge about the PDE system. We can further enhance its search space with adaptive slope. The effectiveness of the proposed adaptive activation function is demonstrated on a series of benchmarks, including the Poisson's equation, Burgers' equation, Allen-Cahn equation, convection equation, Korteweg–de Vries equation and Cahn-Hilliard equation. The performance gain of the proposed method is further interpreted from the neural tangent kernel perspective. Code will be released. 

Under review as a conference paper at ICLR 2023
LEARNING SPECIALIZED ACTIVATION FUNCTIONS FOR PHYSICS-INFORMBL NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
At the heart of network architectures lie the non-linear activation functions, the choice of which affects the model optimization and task performance. In com- puter vision and natural language processing, the Rectified Linear Unit is widely adopted across different tasks. However, there is no such default choice of acti- vation functions in the context of thysics-informee neural networks (PINNs). It is observed that PINNs exhibit high sensitivity to activation functions due to the various characteristics each physics system, which makes the choice the suit- able activation function for PINNs a critical issue. Existing works usually choose activation functions in an inefficient trial-and-ers lem, propose search automatically for the optimal activation function when solving different PDEs. This is achieved by learning an adaptive activation func- tion as linear combinations of set of candidate functions, whose coefficients can be directly optimized by gradient descent. In addition to its efficient optimization, the proposed method enables the discovery of novel activation function and the in- corporation with prior knowledge about the PDE system. We can further enhance its search space with adaptive slope. While being surprisingly simple, the effec- tiveness of the proposed adaptive activation function is demonstrated on a series of benchmarks, including the Poisson's is equation, convection equation, Burgers' equation, Allen-Cabn equation, Korteweg-de Vries equation, and equation. The performance gain of the proposed method is further interpreted from the neural tangent kernel perspective. Code will be released.
manner. To address this prob-
INTRODUCTION
Recent years have witnessed the remarkable progress of physics-inforee informed neural networks (PINNs) on simulating the dynamics of physical systems, in which the activation functions play significant role the expressiveness and optimization of models. While the Rectified Linear Unit (Hahnloser) et al. 2000 Jarrett et al. 2009 Nair& & Hinton| 2010) is widely adopted in most computer vision and natural language processing tasks (Ramachandran et al. (017)). there is no such default choice of activation functions in the context of PINNs. In fact, PINNs show great sensitivity
activation functions when applied to different physical systems, since each system has its own characteristic. On one hand, the utilization of unsuitable activation functions would cause over-parameterization and overfitting. One the other hand, accurate simulations can be achieved with fast convergence and high precision by choosing proper activation functions. For example, the hyperbolic tangent function is shown to suffer from numerical instability when simulating vortex induced vibrations, while a PINN with sinusoidal function can be optimized smoothly (Raissi al. (019b).
The various characteristics of different PDE systems make it critical to select proper activation func- tions in PINNs. The common practice to find the optimal activation functions is by trial-and-error, which requires extensive computational resources and human knowledge. This method is ineffi- cient especially when solving complex problems, where searching for a set of activation functions is necessary make accurate predictions. For instance, combination of the sinusoidal and the exponential function is demonstrated effective to solve the heat transfer equation, whose solution is periodic in space and exponentially decaying in time (Zobeiry & Humfeld, 2021). In this case, the -nd--orr-rrror strategy leads to the combinatorial search problem, which becomes infeasible when the candidate activation function set is large.

---

Under review as a conference paper at ICLR 2023
In this work, we propose simple and effective physics-informe activation function (PIAC), aiming at the automatic design of activation functions for solving PDE systems with various characteristics. Sharing the same spirits with differentiable neural architecture search (Liu et al. 2018), param- eterize the categorical selection of one particular activation function into a continuous search space, leading to end-to-end differentiable problem which can be integrated into the training procedure of PINNs. Specifically, we first define a set of candidate activation functions, which is conceptually composed of simple elementary functions posed PIAC is learned as the linear combination of candidate functions with adaptive coefficients. Besides its efficient optimization, this continuous parameterization also enables the discovery of novel activation functions, whose capacity can be further enhanced by cascading these learnable functions a layer-wise or neuron-wise manner.
commonly-used activation functions. Then, the pro-
Our method can be regraded as adaptive activation functions built on a predefined candidate function set. Although adaptive activation functions have been explored for PINNs, previous works mainly focus on accelerating the convergence of PINNS by introducing the adaptive slope (Jagtap et al. 2020b(a), while leaving the inefficient selection of activation function for different PDE systems unexplored. In fact, method is orthogonal to these methods and can be extended by incorporating the adaptive slope into our search space. Moreover, the introduction of candidate function set can be leveraged to embed prior knowledge about the PDE system into the neural networks. For example, the sinusoidal function can be added into this set to assist the modeling of periodicity.
We evaluate the proposed PIAC series of PDEs, including the Poisson's 's equation, convection equation, Burgers' equation, Allen-Cahn equation, Korteweg-de Vries equation, and Cahn-Hilliard equation. Extensive experiments show that the proposed PIAC can consistently outperform the standard activation functions. We further explain the performance gain from the neural tangent kernel (NTK) perspective (Jacot et al. 2018 Wang et al. 2022)
The main contribution of this paper can be summarized as:
We investigate the influence of activation functions for PINNs to solve different problems and reveal the high sensitivity of PINNs to the choice of activation functions, which can be related to various characteristies of the underlying PDE system;
We explore the automatic design of activation functions for PINNs. The proposed method can be efficiently optimized and adapted to the PDE system, while enabling the utilization of prior knowledge about the problem;
While being simple, the effectiveness of the proposed PIAC is demonstrated on extensive experiments and interpreted from the perspective of neural tangent kernel.
2 PRELIMINARIES AND RELATED WORK
2.1 PHYSICS-INFORIE NEURAL NETWORKS
Physics-informed neural networks have emerged as promising method for solving forward and inverse problems of PDEs (Raissi et al.] 2019a) Chen et al. 2020 Lu et al.] 2021b Karniadakis et al. 202T), fractional PDES (Pang et al. 2201), and stochastic PDES (Zhang et 2019). In this work, we focus on solving the forward problems of PDEs as described in (Raissi et al. 2019a). Specifically, we consider PDEs of the general form
+ NNu(t, æ); A] = 0, a LE C Rd, te [0,T],
(1)
subject the initial and boundary conditions
(0, x) = u (x)), x E
(2)
Blu] = 0, x E on, E [0,T],
(3)
where u(t, æ) denotes the solution, Ni:: A] is a differential operator parameterized by X, B[:] is
boundary operator, and subscripts denote the partial differentiation.
A physics-informed neural network (PINN) u' u'(t, x; 0) optimized to approximate the solution
u ntt x) by minimizing the following objective function
( Lie (0) + (bc(0) (.),),
2

---

Under review as a conference paper at ICLR 2023
where
Nic
(o) = Nic i=1
u (0, z(: 0) u -x
(5)
Noc
Lnc(
B|B/u' (berr Tee: 0)112.
(6)
Nbc
N. (0) = N, Σ lul(t, + NTu (t;, 0); A11
(7)
Here O denotes the parameters of neural networks. The Lie and Lbe measure the prediction error on initial training data (#tc)i Ne and boundary training data (the: be Nbce The residual loss C, is imposed to make the neural network satisfy the PDE constraint on a set of collocation points {G, = To compute the residual loss L., partial derivatives of the neural network output with respect to and æ can be obtained via automatic differentiation techniques (Baydin et al. 2018).
Despite the effectiveness, the introduction of physics- loss function makes the optimization more ill-conditioned (Wang et al.) 2021; Krishnapriyan et al. 2021). Efforts have been made to alleviate this problem from the aspects of loss weight balancing (Wang et al.) 2021 2022), loss function design (Psaros et al.) [2022] Yu et al. 2022). adaptive collocation point sampling (Wight & Zhao, | McClenny & Braga-Neto| 20200 Lu et al. 2222aa] Wuu et al. (022). domain decompo- sition (Jagtap et al. [020cc] Jagtap & Karniadakis] 2020), and curriculum learning (Krishnapriyan et al. 2021). Different from previous works, we shed light on the relationship between the opti- mization difficulty of PDE constraint and activation functions. Our work reveals the high sensitivity of PINNs to the choice of activation function and proposes to reduce the optimization difficulty of PINNs by learning specialized activation functions for different PDEs. We hope our work could inspire further study on the convergence issue of PINNs from the perspective of activation functions.
2.2 ACTIVATION FUNCTIONS
Fixed-shape activation function. Activation functions are crucial for the optimization and perfor- mance of deep neural networks (Glorot & Bengio 2010). The Rectified Linear Units (ReLU) (Hahn- Joser et al. 2000 Jarrett et al.] |009) Nair & Hinton] (010) outperform the logistic and hyperbolic tangent activation functions and have become the default choice for most neural networks. Since then, designing activation functions has been active research direction. Variants with improved learning characteristics are proposed, such as Softplus (Dugas et al. 2000 Glorot et al., 2011), Leaky ReLU (Maas et al.) 2013), ELU (Clevert et al. 2015) and GELU (Hendrycks & Gimpel, 2016). Besides these hand-designed activation functions, Ramachandran et al. (2017) leverages neural architecture search techniques to discover the novel Swish function. In this work, focus on the selection of activation functions for PINNs when solving different problems. While the com- mon practice find the optimal activation function is by trial-and-error, we formulate this manual selection as a learning problem, which can be optimized efficiently.
Adaptive activation functions are explored to find specialized activation functions for different architectures and tasks. The major difference of existing methods lies in the search space. The piece-wise linear function is adopted as the universal function approximator in some methods, such as APL (Agostinelli et al. 2014), RePLU (Li al. 2016) and PWLU (Zhou al. 2021). The formulation of SLAF (Goyal et al. 2019) is based on Taylor approximation with polynomial basis. form its search space. Motivated by the connection between Swish and ReLU, ACON (Ma et al. (02T) is proposed as a smooth approxima- tor to the general Maxout family activation functions (Goodfellow et al. 2013). Our work proposes to learn an adaptive activation function as a weighted sum of candidate functions, whose weights can be adapted to the underlying physics laws when modelling different PDE systems. While sim- ilar ideas have been studied for convolutional neural networks in image classification (Dushkoff & Ptucha| 2016 Qian et al., 2018: Manessi & Rozza 20188) Sütfeld et al.) 2020), some technical chal- lenges remain unexplored in the context of PINNs, which have higher demand for the smoothness and diversity of the candidate functions. First, the optimization of -DEdd constraints needs the activation function provide higher-order derivatives, which causes the failure of widely-used
PAU MMolina al. 2019) leverages Padé approximation
3

---

Under review as a conference paper at ICLR 2023
ReLUs in PINNs. Second, unlike the image classification tasks, different PDE systems could have various characteristics, such as periodicity and rapid decay. This leads to a higher requirement for the diversity of the candidate functions. To overcome these challenges,
propose to build the candidate function set with simple elementary functions to embed the prior knowledge of physics systems, well commonly-used activation functions ensure the diversity.
METHOD
In this section, we first investigate the influence of activation functions in PINNS for solving simple ODE systems with analytical solutions. The results show that the choice of activation functions is crucial for PINNs and depends on the problem. Motivated by this observation, we propose learn specialized activation functions for different PDE systems.
3.1 ACTIVATION FUNCTIONS IN PINNS
Across the deep learning community, the ReLU enjoys widespread adoption and becomes the default activation function. However, there is such default choice of activation functions for PINNs, regardless of the importance of activation functions in the optimization and expressivity of neural networks. One of the reasons lies in the failure of ReLUs in PINNs, whose second-order derivative is zero everywhere. More importantly, PINNs show great sensitivity to activation functions due to the various characteristics of the underlying PDE system. To give a further illustration, we compare the performance of PINNs with different activation functions on the following toy examples.
Problem formulation. Here we consider the one-dimensional Poisson's 's equation as
Au(x) (z), [0, m).
where A is the Laplace operator and f f(x) denotes the source term. As shown in Table 1 we use different source terms to construct analytical solutions with various characteristics. For the problem P1, the solution is composed of trigonometric functions with different periods; for the problem P2, its solution dominant by the exponential function. Following the setting described Yu et al. (2022). we impose the boundary conditions as hard-constraints by choosing the surrogate of solution as
with boundary condition u(0) 0 and u(
(9) where (=:; f) is the output of PINN. Hence, the training objective is reduced to the residual loss L.- Note that we consider extreme case of insufficient collocation points to highlight the difference between activation functions. The details of activation functions and experiment setups can refer to Appendix and Appendix C respectively.
ú (x) = xxx π (x; 0) x,
Table 1: The source terms and analytical solutions for the ID Poisson's 's equation.
Problems
f(z)
w(x)
P1
Σ i=1 isin(ix))
20
20
sin(t
P2
(x3 2Tx²2 + (n² 3)x 24)e
PINNs exhibit high sensitivity to the choice of activation functions. Table 2 shows the results of different activation functions P1 and P2. As we see, the choice of activation functions makes big difference on the prediction accuracy. For example, there exists a large performance gap between the exp and (anh functions on P2 (0.73 + 2.08 vs. 75.77 + 7.93). Moreover, the performance of the same activation function can vary significantly across different problems. One can find that while sin achieves the lowest error on P1. it produces poor results on P2. A similar phenomenon is observed on the exp function. Note that none of these activation functions can achieve the best performance simultaneously on these two problems.
We argue that this sensitivity arises from the various characteristies of each problem. In Figure [ we visualize the second-order derivative of PINNs to analyse the influence activation functions on the optimization of residual loss L,- We find that the commonly-used tanh function
4

---

Under review as a conference paper at ICLR 2023
fails to model the periodic nature of P1 and the rapid decay of P2, exhibiting severe overfitting with insufficient collocation points. In contrast, accurate approximation can be achieved under the same conditions by using activation functions with suitable properties, such as the sin function for P1 and the exp function for P2. However, there dose not exist a generic activation function which could model different characteristics simultaneously. This finding demonstrates the necessity of careful selection of activation functions when solving different problems.
Table 2: Comparisons between different activation functions. L2 relative error (%) is reported.
Problems
exp
tanh
signoold
Softplus
ELU
GFL.U
Swish
Pl
0.91 0.30
39.11
31.32
31.27
45.21
23.51 + 17.82
86.03 4: 64.79
36.68
20.61
19:48 4 18.50
33.16 36.81 1.05 0.42
P2
40.42 = 4.75
0.73 2.08
76.77 + 7:90
56.36 + 10.81
0.65 0.23
40.96 97.75
2.70 4 3.41
1e2
1e3
Exact
Exact
Second-orae oott-oorerrrr derivative oru
sin
Second-orter derivative of
0.6
sin
0.5
exp
tanh
0.2
tanh
0.0
-0.2
0.6
-1.0
0.D
0.5
1.0 1.5
2.0 2.5
3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
(a) The second-order derivative of PINNs for P1. (b) The second-order derivative of PINNs for P2.
Figure I: Visualization of the residual loss L. In the case of ID Poisson's 's equation, the residual loss C, penalizes the deviation of the second-order derivative of PINNs to the exact values on sampled collocation points (black dots in figures). We plot the results of sin, exp and tanh.
3.2 LEARNING SPECIALIZED ACTIVATION FUNCTIONS FOR SOLVING PDES
In light the diversity and complexity of PDEs, it is critical to select proper activation functions in PINNs. Existing works usually make the choice though trial-and-error. However, this strategy is inefficient and leads to the impractical combinatorial search problem in some cases where choosing set of suitable activation functions necessary for solving complex problems. To address this problem, we propose a simple and effective method to search for the optimal activation functions for different problems automatically.
To be specific, we formulate the manual selection of activation functions as learning problem, fol- lowing the spirits of neural architecture search. Our basic idea is to construct the search space with set of candidate activation functions and learn to predict the optimal activation function for differ- ent PDEs. Instead categorically selecting one particular activation function, we relax the search space to be linear combination of all candidate activation functions with learnable coefficients. This continuous search space allows efficient optimization through backpropagation and enables the discovery of novel activation functions.
Formally, we define the physics-informer informed activation function (PIAC) as
exp(a;)
PIAC(x) = Σ G(a))a((x), G(a.) Σ exp(a;)
(10)
where a-))) and a denote candidate activation function and learnable parameter, respectively. Together with gate function G(-), the parameter determines the weight (or coefficient) of its corresponding candidate activation function. We use the softmax as the gate function by default. In this case, the search space is the convex hull of the set of candidate activation functions. When the weights are fixed as one-hot vectors, the proposed PIAC is reduced to its candidate function. When the weights are learnable, a specialized novel activation function can be efficiently searched for different PDE systems thanks to the continuous parameterization. We employ the PIAC by replacing all the activation functions in PINNs. It can be applied in neuron-wiss manner where learnable parameters are allowed to be vary across neurons, or a layer-wise manner where the parameters shared for all neurons in one layer.
5

---

Under review as a conference paper at ICLR 2023
The search space of PIAC is built upon the set of candidate functions F {or. "N). Con- ceptually, we build this candidate function set with simple elementary functions and commonly-use activation functions. First, we can embed prior knowledge about the PDE system into the networks by including appropriate elementary functions. For example, the periodicity and exponential decay commonly seen in physics equations but rarely observed in image data. As shown in Section 3.1 they are difficult to model by neural networks with normal activation functions such as tanh. We can alleviate the modeling difficulty by taking advantage of sinusoidal and exponential functions in our search space. We can also add those elementary functions observed in the initial condition (see the convection equation) or the foree term (see Poisson's 's equation) to help the optimization of corresponding loss function. Second, we add most of commonly-used activation functions into this function set. By doing this, we ensure the diversity of our search space while avoiding the repetitive evaluations of each candidate activation function for different problems. Moreover, the search space can be further enhanced with the adaptive slope, which is proposed to improve the convergence rate of standard activation functions for PINNS (Jagtap al.) [2020baa). This can be achieved by simply introducing a learnable scaling factor 35 for each candidate function os which can be represented as
PIAC(x) = Σ ((o))o.(B.))
(11)
==
EXPERIMENTS
In this section, We evaluate the performance of our method on several benchmarks time-dependent PDEs. Then, provide detailed ablation results analyse the proposed PIAC. Finally, present an intuitive interpretation of the performance gain of our method from the neural tangent kernel perspective. More results on the ID Poisson's 's equation can be found in Appendix C
4.1 EXPERIMENTS ON DEPENDETN PDEs
We demonstrate the generalization ability of PIAC to solve various time-dependent PDEs, ranging from first-order linear PDE to fourth-order nonlinear PDE, including the convection equation, the Burgers' equation, the Allen-Cahn equation, the Korteweg-de Vries equation, and the Cahn-Hilliard equation. Note that for the linear convection problem, we consider the case of high convection coefficient (B=64), which is shown as a difficult problem for vanilla PINNs (Krishnapriyan et al.) 2021 The details of PDEs and experimental configurations can refer to Appendix D
Baselines and PIAC setups. We compare PIAC with several commonly-used activation functions, including the sinusoidal functions (sin), the hyperbolic tangent function (tanh), the logistic function (sigmoid)) the Softplus function (Softplus), the Exponential Linear Unit (ELU), the Gaussian Error Linear Unit (GELU) and the Swish function (Swish). We compare PIAC to other adaptive activa- tion functions which could provide higher-order derivatives, including SLAF Goyal al. (2019), PAU Molina et al. (2019) and ACON Ma et al.] (202I). The details of each activation function can refer to Appendix [ and Appendix B We also compare PIAC with standard activation functions with the layer-wise adaptive slopes (Jagtap et al. 2020a). We employ the PIAC in a layer-wise man- ner by default. We set the candidate function set F as (sin, tanh, GELU, Swish, Softplus]. The learnalle parameters J.N initialized as zeros and optimized jointly with the weights and biases of PINNs. The scaling factors ( are initialized as ones and can be fixed or learnabl.
4.1.1 MAIN RESULTS
Comparisons with fixed activation functions. Table presents the L- relative error of PIAC and seven standard activation functions on five time-dependent PDEs. One can observe that PIAC outperforms standard activation functions on all PDEs. For example, PIAC reduces the error rate by 83% on the convetion equation (0.06 + 0.03% vs. 0.36 + 0.15%), by 46% on the KdV equation (0.34 + 0.05% vs. 0.64 = 0.34%), and by 56% on the Hall--rlll equation (0.50 + 0.15% 1.01 + 1.27%). compared with optimal standard activation function of each problem. More importantly, PIAC performs consistently over all these problems, while standard activation functions suffer from high variance in the performance of different PDEs.

---

Under review as a conference paper at ICLR 2023
Table 3: Comparisons of standard activation functions.PIAC and their counterparts with adaptive
slopes (AS) on time-dependen PDEs. L2 relative error (%) is reported. We also report the average error rate over all problems. The better results are bold-faced.
Convection equation
Burgers' equation
Allen-Cahn
KdV equation
Cahn-HHiliard equation
Average error
Method
equation
sin
0.36 + 0.15
5.18 3.73
3.57 = 0.64
0.63 0.17
1.72 0.61
2.29
tanh
6.83 + 4.79
0.26 1 0.13
1.34 0.54
1.32 + 1.12
4.02 = 4.56
2.75
sigmoid
70.38 + 2.98
1.24+ 1.05
1.63 - 0.13
2.34 + 0.53
3.12 + 2.72
15.74
GELU
39.29 + 35.51
4.43 2.87
3.93 - 0.78
1.21 +
1.01 + 1.27
9.97
Swish
5.59 2.18
8.27 1 4.35
5.56 = 1.28
1.73 4.10
2.22 2.60
4.67
Softplus
55.39 2.46
17.75 8.11
17.72 + 6.04
6.44
9.67 2.57
21.35
ELU
6.67 = 0.96
46.34 2.36
52.55 + 2.95
78.95 + 2.57
2.07 90.77
55.66
SLAF
0.36 = 0.18
43.93 +
33.93 = 9.97
25.23 1.28
1.28
52.57 = 27.93
31,20
PAU
45.78 + 35.47
48.31 8.21
43.83 15.18
68.11 13.49
115.59 3.79
64.32
ACON
3.55 + 1.66
1.18 1.55
3.88 1.82
1.52 0.35
2.46 1.96
2.52
PIAC
0.06 + 0.03
0.21 121:00 0.11
0.76 0.26
0.34 0.05
0.50 - 0.15
0.37
With AS
sin
0.28 + 0.08
1.82 +:2410 1.58
3.57 = 0.46
0.57 0.19
1.73 0.73
1.59
tanh
3.06 = 1.65
0.16
0.81 = 0.21
1.71 + 1.53
2.11 0.49
1.57
GELU
38.00 + 39.53
0.71 0.38
1.99 0.63
0.79 0.17
0.96
0.57
8.49
PIAC
0.05 + 0.02
0.15
0.58 0.15
0.34 + 0.08
0.35 0.07
0.29
Comparisons with other adaptive activation functions. The results different adaptive activa-
tion functions shown in Table B SLAF achieves performance comparable to the best standard activation function on the first-order convection equation, but does not produce accurate predictions for other higher-order PDEs. Although SLAF shares a similar formulation with PIAC, its polyno- mial bases can cause vanishing or exploding gradients due to the high order powers in its derivatives. This problem might hinder the optimization of higher-order PDE-based constraints. PAU suffers from instability of training (see Appendix Figure 2 and performs poorly on all PDEs. We attribute the training instability of PAU in the denominator of its formulation. ACON achieves stable performance but does not surpass the best standard activation function each problem due to the limited flexibility of its formulation. The proposed PIAC outperforms these methods thanks to the diverse and smooth candidate functions, which can be used to incorporate the prior knowledge of different physics systems.
discontinuous derivatives, which arise from the absolute value
Comparisons with activation functions with adaptive slopes. We also compare PIAC with three standard activation function enhanced with adaptive slopes (AS), We find PIAC without AS achieves competitive or even better results. For example. PIAC without AS outperforms the tanh with AS on the Allen-Cahi equation and outperforms the sin with AS on the convection equation. The performance of PIAC can be further improved by incorporating adaptive slopes into the search space. As showcase, PIAC with AS obtains a performance gain by 64% compared to the GELU with AS on the Cahn-Hilliat equation and by 82% compared to the sin with AS on the convection equation. Furthermore, our method has better generalization ability and achieves the lowest average error rate over all problems (0.29% with AS and 0.37% without AS), while the adaptive slope technique cannot alleviate the need the manual selection of activation functions.
Visualization of the learned PIAC. We show the learned PIAC in Appendix Figure One can observe the learned activation functions are different from candidate functions. We also find differ- ences in learned functions of different layers. For example, deeper layer tends to has a larger weight of sin function the cases of convection equation. Moreover, the learned activation functions vary across problems, which conforms to our motivation to learn specialized activation functions for dif- ferent PDE systems. The prediction results of PIAC for all problems can refer to Appendix Figure
4.2 ABLATION STUDY
We ablate different design choices provide better understanding of the proposed method. For all ablation experiments, we deploy waser-wise PIAC to solve the convection equation. The adaptive slope is deactivated for clear comparison.
7

---

Under review as a conference paper at ICLR 2023
Table 4: PIAC ablation experiments on the convection equation. We repeat each experiments 5 times and report the average L2 relative error (%). Default settings are marked in gray
(a) Comprisons of different candidate function set.
(b) Comprisons of different search spaces.
Function set F
#params
error (%)
G(-)
Init
Learnable
emror (5)
(sin, tanh, GELU, Swish, Sofpphs))
16922
0.06 0.03
identity
I/N
0.22 :: 0.22
(sin, tanh, GELU, Swish)
16917
0.06 + 0.03
I--norm
1.18 0.98
(in, tanh, GELU)
16912
0.10 0.12
sigmoid
Zero
<
0.89 + 0.83
(sn,, tanh)
16907
0.33 0.04
softmax
Zero
/
0.06 +: 0.03
sin tanh GELU
16897
0.36 0.15
softmax
Zero
X
1.60 + 1.12
16897
6.83 + 4.79
softmax
Random
X
0.32 + 0.23
16897
39.29 1: 35.51
softmax
Random
V
0.09 + 0.05
Comparing different candidate function set. Here we study the influence of candidate function set by gradually increasing its size as shown in Table 44 We begin with the two most commonly- used activation functions, the sin and tanh functions. In this case, PIAC shown to achieve com- petitive result compared with its two candidate functions. Based on that, we observe that adding new activation functions could lead to better performance. Surprisingly, the introduction of GELU can still improve the performance despite its poor results on this problem. We argue that this perfor- mance gain arises from the enlarged search space. We notice that the performance tends to saturate as the number of candidate functions increases. This indicates that PIAC has good robustness to the choice of candidate function set if this set contains activation functions with sufficient diversity. Notably, the additional parameters of PIAC is negligible compared with total number of weights.
Comparing different gate functions. We also compare the softmax function with other gate func- tions in Table 4b) The identity function G(a;) a includes all linear combinations of candidate activation functions into the search space: while the sigmoid function G(aa)) = 1/(1 + exp(-a:)) aa;) restricts the coefficients to be between 0 and 1. We also consider the L, nooomaaaiiitiion G(a.)
a:/ شكا layl, which allows for negative coefficients and keeps the competition between candidate functions. The trainable {a,) are initialized as zeros for sigmoid and softmax functions and as 1/N for identity and I2 normaaalliniioon functions, where N denotes the number of candidate func- tions. One can observe that the results of softmax function are more stable and more accurate. This implies proper restrictions the search space can work as a regularization to the learning of PIAC.
Effectiveness of learnable coefficients of PIAC. To demonstrate that the learnable coefficients of PIAC effective and necessary, we compare PIAC to two variants with fixed coefficients. The first one initializes its parameters zeros and the second one initializes its parameters from standard normal distribution. As shown in Table the automatically learned coefficients outperforms evenly distributed coefficients and random sampled coefficients by large margin. Furthermore, can observe that the performance gap between the random and constant initialization eliminated these parameters are trainable, which also demonstrates the effectiveness of PIAC's optimization.
4.3 UNDERSTANDING PIAC THROUGH THE LENS OF NEURAL TANGENT KERNEL
The neural tangent kernel is proposed to deseribe the evolution of neural networks (Jacot et al. 2018). Its eigenvalues can be leveraged to analyse the rate of convergence as shown in previous works (Jacot al., 2018 Tancik et al.; 2020 Wang al., 2022). We show that the introduction of PIAC makes the NTK learnable. Empirically, we observe the optimization of PIAC adapts the NTK's eigenvalue spectrum the underlying PDE system and leads to an improvement on the average eigenvalue, which partially explains the stable and fast convergence of PIAC.
Brief introduction NTK.
Specially, the neural tangent kemel of L-layer fully-connected network (4) with parameters 0 is defined as
Ou (z( du (1(")
(1( (x
(x)
g (x) (x) x²)
(12)
00
00
00;
00;
8

---

Under review as a conference paper at ICLR 2023
where æ and x' are two inputs. As demonstrated in Jacot et al. (2018) in the infinite- width limit and under suitable conditions, the NTK +LL) converges to a deterministic kernel O% at initialization and stays constant during the training. This limiting kernel can be represented as
I.
Σ 1=1
(x,
II
((,))))))
1=++1
(13)
ya(1 (0, (0, =
3²
(041) (x, (x,x')
B².
The expectation in Eq. 13 taken with respect to centered Gaussian process f with covariance E(1; no is the dimension of input æ and / a scaling factor; a and denote the activation function and its derivative, respectively. Note that the covariance E() depends on the choice of activation functions and so dose the NTK (o
Note (2)) x²) = 1 x" B2..
PIAC and the learnable NTK. For simplicity, we derive the NTK of standard neural networks with PIAC. The NTK of PINNs with PIAC can be derived in similar way. Firstly, we assume that the optimization of neural networks with PIAC can be decomposed into two phases, where we learn the coefficients of PIAC in the first phase and then train the parameters of neural network in the second phase. This assumption is reasonable as number parameters of PIAC far less than those of networks and they quickly converge at the early stage of training. Empirically, we observe that a short warmup (2.5% of the whole schedule) is sufficient for PIAC to learn suitable activation functions and to achieve competitive performance compared with the counterpart whose coefficients are updated during the whole schedule (0.58% vs. 0.60% on the Allen-Cahn equation). By decou- pling the updates of PIAC and networks, we find the second optimization phase is equivalent the training of standard network with a learned activation function, whose NTK is derived as
N
N
L N
N
((L) = Σ 1=1
G(az)G(a,) (1) ojo (x. x²))
II P=1+1 =1
G(oz)G(az) (5(1)) (ac, (a.
1=1
x((+!) (x, x') = E~ (f)))),
b((+t) (z, x') = 11 (f(
(14)
We omit the factor 3 for convenience. One can observe that the covariance X in Eq 13 is re- placed with a weighted sum of X!!! Lio, which is calculated with different combinations of candidate activation functions. Note that the weights are learned in the first training phase. To conclude, the introduction of PIAC leads to a learnable NTK which could be adapted to the underlying PDE.
The eigenvalue spectrum of NTK with PIAC. Through the optimization of PIAC, we can mod- ify the limiting NTK 6/6) which corresponds to change in the convergence behavior of the neural network. We show empirically that the optimization of PIAC has an effect on the NTK's eigenvalue spectrum. As shown in Appendix Figure 3d PIAC leads to a larger average eigenvalue compared with the best standard activation function on the Allen-Cahn equation (5859 vs. 2407), which im- plies a larger convergence rate (Wang et al. 2022).
CONCLUSION
In this paper, we reveal the high sensitivity of PINNs to the choice of activation functions and relate to various characteristics of the underlying PDE system. We sought to learn specialized activation functions automatically for PINNs to avoid the inefficient manual selection of activation functions and to alleviate the optimization difficulty of PINNs. The proposed physics-informed activation function is presented as learnable combinations of set of candidate functions, whose coefficients can be adapted to the governing PDEs. Intuitively, we show that PIAC makes the neural tangent kernel of PINNs learnable, which partially explains the performance improvement of PIAC. Extensive experiments on a series of challenging benchmarks demonstrate the effectiveness and generalization ability of PLAC.

---

neural networks. Proceedings of the Under review as a conference paper at ICLR 2023
REFERENCES
Forest Agostinelli, Matthew Hoffman, Peter Sadowski, and Pierre Baldi. Learning activation func- tions to improve deep neural networks. arXiv preprint
2014.
Atilim Gunes Baydin, Barak Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. Automatic differentiation in machine learning: a survey. Journal of Marchine Learning Research, 18:1-43. 2018.
Richard H Byrd, Peihuang Lu, Jorge Nocedal, and Ciyou Zhu. A limited memory algorithm for bound constrained optimization. SIAM Journal on scientific computing, 16(3):1100-128, 1995.
Yuyao Chen, Lu Lu, George Em Karniadakis, and Luca Dal Negro. Physics-informed neural net- works for inverse problems in nano-opties and metamaterials. Optics express, 28(8):11618 11633, 2020.
Djmrk-Amé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289, 2015.
Charles Dugas, Yoshua Bengio, François Bélisle, Claude Nadeau, and René Garcia. Incorporating second-order functional knowledge for better option pricing. Advances in neural information processing systems, 13, 2000.
Michael Dushkoff and Raymond Ptucha. Adaptive activation functions for deep networks. Elec- tronic Imaging, 2016(19):1-5, 2016.
Stefan Elfwing, Eiji Uchibe, and Kenji Doya. Sigmoid-weighted linear units for neural network function approximation in reinforcement learning. Neural Networks, 107:3-11, 2018.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256. JMLR Workshop and Conference Proceedings, 2010.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourleenth international conference on artificial intelligence and statisticy, pp- 315-323. JMLR Workshop and Conference Proceedings, 2011.
lan Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio. Maxout networks. In International conference on machine learning, pp. 1319-1327. PMLR, 2013.
Mohit Goyal, Rajan Goyal, and Brejesh Lall. Learning activation functions: A new paradigm of understanding neural networks. arxiv 2019, arXiv preprint arXiv: 1906.0959, 2019.
Richard HR Hahnloser, Rahul Sarpeshkar, Misha Mahowald, Rodney J Douglas, and H Sebastian Seung. Digital selection and analogue amplification coexist in cortex-inspired silicon circuit. nature, 405(6789):447-55. 2000.
Dan Hendrycks and Kevin Gimpel. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. 2016.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kemel: Convergence and gen- eralization in neural networks. Advances in neural information processing systems, 31, 2018.
Ameya Jagtap and George Em Karniadakis. Extended physics-informed neural networks (xpinns): A generalized space-time domain decomposition based deep learning framework for nonlin- car partial differential equations. Communications in Computational Physics, 2865):2002-204) 2020.
Ameya D Jagtap, Kenji Kawaguchi, and George Em Karniadakis. Locally adaptive activation func- tions with slope recovery for deep and Royal Society A, 22222239) 2020a.
Ameya D Jagtap, Kenji Kawaguchi, and George Em Kariiadakis. Adaptive activation functions accelerate convergence in deep and innsice-informed neural networks. Journal of Computational Physics, 404: 109136, 2020b.
10

---

Under review as a conference paper at ICLR 2023
Ameya D Jagtap, Ehsan Kharazmi, and George Em Karniadakis. Conservative physics-informet neural networks on discrete domains for conservation laws: Applications to forward and inverse problems. Computer Methods in Applied Mechanics and Engineering, 365: 13028, 2020c.
Kevin Jarrett, Koray Kavukcuoglu, Mare' Aurelio Ranzato, and Yann LeCun. What the best multi-stage architecture for object recognition? In 2009 IEEE 12th international conference computer vision, pp. 2146-2153 IEEE, 2009.
George Em Karniadakis, Ioannis G Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang. Physics-informed machine learning. Nature Reviews Physics, 3(6))422-440, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Aditi Krishnapriyan, Amir Gholami, Shandian Zhe, Robert Kirby, and Michael W Mahoney. Char- acterizing possible failure modes in physics-informed neural networks. Advances Neural In- formation Processing Systems, 34, 2021.
Jun Li, Tong Zhang, Wei Luo, Jian Yang, Xiao-Tore Tong Yuan, and Jian Zhang. Sparseness analysis in the pretraining deep neural networks. IEEE transactions on neural networks and learning systems, 28(6): 14t--14425848 2016.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. In International Conference Learning Representations, 2018.
Lu Lu, Xuhui Meng, Zhiping Mao, and George Em Karniadakis. Deepxde: deep learning library for solving differential equations. SIAM Review, 63(1):208-222 2021a.
Lu Lu, Raphael Pestourie, Wenjie Yao, Zhicheng Wang, Francesc Verdugo, and Steven G Johnson. Physics-informed neural networks with hard constraints for inverse design. SIAM Journal on Scientific Computing, 43(6):B1106 B11322, 2021b.
Ningning Ma, Xiangyu Zhang, Ming Liu, and Jian Sun. Activate or not: Learning customized activa- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 8032-8042, 2021.
Andrew L Maas, Awni Y Hannun, Andrew Y Ng. et al. Rectifier nonlinearities improve neural network acoustic models. In Proc. ieml, volume 30, pp. 3. Citeseer, 2013.
Franco Manessi and Alessandro Rozza. Learning combinations of activation functions. In 2018 24th international conference on pattern recognition (ICPR), pp. 61-66. IEEE, 2018
Revanth Mattey and Susanta Ghosh. A novel sequential method to train physics informed neural networks for allen cahn and cahn hilliard equations. Computer Methods in Applied Mechanics and Engineering. 390:1 14474, 2022.
Levi McClenny and Ulisses Braga-Neto. Self-adaptive physics-informed neural networks using soft attention mechanism. arXiv preprint arXiv:2009.044 2020.
Alejandro Molina, Patrick Schramowski, and Kristian Kersting. Padé activation units: End-to-end learning of flexible activation functions in deep networks. arXiv preprint arXiv: arXiv: /973.0677, 2019.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In lcml, 2010.
Guofei Pang, Lu Lu, and George Em Karniadakis. fpinns: Fractional physics-infomme neural networks. SIAM Journal on Scientific Computing, A263326660- 2019.
Apostolos F Psaros, Kenji Kawaguchi, and George Em Karniadakis, Meta-learning pinn loss func- tions. Journal of Computational Physics, 458:111121, 2022.
Sheng Qian, Hua Liu, Cheng Liu, Si Wu, and Hau San Wong. Adaptive activation functions in convolutional neural networks. Neurocomputing, 272:204 212, 2018.

---

Under review as a conference paper at ICLR 2023
Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-inis inysice-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686-707, 2019a.
Maziar Raissi, Zhicheng Wang, Michael S Triantafyllou, and George Em Kamiadakis. Deep learn- ing of vortex-induced vibrations. Journal of Fluid Mechanicy, 861:119-137, 2019b.
Prajit Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions. arXiv preprint arXiv: vrXiv:/710.05411 2017.
Leon René Sütfeld, Flemming Brieger, Holger Finger, Sonja Füllhase, and Gordon Pipa. Adaptive blending units: Trainable activation functions for deep neural networks. In Science and Informa- fion Conference, pp. 37-50. Springer, 2020.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Ken Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Process- ing Systems, 33:7537-75477 2020.
Sifan Wang, Yujun Teng, and Paris Perdikaris. Understanding and mitigating gradient flow patholo- gies in physics-informee neural networks. SIAM Journal on Scientific Computing, 43055) A3081, 2021.
Sifan Wang, Xinling Yu, and Paris Perdikaris. When and why pinns fail to train: A neural tangent kernel perspective. Journal of Computational Physics, 449: 10768, 2022.
Colby Wight and Jia Zhao Solving allen-cahn and cahn-hilliard equations using the adaptive physics informed neural networks. arXiv preprint arXiv:2007.04542, 2020.
Chenxi Wu, Min Zhu, Qinyang Tan, Yadhu Kartha, and Lu Lu. A comprehensive study of non- adaptive and residual-based adaptive sampling for physics-informed neural networks. arXiv preprint arXiv:2207.10288 2022.
Jeremy Yu, Lu Lu, Xuhui Meng, and George Em Kariiadakis. Gradient-enhanced physics-infomme neural networks for forward and inverse pde problems. Computer Methods in Applied Mechanics and Engineering, 393:1 14823, 2022.
Dongkun Zhang, Lu Lu, Ling Guo, and George Em Karniadakis. Quantifying total uncertainty in physics-informed neural networks for solving forward and inverse stochastic problems. Journal of Computational Physics, 397:108850, 2019.
Yucong Zhou, Zezhou Zhu, and Zhao Zhong. Learning specialized activation functions with the piecewise linear unit. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 12095-12104, 2021.
Navid Zobeiry and Keith D Humfeld. A physics- machine learning approach for solving heat transfer equation in advanced manufacturing and engineering applications. Engineering Applications of Artificial Intelligence, 101:104232, 2021.
12

---

Under review as a conference paper at ICLR 2023
A STANDARD ACTIVATION FUNCTIONS
We compare the proposed PIAC with several standard activation functions for solving series of PDEs. The details are shown as follows.
sin:
f(x) sin (3 * x)
(15)
where scaling factor 8 is set to I as default.
exp:
f(z) ((Bmx)
(16)
where 3 to as default. For the P1 setting of ID Poisson's equation, 8 is set 0.25 for better performance.
tanh:
f(x) e + 6-
(17)
sigmoid:
f(x) =
(18)
1 1
Softplus
DDgass et al. 2000; Glorot et al.
2011b: p:
((x log(1 + Bxr)
(19)
8
where 3 = 1. Softplus can be regard as a smooth version of ReLU.
the Exponential Linear Unit (ELU) (Clevert et al.
2013):
if x 2 0,
f(x)
(20)
1,, if x < 0.
the Gaussian Error Linear Unit (GELU) (Hendrycks & Gimpel 2016):
(xx x (xx)
(21)
where ((x) is the cumulative distribution function of the standard Gaussian distribution.
Swish (Hendrycks & Gimpel 2016 Ramachandran al. 2017: Elrwing et 2018):
2
f(x) 1+e-
(22)
B ADAPTIVE ACTIVATION FUNCTIONS
We further compare the proposed PIAC with other adaptive activation functions which could provide higher-order derivatives. The details are shown as follows.
The SLAF et al. (2019) proposes to learn activation functions with Taylor polynomial bases, which is presented as:
((x)
(23)
i=0
where m is set to 5 in our experiments. The learnable parameters {a.)m are initialized from a normal distribution.
The PAU Molina et al. (2019) is presented the rational functions of the form
Σn 0,
f(x)
(24)
1+ C ba
where m is to 5 and is 4 following the default setting Molina al. (2019) The
absolute value is to avoid a zero-valued denominator. The learnable parameters {a, /e and
{ are initialized from a normal distribution.
The ACON Ma et al. (2021) is smooth approximator to the general Maxout family acti- vation functions, which is presented
f(x) = (pr P2)æ * sigmoid(3 - -22)x) + P2æ,
(25)
where P1, P2 and 8 learnable parameters and initialized from normal distribution.
13

---

Under review as a conference paper at ICLR 2023
C EXPERIMENTS ON ID POISSON'S EQUATION
Experiment setups. For each problem, we use 4-layer multilayer perceptron (MLP) with the hidden dimension set to 16. The model is initialized by the Xavier initialization (Glorot & Bengio. 2010) and trained for 20000 iterations using the Adam optimizer (Kingma & Ba 2014). We initialize the learning rate as 1e-3 and adapt it with a half-cycle cosine decay schedule. We use 32 collocation points for both P1 and P2. For each problem, we experiment with 8 standard activation functions as described in Appendix A We repeat each experiment 10 times and report the mean and standard deviation the L2 relative error.
Comparisons between PIAC and standard activation functions. To demonstrate the effective- ness of PIAC, we use it to learn combinations of the sin and exp functions for solving the ID Poisson's 's equation described in Section As shown in Table although the sin and exp func- tions can achieve the lowest l2 relative error P1 and P2 respectively, they perform poorly on the other problem. We expect the PIAC could adapt its attention over these two candidate functions ac- cording to the problem. We define F {sin, exp). The trainable parameters {a; ) i= are initialized as zeros, which implies equal attention over all candidate functions.
The results of PIAC and its two candidate functions presented in Table 5a It can be observed that PIAC can achieve comparable accuracy with the sin on P1 and outperform the exp on P2, despite the large performance gap between its two candidate functions. We also report the values of learned coefficients in Table 5b For P1, PIAC pays more attention on the sin function as expected. For P2, different across layers. The exp preferred only
observe that the learned activation functions on the third layer, which still leads a performance gain compared with standard activation functions. Overall, the results indicate that the proposed method can learn useful combination coefficients by considering the effectiveness of candidate functions for different problems.
Table 5: Comparisons of standard activation and PIAC on the ID Poisson's 's equation. L2 relative error (%) is reported. We also report the learned coefficients of sin, Osgin- Note that the coefficients of exp can be obtained by Cexp = 1 Osins since the softmax gate function is used.
(a) The La relative error (%).
(b) The learned coefficients Qain of different layers.
Problem
P1
P2
Layers
P1
P2
sin
0.91 + 0.30
49.42 + 4.75
1
0.81 + 0.02
0.58 + 0.08
exp
39.11 + 32.32
0.73 + 2.08
2
0.75 = 0.06
0.53 0.05
PIAC
0.88 + 0.42
0.01 + 0.006
3
0.84 = 0.04
0.45 + 0.06
D EXPERIMENTS ON TIME-DEPENDENT PDEs
D.1 PROBLEM FORMULATIONS
Convection equation.
Consider a one-dimensional linear convection problem as
u + Bu, = 0, x E [0, 2m]. E [o, 1]. u(0,x)) sin(x), ult, 0) utt, 2T))
(26)
The solution of this problem is periodic over time, whose period is inversely proportional to the con- vection coefficient B. Previous work (Krishnapriyan et al. 2021 finds it difficult for vanilla PINNs to learn the solution with a large B and proposes to train the PINNs with curriculum learning. To be specific, the 3 initialized as small value and gradually increases during the training. Despite its effectiveness, the training cost is increased significantly. We show that accurate prediction can be achieved with a normal training strategy by simply using a suitable activation function. Moreover, the performance can be further improved by our method. Here we set the convection coefficient to 64. We use N₁, = 512 training points for the initial condition and Nic = 200 for the boundary condition. The residual loss L, is computed N, 6400 randomly sampled collocation points.
14

---

We choose the following surrogate of Under review as a conference paper at ICLR 2023
Burgers' equation.
We consider the mne-iimensional Burgers' equation:
+ uty Vulzz [-1. t E [0. x(0,z) = -sin(Txr) u(t, -1 = w(t, = 0,
(27)
where y/ = TT. Following (Lu et al. 2021a), we uniformly sample Mic - 256 and Nbc 100 points for initial and boundary training data, respectively. We compute the residual loss C, on N, 4800 collocation points, which are randomly selected from the spatial-temorral domain.
Allen-Cahn equation.
We next consider the Allen-Cahn equation as
"4 Duza 5(u n²),, 1-1, 11, E [0,
(28)
u(0, x² coccos(TT) (TT), u(l, -1) 1t
where D 0.001 following the setting in YYu et al.
2022)
solution to enforce the initial and boundary conditions:
it x) x cos((T)))) + ((1 (1,,
(29)
where vt(t,z;0) is the output of neural networks. We use N, 8000 collocation points.
Korteweg-de Vries equation.
We consider the Korteweg de Vries (KdV) equation as
+ X uu, + A2 = O, x [-1,1]] t E [0, 1]. w(0, x) cos( u(t, 11 u(t, 1), u2 (t, -1) u2 (t, 1),
(30)
where A = and A2 = 0.0025 following the setting in (Raissi et al., 2019a). We choose the following surrogate of solution to enforce the initial conditions:
ā(t, cos( + tu tu'(t, 2t, 0),
(31)
where (t(t, 2);) 8) is the output of neural networks. We Nbc 200 training points for the bound- ary condition and N, = 8000 collocation points for the residual loss L.
Cahn-Hilliare equation.
We consider the phase space representation of Cahn-Hilliard equation
Ue (2(-11 + Az(n² - -)) = =0, h V x E [-1,1, / E [0, 1], u(0,x)) cos(TT)) - exp( - )(Tx)))))))))))))>)))))))))))) u(1, -1) 100=-((
- u(t, 1), un,(, - (1, (1, 1), h(tt, 1), ht, -1) (t,
(32)
where X 0.02 and X2 = following the setting in (Mattey Ghosh, 2022). Previous work proposes novel sequential training method to solve this strongly non-linen non-linear and high-order prob- lem (Mattey & Ghosh 2022). We find that a normal training strategy can lead to a comparable result if we add more collocation points around = 0. To be specific, sample 4000 collocation points from time interval [0, 0.05) and 8000 points from [0.05, 1]. We set Ni, 256 and Nbc. = 100. We show that PINNS under this training setting work strong baseline and can be further improved by PIAC.
D.2 EXPERIMENTAL CONFIGURATIONS
Experiment setups. We train the PINNs by a two-stage optimization except the convection equa- tion. At the first stage, the model trained for certain number of iterations by Adam (Kingma & Ba 2014). Then, the L-BFGS (Byrd et al. 1995) is used to train the network until convergence. The first stage is to provide good initialization for the L-BFGS optimizer. In the case of convection equation, only the Adam optimizer is used. The detailed experimental configurations can be found in Table The model is initialized by the Xavier initialization. The learning rate adapted with a half-cycle cosine decay schedule in the first stage with Adam optimizer. We repeat each experiment 5 times and report the mean and standard deviation of the Lg relative error. We run all experiments on one NVIDIA GeForce RTX 2080Ti GPU.
15

---

Under review as a conference paper at ICLR 2023
Table 6: The experimental configurations for all problems.
Network
Adam
L-BFGS
Data
Problems
Depth
Hidden dims
Iters
Max Iters
Nic
Nbc
N,
Convection equation
6
64
100k
2e-3
512
200
6400
Burgers' equalion
4
32
15k
le-3
15k
256
100
5300
Allen-Cahn equation
4
32
40k
le-3
15k
8000
KdV equation
4
32
40k
1e-3
15k
200
8000
Cahn-Hilliard equation
4
32
100k
le-3
15k
256
100
15000
D.3 VISUALIZATION
We show the training losses of different activation functions in Figure 2 One can observe SLAF seems to get stuck local minima when solving the Burgers' equation and the Allen-Cahn equation. We find PAU suffers from instability of training its loss oscillates severely. ACON achieves better performance than SLAF and PAU, but does not surpass tanh on both PDEs. Compared to these learnable activation functions, the training loss of PIAC is more stable and converges faster.
We present the learned activation functions of PIAC Figure B and the prediction results of PLAC in Figure 4 We also visualize the eigenvalue spectrum of NTK with PIAC in Figure [3d which shows a larger average eigenvalue compared with the best standard activation function on the Allen-Cahn equation.
0.0200
tanh
0.175
0.0175
SLAF
0.150
ACON
PAU
0,0150
PAUI
0.125
0.0125
PIAC
0.100
0.0100
0.075
0.8075
0.050
0.0050
0.4
0.6
1.2
1.4
0.0
2.0
2.5
3.0
3.5
Iterations
1e4
Iterations
1e4
(a) The Burgers' equation
(b) The Allen-Cahn equation
Figure 2: The training loss of different activation functions.
Tanh
8-
GELU
Sertpaux
3
19m1 auttage: 2407
PAC: average: 3252
100
10
-10
19
20
(a) Candidate functions (b) Convection equation (c) KdV equation
(d) Eigenvalue spectrum
Figure 3: Visualization of the learned activation functions and the eigenvalue spectrum. (a) Candi- date functions of PIAC. (be) The learned activation functions for the convection equation and the KdV equation. The curves of PIAC with and without the adaptive slope are presented. (d) The eigenvalue spectrum of NTK matrix the training data of the Allen-Cahn equation.
16

---

Under review as a conference paper at ICLR 2023
075
0.73
0.5
0.00
2,000
1,000
-1.00
4.50
(a) The convection equation
025 123
625
6.003
-120m
4w
-1 20
-1 200
(b) The burgers' equation
0.05
0.53
32
056
0001
0.00
8.00
324
-225
-22
1.00
0.00
(c) The Allen-Cahn equation
130
0.004
4.000
(d) The KdV equation
52-
10-
k2
3.00
800
4,004
-25-
20
30as
(c) The equation
Figure 4: Visualizations of the predictions PIAC. For each problem, we show the exact solution (left), the prediction (middle) and the absolute error between them (right).
17


Reviews:
[{"review_id": "pSZh185xr_", "reviewer": "Reviewer_5qLY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The goal of this work is to automatically learn the activation functions via the combinations of different candidate activation functions for PINN. Specifically, it adopted gate function with a learnable parameter to identify the coefficients for different candidate activation functions. Extensive experiments are carried out to evaluate the proposed physics-informed activation functions on a variety of partial differential equations (PDEs). The results demonstrate the efficacy of the simple approach. ", "summary_of_the_review": "It is interesting to automatically learn the activation functions for PINNs. The proposed method is simple yet effective for PDEs systems. However, this work adopted the same gate function method from prior work and apply it to PINNs. Hence, the technical innovation is somewhat limited.", "strength_and_weaknesses": "* Strength\n1. It is interesting to automatically choose the activation functions for PINN\n2. The proposed method is simple yet effective to solve PDE systems\n3. Conduct extensive experiments on various PDEs to verify the effectiveness of the proposed approach\n\n* Limitations\n1. Technical innovation is somewhat limited. The proposed gate function to learn the coefficients of activation functions is the same as that in the literature [Qian et al. 2018]\n2. Compare to more baselines. I am curious if you could compare the prior works that learn combinations of activation functions?\n3. A few grammatical errors. For example, \"by minimize the following objective function\" in the bottom of page 2; and \"we consider a extreme case of insufficient collocation points\", a-> an\n\nReferences:\\\n[Qian et al. 2018] Adaptive activation functions in convolutional neural networks, 2018.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "emIJpG4HSH", "reviewer": "Reviewer_qn6w", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors present a formulation to create a trainable activation function based on a convex combination of other activations.\nThis new mixture approach is intended to work on physics-informed neural networks (PINNd) where higher order derivatives play a role, thus causing problems for some of the piecewise linear based common activations.\n\nThe authors the provide an empirical evaluation as well as a link to the neural tangent kernel.", "summary_of_the_review": "In this paper, the authors present an activation function that is amenable to PINNs. The authors present the problem clearly and offer a solution for this particular context based on previous work. They however do not compare to other adaptable activation functions that might still behave well in this domain. \nThe empirical evaluation would be much stronger, if you were to compare to that class of activations. In particular, it would be great to show the benefits of your activation function, or explain why the others are not suitable and demonstrate that empirically.", "strength_and_weaknesses": "The authors present a technically sound approach for PINNs. Evaluated on datasets that clearly present the difficulties in this domain. \n\nHowever, the main weakness of this paper is in the lack of comparison to other learnable activation functions. The authors present these family of activation functions in the section \"Adaptive activation functions\", yet they are not present in the evaluation.\nSome of them could be relevant as they do provide higher oder derivatives. Maybe there is an issue with those activation functions and that is a good reason not to include them, but this is not clear from the paper.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "D_mJouO-0k1", "reviewer": "Reviewer_E27E", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes trainable activation functions for physics-informed neural networks (PINNs). The basic idea is to consider a set of base activation functions, and build a trainable one as a convex combination of the base ones. They provide experiments on many problems, showing that PINNs are generally sensitive to the choice of AF, while the trainable one provides good performance across most cases. They also try to motivate the approach from the point of view of the neural tangent kernel (NTK).", "summary_of_the_review": "The method is a rebranding of a known technique, applied on a different domain. While this is potentially interesting for PINNs, I feel this is not a good practice and it is also an artificial construct to increase novelty.", "strength_and_weaknesses": "Strength: From the point of view of PINNs, the proposed neural networks have better accuracy and performance than the base ones. \n\nDrawbacks: \n- From the methodological point of view, the method is *equivalent* to the method proposed in (Manessi and Rozza, 2019), referenced in the paper, which was also proposed under the name of \"adaptive blending unit\" in (Sutfeld et al., 2020), which is not referenced here. The authors are renominating them as \"physics-informed activation function\" (PIAC), but as far as I can see there is no difference that warrants a change of name (apart from slightly modifying the number of base AFs). They mention that \"previous methods experiment with limit choice of activation functions\", but they use 5 base AFs which is similar to 6 base AFs in (Sutfeld et al., 2020). \n- The method is also compared only with fixed or parametric AFs, and not against similar expressive AFs (e.g., APLs).\n- The NTK derivation is quite shallow and it does not provide any special insight into the AFs themselves.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "8pBTbDHbWu", "reviewer": "Reviewer_J89g", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present a method for parameterizing activation functions to improve convergence in physics-informed-neural-networks. They demonstrate through empirical experimentation that the proposed method converges to better solutions than non-learned activation functions or adaptive slope methods. Finally, the authors provide an analysis based on Neural Tangent Kernel that sheds light on why the proposed method is effective.\n", "summary_of_the_review": "Overall this paper is a solid demonstration of an approach to address the training difficulty empirically observed in PINNs.\n\n", "strength_and_weaknesses": "The method proposed by the authors addresses a significant issue in PINNs. The approach is demonstrated on 5 different PDEs. The method is compared to several baselines as well as some ablations. \n\nThe NTK based analysis is quite hand-wavy. There is no treatment of the approximation error of the two-phase analysis and the notion that the NTK is learned to suite the underlying PDE is approached empirically with a single example.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject