On training Physics-Informed Neural Networks for Oscillating Problems

Authors: martin  hofmann-wellenhof
alexander  fuchs
franz  pernkopf
Venue: ICLR_cc_2024_Workshop_AI4DiffEqtnsInSci
Published: 2024-02-03 09:51:26

Abstract: Physics-Informed Neural Networks (PINNs) offer an efficient approach to solving partial differential equations (PDEs). In theory, they can provide the solution to a PDE at an arbitrary point for the computational cost of a single forward pass of a neural network.
However, PINNs often pose challenges during training, necessitating complex hyperparameter tuning, particularly for PDEs with oscillating solutions.
In this paper, we propose a PINN training scheme for PDEs with oscillating solutions. We analyze the impact of sinusoidal activation functions as model prior and incorporate self-adaptive weights into the training process.
Our experiments utilize the double mass-spring-damper system to examine shortcomings in training PINNs.
Our results show that strong model priors, such as sinusoidal activation functions, are immensely beneficial and, combined with self-adaptive training, significantly improve performance and convergence of PINNs.

Main paper: 
 introduction: 
 A multitude of practically relevant systems can be modelled and described via partial differential equations (PDEs), creating the demand for fast and reliable PDE solvers in both research and industry. In PINNs (Raissi et al., 2019) , PDEs are used as constraints that are implemented via a problem-specific loss function. Through automatic differentiation (Baydin et al., 2018) , partial derivatives can be obtained efficiently and are combined to form the PDE. This PDE constraint is then enforced during optimization and provides strong priors to the model without problem-specific architectures. PINNs can solve equations over a continuous domain and obviate the need for expensive mesh creation.Physics-informed models have successfully been applied to problems in fluid mechanics, solid mechanics, and optics (Wong et al., 2022) . However, they are known to require cumbersome hyperparameter tuning, especially if the solution of the learned PDE contains high-frequency oscillations (Wang et al., 2020) . Since oscillating phenomena are ubiquitous, e.g. in multi-body systems, an improved training scheme for these types of problems is of particular interest. In recent years, multiple methods for improving the performance of PINNs have been proposed, ranging from domain decomposition (Jagtap & Karniadakis, 2021) , over variational formulation of PINNs (Kharazmi et al., 2019) to transfer learning (Desai et al.) . Two methods that should be especially suited to improve the convergence behaviour for oscillating problems are sinusoidal activation functions in the initial network layer (Wong et al., 2022) and self-adaptive PINNs (McClenny & Braga-Neto, 2020) .While prior works have motivated the sinusoidal activations from a learning perspective or a general ability to improve high-frequency model behavior (Wong et al., 2022; Sitzmann et al., 2020) , recent theoretical advances in this field argue that the neural activation function should optimally also represent a fundamental solution of the PDE (Ranftl, 2022) . This result implies an application of the fundamental solution activation in the model's last layer. With the fundamental solution being challenging to obtain for complex systems, we propose using sinusoidal activation functions repeatedly throughout the model, as it (i) approximates fundamental PDE solutions for oscillating systems as well as (ii) leverages the beneficial learning properties discussed in (Wong et al., 2022) . This paper analyzes the impact of introducing sinusoidal activations throughout the PINN model in combination with self-adaptive PINNs. We analyze its potential for improving accuracy and reducing training time. We also show that for high-frequency problems, employing multiple layers of sine activation functions in PINNs is beneficial, and a single layer might not be enough. This also translates to physics-informed deep neural operator networks trained for multiple initial conditions, where sinusoidal activation functions provide considerable performance gains, as shown in Appendix A.4. Our experiments use the double mass-spring-damper (dMSD) model as a benchmark problem (results for the wave equation can be found in Appendix A.3). 
related work: 
 The idea of incorporating prior knowledge as a PDE into a neural network has existed since the 1990s (Dissanayake & Phan-Thien, 1994) . However, the method became viable by exploiting automatic differentiation. Raissi et al. (2019) coined the term physics-informed neural networks in 2019 and reignited interest in this method. The spectral bias of PINNs (Wang et al., 2020) , the difficulty in training PINNs, and the subsequent study of potential failure modes during training (Rohrhofer et al., 2022) , (Wang et al., 2021a) lead to a multitude of extensions of the original PINN framework. For oscillating problems, two research directions are of particular interest: sinusoidal feature mapping and loss weighting schemes. Multiple works report the performance benefits of using sinusoidal feature mapping. Here, the two main approaches either utilize sinusoidal activation functions (Raissi et al., 2018) , (Fang, 2021) , (Zobeiry & Humfeld, 2021 ), (Sitzmann et al., 2020) or employ multi-scale Fourier feature architectures (Wang et al., 2021b) . Wong et al. (2022) are the first to systematically study the impact of sine activation functions. They advocate employing them in the first layer only and show that using multiple layers with sine activation functions performs similarly. However, in (Ranftl, 2022) , it is shown that the activation function should represent a fundamental solution of the PDE, which motivates the use of a sine activation function in the last layer.Several methods were proposed to ease the challenge of balancing the data and physics residuals in the loss function. Non-adaptive weighting (Wight & Zhao, 2020) emphasises the importance of fulfilling the initial condition first by scaling the corresponding residual by a predefined factor. Learning rate annealing (Wang et al., 2021a) considers that finding these scaling factors might be challenging and uses gradient statistics during training to balance the different loss terms. Minimax weighting (Liu & Wang, 2021) uses a different balancing approach, updating the network weights via gradient descent and the residual weights via gradient ascent. The weighting scheme in selfadaptive PINNs (McClenny & Braga-Neto, 2020) is closely related to Minimax weighting but differs in scaling not the entire term by one scalar value but every collocation and data point individually. 
methodology: 
 Physics-informed neural networks: To solve a PDE with a PINN, we assume that the parameters of the PDE are fixed and known. We can then use a neural network to approximate the desired function and model the PDEs by differentiating the network output with respect to the inputs using automatic differentiation.The total loss, in our use-case, consists of an initial condition residual L ic and a physics residual L p , which can be defined as:EQUATIONwith:L ic (w, λ ic ) = 1 N ic Nic i=1 m(λ ic i )(y(t i , x i ) -ŷ(t i , x i ; w)) 2 ,andL p (w, λ p ) = 1 N p Np j=1 m(λ p j ) ∂ ŷ(t j , x j ; w) ∂t j + N [ŷ(t j , x j ; w); γ] 2 ,where N [•, γ] is a nonlinear operator parameterized by γ and y(t i , x i ) are known values of the PDEs solution at (t i , x i ). Depending on the studied problem, y(t i , x i ) can be obtained from known initial conditions, boundary conditions, or measured data. The predicted solution of the PINN with weights w at collocation points (t j , x j ), sampled over the whole domain, is given by ŷ(t j , x j ; w), and β is a scaling factor to balance the individual loss terms (Moseley et al., 2020) . The self-adaptation weights λ are an extension of the original PINN (McClenny & Braga-Neto, 2020) , and m(λ) = 1 for the baseline PINN. The residual L p enforces the PDE at the collocation points.PINNs with sinusoidal activation functions: Using a sine instead of or in combination with the more common tanh as an activation function was shown to improve the trainability of PINNs (Raissi et al., 2018) , (Huang et al., 2021) . Wong et al. (2022) provide insights into why this might be the case. The tanh function is known to saturate for large inputs, which can lead to vanishingly small gradients. Additionally, the more expressive the PINN gets, the more it suffers from an initial bias toward flat output functions due to a near-zero input gradient. Compared to traditional neural networks, this is especially problematic for PINNs because a zero output ŷ(t j , x j ; w), minimizes many PDEs. The PINN is trapped in a local minimum where the physics residual is low or even zero and might not be able to minimize the initial condition residual additionally. Another aspect is that sinusoidal patterns often occur in physical phenomena, such as mass-spring-damper systems, wave propagation, or the heat equation. Therefore, using a sinusoidal activation function is valuable for incorporating prior knowledge into the network. Using a sine as an activation function in the first layer as proposed in (Wong et al., 2022) leads to the input of the second layer being a weighted sum of sine functions, which can be used to approximate periodic waves. Naturally, the usage of sinusoidal activation functions can be extended to the subsequent layers (Sitzmann et al., 2020) .Sinusoidal activation functions should be especially suited for problems with oscillating solutions.Self-Adaptive PINNs (SA-PINNs): SA-PINNs (McClenny & Braga-Neto, 2020) were introduced to ease the challenge of balancing the individual components of the loss function. The residual weights are jointly optimized with L and do not need to be tuned by hand after the initialization.Since the weights are applied to each training point individually, the network can balance not only the loss terms as a whole but also the individual training samples. The network focuses on training samples with high weights, effectively concentrating on complex regions of L.To achieve this, the authors introduce the nonnegative and trainable self-adaptation weights λ alongside the strictly increasing, differentiable, nonnegative function m(λ) defined on [0, ∞). In the loss equations (1), λ ic i and λ p j are the self-adaptation weights for the initial condition and physics collocation points. The core optimization problem of SA-PINNs is to minimize the loss with respect to the network weights w but maximize it with respect to the self-adaptation weights λ (McClenny & Braga-Neto, 2020) .Training scheme: In our proposed training scheme, we combine several findings in PINN literature to improve performance, improve convergence speed, and reduce the required amount of hyperparameter tuning for specific problems. Our training scheme for physics-informed models for oscillating problems comprises two main parts: (i) Applications of sine activation functions in all layers of the model, to more closely approximate the fundamental PDE solution, as well as reducing vanishing gradient effects, and (ii) combining it with self-adaptive weights to overcome local minima during PINN training. 
experiments and discussion: 
 We will test the original PINN and the self-adaptive PINN with up to three different activation function configurations in the hidden layers: the usage of tanh in all layers, denoted by the token all-tanh, sine in the first layer and tanh in the subsequent layers denoted as sine-first, and sine in all layers denoted as all-sine (detailed configurations in Appendix A). The following set of differential equations describes the double mass-spring-damper system (see also Appendix A, Figure 3 ):EQUATIONIn the often-used quarter-car suspension model, d 1 and c 1 represent the damping and stiffness of the tyre, while d 2 and c 2 embody the primary suspension of the car. In the following experiments, we assume fixed PDE parameters and a fixed initial condition (detailed parameter settings can be found in Appendix A).Figure 1 shows the minimal obtained test MSE for every 100k of training iterations. It is apparent that using a sinusoidal activation function in all layers is the best-suited approach for this problem.Examining the performance of the all-tanh and sine-first networks (squares/triangles), we see the results are almost indistinguishable. The introduction of self-adaptive weights (red) improves the performance only marginally. The all-sine networks can distinguish themselves with a considerable performance gap. Notably, the PINN all-sine network outperforms two of the three self-adaptive networks. For this problem, the proper activation function configuration outweighs the loss-balancing benefits of the self-adaptive weights. However, the combination of both techniques converges not only the fastest but also to the lowest error overall.Predictions further away from the initial condition are more challenging for PINNs. Therefore, Figure 2 shows a cutout of the averaged predicted solution after 1e6 epochs at the end of the temporal domain. One second after the initial condition, networks without all-sine activation functions predict a flat output close to zero, here exemplified by SA-PINN sine-first prediction, indicating what is likely an attractive local minimum in the loss landscape and illustrating one of the challenges in training PINNs for oscillating problems. Even though the PINN all-sine configuration can capture the low-frequency features in the solution after 1 second for both masses, it misses the high-frequency features (see also Appendix A.2, Figure 4 ). In contrast, SA-PINN all-sine can approximate them reasonably well, encouraging the use of our proposed training scheme of combining sinusoidal activation functions in all hidden layers with self-adaptive weights. In Appendix A.4, we demonstrate that the benefits of employing multiple sinusoidal layers extend to deep neural operator networks by solving the dMSD model for various initial conditions without retraining. 
conclusion: 
 In this work, we propose a training scheme for PINNs tailored to oscillating problems. By combining several findings in PINN literature, we are able to improve performance and reduce training time.We show the advantages of applying sine activation functions in all layers of the model to more closely approximate the fundamental PDE solution, as well as reducing vanishing gradient effects. In combination with self-adaptive weights, this improves convergence speed during PINN training. Our proposed scheme outperforms all baseline models for the considered double mass-spring-damper model. Here the usage of sine activation functions in all, not only the first layer, provides the most benefits. The networks using all-tanh or sine-first activation functions do not even capture the low frequencies. The PINN all-sine network approximates the low frequencies well but misses the strongly oscillating parts of the solution. Only the SA-PINN all-sine is able to approximate both ends of the spectrum well, showcasing that employing a single layer with sine activation functions does not yield the desired benefits. 
 
 Appendix: The wave equation occurs in many fields, such as acoustics, electromagnetics, cosmology, and fluid dynamics. We consider the wave equation for a medium with constant density:EQUATIONwith the following initial condition: and boundary conditions:EQUATIONEQUATIONwhere v is the velocity of the medium, x 1 , x 2 represent the position, and t the time Moseley et al. (2020) .We train a network with four hidden layers of width 256 and apply min-max normalization to the input features. The training dataset incorporates 10,000 samples each for the initial and boundary conditions. To calculate L p , we use 50, 000 collocation points distributed evenly over the domain and set β = 5e -4.The results in Figure 6 might appear similar to the ones discussed above for the experiments with the double mass-spring-damper model. Once again, the SA-PINNs outperform the baseline PINN in the beginning. However, after about 8,000 epochs, the initially slower converging PINN all-sine surpasses it. Therefore, starting with self-adaptive weights and fixing them later in the training process combined with all-sine activation functions might be a viable approach for the wave equation.As shown by the results in (McClenny & Braga-Neto, 2020) , employing stochastic gradient training is another way of improving the accuracy of SA-PINNs for this particular problem.Physics-informed deep neural operator networks: Deep neural operator networks (DeepONets) (Lu et al., 2019) are designed to efficiently learn mappings from a space of functions to another space of functions (operators). Their architecture is based on a universal approximation theorem for operators (Chen & Chen, 1995) and features two sub-networks, a branch and a trunk network, whose outputs are combined via a dot product to produce the final result. The branch net encodes the input function. In our experiments with the dMSD system, the input functions will be different initial conditions. The second network, the trunk net, encodes the locations for the output functions. The locations are given by the input time in the dMSD example. Since the outputs of both networks are combined via a dot product, the framework needs to be slightly altered to deal with multiple outputs.In order to obtain outputs for m 1 and m 2 , we split the output neurons of the trunk and branch net in half and compute two dot products as suggested in (Lu et al., 2022) . To decrease the inputoutput pairs needed for training and to produce physically plausible predictions, physics-informed DeepONet were proposed (Wang et al., 2021c) . They combine the benefits of DeepONets and PINNs by incorporating an additional residual based on a PDE, as discussed in Section 3, into the loss function of a DeepONet. We will use physics-informed DeepONets in the following experiments but simply denote them as DeepONets. DeepONets are trained to solve the PDE over a predefined input space and only require retraining for out-of-domain samples.Experiments and discussion: Returning to the dMSD model discussed above, we want to assess if the advantages of all-sine activation functions persist when using multiple initial conditions. We define the temporal domain as t ∈ [0, 1] seconds and set c 2 = 1e7. We train the networks by sampling 900 initial conditions y 1 , y 2 ∈ [-1, 1] exempt two out-of-domain gaps defined on the intervals [-0.6, -0.4] and [0.4, 0.6]. To evaluate the test performance, we randomly sample 100 initial conditions from within the domain and, as a separate metric, 100 initial conditions from the out-of-domain gaps. To calculate L p , we use 5, 000 collocation points distributed evenly over the temporal domain. The branch net always features all-tanh activation functions, whereas we test different configurations of activation functions for the trunk net. Figure 5 illustrates the superior performance the DeepONet architecture enables, compared to PINNs for handling multiple initial conditions. Once again, employing a single layer of sine activation functions performs similarly to the original DeepONet all-tanh, but DeepONet all-sine converges considerably faster than the other configurations. Table 1 shows the MSE for the in-domain (ID) and out-of-domain (OOD) test set after training. In both metrics, DeepONet all-sine performs best. Since the MSE depends on the amplitude of the solution, which is influenced by the sampling intervals of the initial conditions, the ID MSE and OOD MSE are not directly comparable. The similar performance of the original all-tanh DeepONet and the sine-first configuration indicates that also in DeepONets, a single layer with sine activations is not enough to profit from the benefits of sinusoidal activation functions.In future work, we would like to explore if the benefits of self-adaptive weights in PINNs also translate to DeepONets. 


Decision: Dear Authors,

Thank you for submitting the draft.

Both reviewers agree that the presented results show some advantages and the draft is clearly written. However, both reviewers do also raise some major points of concern. It is expected that authors will be addressing comments by the reviewers in the final draft. 

regards

AC

Reviews: review: title: title: Interesting results but unsure about broader applicability
main_review: main_review: The authors propose using sine activation function for all hidden layers in PINNs. They use SA-PINN as their base model to showcase the proposed usage. The results show the advantage of using sine activation function. However, I have some concerns:

1. The authors should consider comparing with some more activation functions, lets say swish and mish - do they provide accuracy comparable to tanh case?

2. Is it because of the underlying harmonic problem that this fix works? or the authors consider it as widely applicable to other problems?

3. The authors can also combine adaptive activation function while using sine and see if it further improves the learning

4. The paragraphs have different spacing, kindly check that to maintain consistency.

I am happy with the results for acceptance in ICLR workshop because in a way the paper is using the known physical prior in case of a mass damper system to choose the activation function. The choice is intuitive, but is validated empirically and hence would be interesting for this workshop.
title: title: The findings of this work are promising for one case, but I am uncertain about the effectiveness of the method for canonical PDE problems
main_review: main_review: Quality: The quality of this work is satisfactory.

Clarity: Very well-written and easy to follow regarding their aim.

Originality: This framework is somewhat novel, although not entirely unprecedented.

Significance: The findings of this work are promising for one case, but I am uncertain about the effectiveness of your method for canonical PDE problems