Approximating Family of Steep Traveling Wave Solutions to Fisher's Equation with PINNs

Authors: franz m rohrhofer
bernhard c geiger
stefan  posch
clemens  gößnitzer
Venue: ICLR_cc_2024_Workshop_AI4DiffEqtnsInSci
Published: 2024-02-07 15:25:05

Abstract: In this paper, we adapt Physics-Informed Neural Networks (PINNs) to solve Fisher's equation with solutions characterized by steep traveling wave fronts.
We introduce a residual weighting scheme that is based on the underlying reaction dynamics and helps in tracking the propagating wave fronts.
Furthermore, we explore a network architecture tailored for solutions in the form of traveling waves. 
Lastly, we assess the capacity of PINNs to approximate an entire family of traveling wave solutions by incorporating the reaction rate coefficient as an additional input to the network architecture.

Main paper: 
 introduction: 
 Reaction-diffusion systems constitute a wide class of mathematical models used in biology, physics, chemistry, ecology, and engineering. Fisher's equation, a simple yet profound reaction-diffusion system, was introduced in (Fisher, 1937) and readsEQUATIONwhere u is the concentration of a chemical substance, and ρu(1u) ≡ R(u; ρ) is a reaction term, parameterized by the reaction rate coefficient ρ. This equation admits traveling wave solutions of the form u(x, t) = u(x ± ct) ≡ u(z) for every wave speed c ≥ 2 √ ρ. For values ρ ≫ 1, the solution exhibits a sharp and fast traveling wave front which demands fine spatial and temporal resolutions when studied numerically (Zhao & Wei, 2003) .Physics-informed neural networks (PINNs) have emerged as a deep learning method that is applicable to physical systems such as described by Eq. ( 1) (Raissi et al., 2019; Pan et al., 2023) . The utility of PINNs in solving differential equations is still under investigation, but their flexibility and the fact that they do not need discretization has already shown great potential for solving all kinds of problems involving differential equations (Raissi et al., 2020) . Training PINNs -i.e., obtaining the neural approximation u θ of the solution function u in Eq. ( 1) by minimizing residuals on collocation points -however, is not straightforward and demands a carefully chosen optimization procedure.In this work, we apply PINNs to approximate solutions to Fisher's equation with large reaction rate coefficients ρ. We observe that standard PINNs struggle in approximating the resulting sharp solutions, and introduce a novel residual weighting method as a remedy. Additionally, we assess the effectiveness of a specific network architecture, designed to automatically adapt to the shape of traveling wave fronts. Finally, we investigate the generalization capability of PINNs by employing a single PINN to approximate a family of traveling wave solutions parameterized by the reaction rate coefficient ρ, where said coefficient is an additional input to the PINN.Figure 1 : (Left) Wave layer structure compelling the network function to u(z) with z(x, t) = θ 1 x + θ 2 t + θ 3 for the discrete-ρ approximation, or z(x, t; ρ) = θ 1 ρ 1 x + θ 2 ρ 2 t + θ 3 for the continuous-ρ approximation. (Right) Reference and predicted wave front for ρ = 10,000. 
methodology: 
 Steep Traveling Wave Solutions. For Eq. ( 1), the traveling wave solution with a constant wave speed of c = 5 ρ/6 is of particular interest, as it admits the analytical solution (Ablowitz & Zeppetella, 1979) :EQUATIONthat satisfies lim z→-∞ u(z) = 0 and lim z→∞ u(z) = 1. Although Eq. (1) and Eq. ( 2) can be brought into a nondimensionalized form, e.g. by using x ← √ ρx and t ← ρt, the system is numerically studied often with the dependency on the reaction rate coefficient ρ. This is due to the fact that for a fixed computational domain, the reaction term, and thus the steepness of the traveling wave front, can be made arbitrarily large by adjusting ρ. Typical values fall into the range ρ ∈ (1,000, 10,000) and these settings commonly serve as a numerical challenge for discretization schemes (Li et al., 1998; Qiu & Sloan, 1998; Kırlı & Irk, 2023) .Residual Weighting. PINNs are known to suffer from optimization issues when applied to problems with sharp and shock-like solutions (Liu et al., 2024; Mao & Meng, 2023) . Therefore, with u θ being the neural approximation corresponding to network parameters θ, we propose a residual weighting scheme that is based on the reaction dynamics R and assigns the weight 3) in (Raissi et al., 2019) ). Intuitively, for λ > 0, the weights ω reduce the influence of steep gradients in regions with predominant reaction dynamics, while leaving residuals at predominant diffusion dynamics unaffected. Our approch is thus similar in spirit to (Liu et al., 2024) and in contrast to approaches that place additional residual points in regions of high residuals with (Mao & Meng, 2023) or that weight collocation points with high residuals more strongly (McClenny & Braga-Neto, 2023) .ω = 1 λ|R(u θ ; ρ)| + 1 (3) to each residual f in the physics loss function L(θ) ∼ i |ω i f i | 2 (cf. Eq. (Standard-& Wave-Architecture. In the standard architecture, hidden layers directly follow the input layer, aiming to approximate the solution function in the general spatial-temporal form u(x, t). Conversely, in the wave architecture, an additional wave layer is introduced that directly follows the input layer and precedes the hidden layers. The wave layer introduces a latent variable representation (cf. Figure 1 ) to compel the network function to solutions that take the traveling wave form u(z). This approach is similar in spirit to the Lagrangian PINN (Mojgani et al., 2023) and the characteristics-informed neural network (Braga-Neto, 2022), which both adapt the network architecture to the solution family of the respective (class of) PDEs.  
results: 
 For our experiments, we consider the following two applications: the discrete-ρ approximation (i.e., approximating u(x, t) for a single ρ), and the continuous-ρ approximation (i.e., approximating an entire family of solutions u(x, t; ρ)) by applying a generalizing architecture that takes ρ as an additional input. For each application, we assess the L 2 -error of four specific models: (i) standard-ANN and (ii) wave-ANN, which are purely data-driven models trained on the solution function (2), by sampling labeled data from it anew at each epoch; and (iii) standard-PINN and (iv) wave-PINN, which infer the solution function by minimizing the physics loss and a data loss that encodes the initial and boundary conditions. 1We consider x ∈ [-5, 5] as the spatial and t ∈ [0, 0.004] as the temporal domain and vary reaction rate coefficients in the range ρ = [100, 10,000]. In the standard architecture, all input variables are scaled to [0, 1]. Conversely, in the wave architecture, only the temporal variable is scaled to [0, 1] to ensure reduced sensitivity of the θ j weights in the wave layer. The reaction rate input ρ is further transformed to ρ 1 ← √ ρ and ρ 2 = ρ (cf. Figure 1 ).Datasets of size N = 1024 are sampled anew at each training epoch using Latin hypercube sampling; the test set is taken with N = 10, 000. The PINN for a single value of ρ and the generalizing architecture have two and three hidden layers, respectively, with 20 neurons each. We use the hyperbolic tangent (tanh) as hidden layer activation and the Sigmoid activation as output layer activation to ensure that any prediction lies in the bounded interval u θ ∈ [0, 1]. Optimization of all models is carried out with Adam for 50k training epochs for the discrete-ρ approximation and for 100k epochs when applying the generalizing architecture. A default learning rate of 0.001 is applied.For the discrete-ρ experiment, the model performance is evaluated by approximating the solution function for three individual reaction rate coefficients ρ ∈ {100, 1,000, 10,000} while applying separate models for each case. The result can be found in Table 1 which shows the mean (and standard deviations) of the L 2 -error over ten independently trained model instances.As evident from the table, the wave-PINN with λ = 1 achieves the best results, even outperforming both data-driven models across all ρ values. This is particularly intriguing, as the data-driven ANNs have effectively (and without overfitting) learned the solution from labeled data, making them highly accurate for the given network size. Furthermore, the right panel of Figure 1 demonstrates that the wave-PINN with λ = 1 accurately captures the steep wave front at ρ = 10,000.By comparing the standard and wave architecture, we observe that the wave models generally outperform the standard models, with a notable discrepancy observed within the PINN models. Across nearly all residual settings (λ), the wave-PINN consistently achieves lower prediction errors compared to the standard-PINN, while the difference is less pronounced for the data-driven ANNs.Regarding the optimal residual weighting, we find that a value of λ = 1 is effective for the wave-PINN, whereas for the standard-PINN, the optimal choice is less evident. Nevertheless, the results clearly indicate that conventional, unweighted (λ = 0) PINNs exhibit significant performance issues, particularly for large ρ values. Across all values of ρ, employing a residual weight with λ > 0 yields considerably better performance compared to λ = 0.For the continuous-ρ experiment, we utilize labeled data sampled from the analytical solution for ρ min = 100 and ρ max = 10,000 and aimed for obtaining a reliable approximation within the continuous range ρ ∈ [ρ min , ρ max ]. While collocation points for the PINN instances are sampled within this range, the data-driven models are exclusively trained on labeled data corresponding to ρ min and ρ max . The results for this interpolation task are depicted in Figure 2 which shows the median (and 25%-and 75%-quantiles) of the L 2 -error across ten independently trained model instances.Consistent with the results from the previous experiment, the wave-PINN achieves the best performance across the continuous ρ domain. However, we also observe remarkable performance from the wave-ANN, considering that it is trained solely on the solution at the two specific ρ values (indicated by dashed black lines in Figure 2 ). The inclusion of the wave layer appears to offer a highly effective inductive bias that facilitates finding solutions over a continuous parameterization. Notably, even within the range of ρ = 100,000, both wave models continue to yield strong approximations in the extrapolation regime, with only a slight degradation compared to the interpolation regime.Overall, we see that the PINN models outperform their data-driven counterparts and that the wave architectures outperform the standard architectures. Apparently, the inductive bias induced by the wave layer is stronger than the learning bias induced by the physics loss function. This is expected, as the wave layer, especially in the generalizing architecture, incorporates explicit knowledge about traveling wave nature of the solution function. 
conclusion: 
 In this paper, PINNs were applied to solve Fisher's equation with large reaction rate coefficients and sharp solutions representing steep traveling wave fronts. To address the challenges posed by sharp transitions in traveling waves, a residual weighting scheme was introduced and integrated into the optimization process of PINNs. The residual weighting scheme is based on the underlying reaction term and effectively facilitates optimization in sharp regions of the solution function. Additionally, a specific network architecture designed to approximate traveling waves was tested, and the results demonstrated outstanding improvements compared to conventional network architectures. Finally, a generalized PINN architecture was explored that incorporates the reaction rate coefficient as an additional input, enabling the approximation of an entire family of solutions to Fisher's equation. The results showed that the PINN with the introduced modifications can effectively solve Fisher's equation with large reaction rate coefficients and, furthermore, can be extended to approximate a family of sharp solutions with a single PINN instance. Furthermore, our modifications complement and are compatible with other approaches to improving PINN training, such as methods for collocation point sampling (Mao & Meng, 2023)  
 
 Appendix: Initial and boundary conditions are imposed by sampling labeled data from the analytical solution at the computational boundary. 


Decision: I would like to thank reviewers for their careful review. Both reviewers seem to agree on accepting the paper. I also went through it and the study of how to enhance PINNs model for capturing sharp moving interfaces (shack) is quite interesting and valuable. I just ask authors to clearly address reviewers' comments in their final paper.  Also, I would like to ask authors to review their references and make sure they cite related papers, I have seen more papers on wave propagation using PINNs, please review and acknowledge those works too.

Reviews: review: title: title: The authors demonstrate the use of a weighting scheme to balance physics loss term and data loss. They also propose a wave layer that approximates wave form u(z) to guide the neural network for capturing the structure of wave. The proposed solution significantly outperform neural networks without standard ANN and PINN, contributing to solve the target PDE when solution is characterized with steep wave fronts.
main_review: main_review: The paper is well written, particularly in its coverage of background, motivation, and experimental details. The experiments are thorough and well-detailed. However, it is unclear in the explanation of the entire loss function, which is not explicitly outlined.

While the idea of weighting and/or adjusting the residual term is not novel, the proposed approach shows its simplicity and effectiveness. However, it remains unclear whether the proposed method outperforms the effectiveness of the approach discussed in the referenced paper by Liu et al. (2024), which introduced gradient-dependent weights to capture solutions of PDEs. A clearer comparison with existing methods would strengthen the originality aspect of the paper.

The promising results observed in both discrete and continuous experiments highlight the potential effectiveness of the proposed solution. Particularly the proposed weight scheme is effective especially in scenarios where the reaction rate coefficient is smaller. However, the authors fall short in providing a benchmarking case that demonstrates the importance of solving the addressed problem using the proposed solution compared to other existing PINN approaches. Clarifying the extent to which the weighting scheme enhances PINN capabilities would help the significance of the contribution.
title: title: Approximating Family of Steep Traveling Wave Solutions to Fisher's Equation with PINNs
main_review: main_review: ## General comments
The manuscript titled 'Approximating Family of Steep Traveling Wave Solutions to
Fisher’s Equation with PINNs' presents an innovative approach using Physics-
Informed Neural Networks to model steep traveling wave solutions in Fisher’s
equation. The proposed method, including a novel residual weighting technique
and a specialized network architecture, demonstrates improved accuracy in
approximating sharp solution profiles.
The findings are promising. I therefore recommend that the paper be published but
after the authors address the comments presented below, which are mostly minor.

## Comments
1. Here the efficacy of the proposed method is compared with traditional PINN
only. There are multiple variants of PINN [1,2,3,4, etc.]. The authors are
encouraged to spend a few words on better contextualizing their work
within the broader context of the current body of literature.
2. The values of the reaction rate coefficients are written as 1.000, 10.000. I am
not sure if they mean 1 and 10. However, if the authors mean 1,000 or 10,000,
those should be revised.
3. “Optimization is carried out with Adam for 50k training epochs for the
discrete-ρ approximation and for 100k epochs when applying the
generalizing architecture. A default learning rate of 0.001 is applied.”
Authors should clarify if these are true for both PINN and wave-PINN ?
4. The authors are advised to add a brief discussion on the limitations of the
method and the NN architecture.

## References:
[1] L. D. McClenny and Ulisses Braga-Neto, “Self-adaptive physics-informed neural
networks,” Journal of Computational Physics, vol. 474, pp. 111722–111722, Feb. 2023.
[2] U. Braga-Neto, “Characteristics-Informed Neural Networks for Forward and Inverse Hyperbolic
Problems,” arXiv.org, 2022.
[3] R. Mojgani, M. Balajewicz, and P. Hassanzadeh, Lagrangian pinns: A causality-conforming
solution to failure modes of physics-informed neural networks, arXiv preprint arXiv:2205.02902,
2022.
[4] A. D. Jagtap and G. E. Karniadakis, Extended physics-informed neural networks (xpinns): A
generalized space-time domain decomposition based deep learning framework for nonlinear partial
differential equations, Communications in Computational Physics, vol. 28, no. 5, pp. 20022041, 2020.