RBF-PINN: NON-FOURIER POSITIONAL EMBEDDING IN PHYSICS-INFORMED NEURAL NETWORKS

Authors: chengxi  zeng
tilo  burghardt
alberto m gambaruto
Venue: ICLR_cc_2024_Workshop_AI4DiffEqtnsInSci
Published: 2024-02-09 15:57:39

Abstract: While many recent Physics-Informed Neural Networks (PINNs) variants have had considerable success in solving Partial Differential Equations, the empirical benefits of feature mapping drawn from the broader Neural Representations research have been largely overlooked. We highlight the limitations of widely used Fourier-based feature mapping in certain situations and suggest the use of the conditionally positive definite Radial Basis Function. The empirical findings demonstrate the effectiveness of our approach across a variety of forward and inverse problem cases. Our method can be seamlessly integrated into coordinate-based input neural networks and contribute to the wider field of PINNs research.

Main paper: 
 introduction: 
 Many scientific phenomena can be described by sets of differential equations (DEs). The prior physics knowledge is then formulated as regularisers that can be embedded in modern machine learning algorithms supported by the Universal Approximation Theorem Hornik et al. (1989) . Physics-Informed Machine Learning (PIML) Karniadakis et al. ( 2021) is a learning paradigm that combines data-driven models with physical laws and domain knowledge to solve complex problems in science and engineering. It has recently attained remarkable achievements in a wide range of scientific research such as Electronics Smith et al. (2022) ; Hu et al. (2023) ; Nicoli et al. (2023) , Dynamical System Thangamuthu et al. (2022) ; Ni & Qureshi (2023) , Meteorology Kashinath et al. (2021) ; Giladi et al. (2021) and Medical Image Goyeneche et al. (2023) ; Salehi & Giannacopoulos (2022) ; Pokkunuru et al. (2023) . One of the leading methods is called Physics-Informed Neural Networks (PINNs) Raissi et al. (2019) . By adding the DEs as penalty terms in the deep Neural Networks (NN), it exploits the differentiability of NN to compute derivatives of the explicit functions and introduces domain-specific regularisation during optimisation. Adhering to conventional solvers, the PINNs formulation necessitates the specification of initial/boundary conditions (IC/BC) within a confined spatial-temporal domain. Boundary sampling points and domain collocation points are used to evaluate the residuals of the conditions and DEs via an overparameterised NN. The objective is to optimise the NN by minimising the residuals, resulting in a converged parameter space. This parameter space can then serve as a surrogate model that accurately represents the solution space of the DEs. The PINNs can be formulated as follows:D[u(x, t; α i )] = F (x, t), t ∈ T [0, T ], ∀x ∈ Ω and B[u(x, t)] = H(x, t), t ∈ T [0, T ], x ∈ ∂Ω.(1) where D [•] is the differential operator and B[•] is the boundary operator, x and t are the independent variables in spatial and temporal domains Ω and T , respectively. The α i are coefficients of the DE system and remain wholly or partially unknown in Inverse Problems. For time-dependent PDEs, the initial condition can be treated as a special type of BC. F and H are arbitrary functions. PINNs are parameterised by θ, the solution space represented can give numerical solution ûθ at any x and t within the domain. And the training loss functions are defined as follows:L(θ; X (x,t) ) = λ r N r Nr i=1 D[û θ (x i r )] -F (x i r ) 2 + λ bc N bc N bc i=1 B[û θ (x i bc )] -H(x i bc ) 2 (2)where {x i r } Nr i=1 and {x i bc } N bc i=1 are domain collocation points and boundary points, they are evaluated by computing the mean squared error. λ r and λ bc are the corresponding weights of each term. 2022) recognising its potential within PINNs. Originally introduced in Natural Language Processing (NLP), feature mapping aims to map the input to a higher-dimensional feature space for a better neural representation. It was subsequently identified as an effective approach for mitigating spectral bias in visual representations Tancik et al. (2020) . Feature mapping is a broader term for positional encoding that can involve either fixed encoding or trainable embedding. This simple one-layer projection can map the spatio-temporal input x to a higher dimension feature space, Φ : x ∈ R n → R m , and typically n ≪ m.In PINNs, Wang et al. (2020) has leveraged the 'Neural Tangent Kernel' (NTK) theory that reveals PINNs suffer from 'Spectral Bias' in the infinite-width limit. The NTK exhibits sensitivity to both input and network parameters. Its trait is particularly contingent on factors such as the input gradient of the model, the variance at each layer, and the nonlinear activations. Hence, the training dynamics of PINNs are significantly influenced by the input before parameterised layers.Our contribution in this paper can be summarised as: First, we show the limitations and shortcomings of the widely used Fourier-based feature mappings in some Partial Differential Equations (PDEs) and thoroughly benchmark a wide range of feature mapping methods, including some which have not been employed in PINNs before. Secondly, we present a framework for designing feature mapping functions and introduce a conditional positive definite Radial Basis Function. This method surpasses Fourier-based feature mapping in various forward and inverse tasks. 
limitation of fourier features: 
 Basic Fourier features can lead to undesired artifacts, as illustrated in Figure 1 (a) (detailed equations can be found in Appendix G). When applying Positional Encoding to address Burgers' equations, a notable prediction error is observed in the area approaching a jump solution. This phenomenon is akin to the Gibbs phenomenon in signal processing, where the approximated function value by a finite number of terms in its Fourier series tends to overshoot and oscillate around a discontinuity. Another unexpected experimental result that shows poor performance is observed when utilising Random Fourier Features Tancik et al. (2020) in high-dimensional problems, as depicted in Figure 1(b) . Two cases are set up, one case with each dimension of evenly sampled points and another with uneven sampling points, where the number of sampling points x r is set as 1 D . The latter example resembles the unsteady Navier-Stokes equations for fluid dynamics, which have dense samples in the spatial domains, but sparse sampling in the temporal dimension. Despite hyperparameter tuning for the Fourier feature, including the arbitrary scale σ and the number of Fourier features, none of the attempts reduce the elevated error in high-dimensional cases. The experiments are repeated with 3 random seeds, and standard deviations are displayed in the highlighted area. 
proposed method: 
 In NTK theory, the multi-layer perceptions (MLPs) function is approximated by the convolution of the stationary composed NTK function K COMP = K N T K • K Φ with weighted Dirac delta over the input x (background in Spectral Bias and Composed NTK are in Appendix C), we can formulate the K COMP by:EQUATIONThe accuracy of the continuous approximation can be analysed by Taylor series expansion:K COMP (x) = (K COMP (x) + ∇ x K COMP (x -x ′ ) + 1 2 (x -x ′ )∇ 2 K COMP (x -x ′ ) + O((x -x ′ ) 3 ))K Φ (x -x ′ )dx = K COMP (x) K Φ (x -x ′ )dx + ∇ x K COMP (x -x ′ ) (x -x ′ )K Φ (x -x ′ )dx + O((x -x ′ ) 2 ) (4)Ensuring the first-order accuracy of the composing kernel requires the term K Φ (x -x ′ )dx = 1, and the second term in Equation 4 must be 0. This can be accomplished by normalising the feature mapping function and ensuring that it satisfies a symmetry condition. We propose a positive definite Radial Basis Function (RBF) for such a kernel, and its formulation is given by:EQUATIONwhere x ∈ R n is the input data, c ∈ R n×m are the centres of the RBFs and are trainable parameters and w is the weight matrix for the feature mapping layer. A natural choice for the RBF can be the Gaussian function,φ(x) = e -|x-c| 2 σ 2, where σ is a random initialised trainable parameter. If we choose the same number of features as the input size (i.e. n = m), this method provides an approximate computation of the desired function value through kernel regression. Unfortunately, in the context of PINNs, the training input size is typically large, making it impractical to scale in this manner. Through empirical study, we demonstrate that a few hundred RBFs prove sufficient to outperform other types of feature mapping functions. During initialisation, c is sampled from a standard Gaussian distribution.In the infinite-width limit, each layer of the Neural Network is treated as a linear system. To guarantee a unique solution, one approach involves introducing conditionally positive definite radial functions by incorporating polynomial terms. The weights serve as Lagrange multipliers, enabling constraints on the RBF coefficients in the parameter space Farazandeh & Mirzaei (2021) . We denote this method as RBF-P throughout the paper. Therefore, the feature mapping function is adjusted to:EQUATIONWhere P is the polynomial function. In the feature mapping layer, it can be represented as:EQUATIONwhere r = x -c and P is the order of the polynomial term.Based on empirical findings, we observe that the polynomial term is highly effective in nonlinear function approximation, particularly in equations like the Burgers Equation and Navier-Stokes Equation, all while incurring minimal computational overhead. By this principle, we can use many other types of RBF without too many restrictions. Other types of RBF are shown in Appendix E Table 5 . I-Burgers 2.39e-2 2.43e-2 1.74e-2 1.57e-2 I-Lorenz 6.51e-3 6.39e-3 6.08e-3 5.99e-3 I-Burgers* 2.50e-2 2.91e-2 1.99e-2 1.75e-2 I-Lorenz* 7.93e-3 6.85e-3 6.69e-3 6.34e-3 
empirical results: 
 Time-dependent PDEs. Our solution in the Diffusion equation demonstrates superior performance compared to other methods by an order of magnitude. Boundary errors are notably more perceptible in Fourier-based methods, as illustrated in Appendix H, Figure 7 .The RBFs demonstrate enhanced capability in addressing multiscale problems, as illustrated in the Heat equation G.3. In the Heat equation formulation, there exists a substantial contrast in coefficients: 1 500π 2 for the x-direction and 1 π 2 for the y-direction. Figure 8 illustrates that the RBF method effectively preserves the details of the solution at each time step. Non-linear PDEs. We assess the methods using two classic non-linear PDEs: the Burgers equation and the Navier-Stokes equation. Figure 10 illustrates that RBFs with polynomial terms are better in addressing the discontinuity at x = 0 in the Burgers equation. Inverse Problems. A major application of the PINNs is able to solve Inverse Problems. The unknown coefficients in the differential equations can be discovered by a small amount of data points. our methods have shown their efficacy in two Inverse Problems, shown in 2 Another experiment aimed to test the robustness of feature mapping functions to noise. 1% Gaussian noises are added to the inverse Burgers problem and 0.5% to the Lorenze system data. The results presented in Table 2 reveal that the four tested feature mapping methods indicate a degree of immunity to noise. Furthermore, RBF-P stands out as the most resilient feature mapping function to noise. All benchmarked method can be found in Appendix B.We carried out ablation studies on the number of RBFs in the feature mapping layer, the number of polynomials for RBF-P and different types of RBFs. Generally, a higher number of RBFs perform better but requires high computation resources. For different cases, the number of polynomials terms required varies. And among all test RBF functions, Gaussians present more stable results. The complete results can be found in Appendix E. The complexity and scalability of different feature mapping functions are included in Appendix F. 
conclusion and future work: 
 In conclusion, we have introduced a framework for designing an effective feature mapping function in PINNs and proposed Radial Basis Function-based approaches. Our method not only enhances generalisation across a range of forward and inverse physics problems but also surpasses other feature mapping methods by a substantial margin. The RBF feature mapping has the potential to be compatible with various other PINNs techniques, including novel activation functions and loss functions or training strategies such as curriculum training or causal training. While the primary focus of this work has been on solving Partial Differential Equations, the exploration of RBF feature mapping extends to its application in other coordinate-based input neural networks for different tasks. 
 
 Appendix: Coordinate Sampling. As a mesh-free method, PINNs are normally evaluated on scattered collocation points both on the interior domain and IC/BC. Therefore, the sampling strategy is crucial to PINNs' performance and efficiency. A poorly distributed initial sampling can lead to the PDE system being ill-conditioned and NN training instability. The whole design of experiments on the fixed input sampling is reviewed by Das & Tesfamariam (2022) . Based on the study of uniform sampling, Wu et al. (2023) 2023) are proven to be effective alternatives. Positional Embedding. Broadly speaking, PINNs can also be considered as a special type of Neural Fields Xie et al. (2021) in visual computing, which specifically feed coordinate-based input to MLPs that represent continuous field quantity (e.g. velocity field in fluid mechanics) over arbitrary spatial and temporal resolution. However, the PINNs community often ignores the fact both perspectives function the same way as Implicit Neural Representations. In the Neural Field, images and 3D shapes are naturally high-frequency signals, whereas deep networks are inherently learning towards the low-frequency components Rahaman et al. (2018) . Feature mapping hence has become a standard process in practice that maps the low-dimension coordinates to high-dimension space. The pioneering work was conducted by Rahimi & Recht (2007) , who used Fourier features to approximate any stationary kernel principled by Bochner's theorem. the derivative works are done such as Positional Encoding Mildenhall et al. (2020 ), Random Feature Tancik et al. (2020) and Sinusoidal Feature Sitzmann et al. (2020) . Another concurrent work discusses non-periodic feature mapping Zheng et al. (2022) ; Ramasinghe & Lucey (2021) ; Wang et al. (2021a) . To the best of our knowledge, feature mapping in PINNs has been largely uninvestigated. Only a few work preliminarily adopted Fourier-feature-based methods in PINN Wang et al. (2021b; 2023) ; Wong et al. (2022) . 2022) φ(x) = [max(1 -|x1-t| 0.5d , 0), max(1 - |x2-t| 0.5d , 0), • • • , max(1 -|xi-t| 0.5d , 0)] T, where t is uniformly sampled from 0 to 1. Complex Gaussian: Zheng et al. (2022) φ(x) = [e -0.5(x1-τ /d) 2 /σ 2 • • • e -0.5(x d -τ /d) 2 /σ 2 ]T , where τ is uniformly sampled from [0, 1], and is the Kronecker product.C.1 SPECTRAL BIAS Normally PINNs are setup as a standard MLP model f (x; θ), and θ is optimized on the loss functionL(θ) = 1 2 |f (x; θ) -Y | 2 = 1 2 N i (f (x i ; θ) -y i ) 2, where X, Y and θ are training input, training ground truth and model parameters. For an easier formulation, we replace the conventional gradient descent formulation θ t+1 = θ t -α∇ θ L(θ t ) to a gradient flow equation:EQUATIONwhere α should be an infinitesimally small learning rate in the NTK setting. Given PDE collocation data points{x i r , D(û θ (x i r ))} Nr i=1 , and boundary training points{x i bc , B(û θ (x i bc ))} N bc i=1 . The gradient flow can be formulated as Wang et al. (2020) :EQUATIONwhere the Kernels K are:EQUATIONSince K remains stationary, then K t ≈ K 0 as NN width tends to infinity, Equation 9 is rewritten as:EQUATIONBy Schur product theorem, K 0 is always Positive Semi-definite, hence it can be Eigen-decomposed to Q T ΛQ, where Q is an orthogonal matrix and Λ is a diagonal matrix with eigenvalues λ i in the entries. We can rearrange the training error in the form of:EQUATIONwheree -Λt =    e -λ1t . . . e -λ N t   .This indicates the decrease of training error in each component is exponentially proportional to the eigenvalues of the deterministic NTK, and the NN is inherently biased to learn along larger eigenvalues entries of the K 0 .The Fourier feature layer is defined as:φ(x) = a 1 cos 2πb T 1 x , a 1 sin 2πb T 1 x , . . . , a m cos 2πb T m x , a m sin 2πb T m x T (13)Hence the NTK is computed by:EQUATION)where A is the Fourier Series coefficients, b is randomly sampled from N (0, σ 2 ) and σ is an arbitrary hyperparameter that controls the bandwidth. Thereafter, the feature space becomes the input of the NTK which gives the identities: The Figure 4 has shown Gaussian RBF is dominating all types of PDEs. However other types of RBF are in similar performance. We generally prefer Gaussian RBF in all cases due to its nice properties.K N T K (x T i x j ) = K N T K (φ(x i ) T φ(x j )) = K N T K (K Φ (x i - x j )).The comparison of complexity and scalability of feature mapping methods are shown in this section.Although all feature mapping methods are similar in computational complexity, for completeness, we include the complexity of the feature layers that map 128 features and 4 fully connected layers with 50 neurons each. EQUATIONThe analytical solution of the equation is:u(x, t) = sin(πx)cos(2πt) + 1 2 sin(4πx)cos(8πt).  



Reviews: review: title: title: Official review of Submission39
main_review: main_review: This paper investigates the benefits of feature mapping in physics-informed neural networks. The authors show that Fourier-based feature mapping has certain limitations, and conditionally positive definite radial basis functions provide better results for solving partial differential equations using PINNs. Different PDEs are investigated in this work for both forward and inverse problems, and overall results with RBF-based feature mapping are better compared to other methods. However, the motivation behind using feature mapping in PINNs and how it exactly addresses the limitations of PINNs are missing. Addressing this would enhance the credibility of the proposed approach. There are several critical points that require attention to strengthen the paper's contribution and clarity.

1. The error for PINNs and the proposed RBF is similar for even and uneven sampling for nD Poisson equations. What is the justification for using RBF for feature mapping for this problem?
2. MLP on page 2 is not defined.
3. There is no visual comparison of solutions from PINNs in Figs 6-11.
4. The number of RBFs does not seem to have a significant impact on L2 error, and in the case of Burger’s equation, the error seems to increase as the number of RBFs is increased. Is it due to overfitting? Similar observations for the number of polynomials are also noted.
5. In the conclusion, the authors mentioned that this paper introduces a framework. However, the present paper does not discuss how to decide the feature mapping depending on the complexity of the problem. Hence, we cannot call the present study on feature mapping a framework.
6. In the conclusion section, the author mentioned that the RBF feature mapping enhances generalization across forward and inverse problems. However, the paper does not discuss anything about the errors for out-of-distribution data.
7. Detailed experimental settings, specific implementations for each numerical experiment, and hyperparameters are not provided. The code is also not made open-source, making it difficult to reproduce the results.
title: title: Feature Mapping Scheme for PINNs
main_review: main_review: The authors propose a feature mapping scheme termed RBF-PINN that uses radial basis functions to map/transform the inputs of a PINN network. They apply this to six cases for solving the forwards problem and four cases for solving the inverse problem. Their approach is contrasted with the original PINNs approach and several other feature mapping approaches.

I believe the authors have done a lot of work and the paper for the most part is clear to the reader. 

I am familiar with most of the discussed concepts in the paper (e.g. PINNs and feature mapping) but I have followed only to some extend the literature on PINNs to make strong claims regarding the novelty of the work. One of the things I would expect is to have a comparison with the work ``Adaptive activation functions accelerate convergence in deep and physics-informed neural networks’’ but since it is mentioned that this future step I don’t think is necessary here.

My main comments for the paper are:

(i)	Even though the paper is well written there are some cases the notation becomes confusing in my opinion. The author start by using the usual notation u(x,t) for the (time-dependent) PDEs and in equation (2) x_r for the collocation points. However, when they introduce Φ it seems to depend only on x (or bold x). This in my opinion might create some confusion. 
(ii)	Equation 5, a definition of $w_i$ is missing.
(iii)	Regarding the conditionally positive definite RBF. The authors introduce polynomial terms, and they claim this guarantees uniqueness. I would like to see a few references there that support this claim.
(iv)	Can the authors comment if there cases for which the introduction of the polynomial terms can lead to “explosions” of the gradient or to vanishing gradients? For example, when the degree of polynomial is large.
(v)	How the authors chose the parameter σ in the kernel? 
(vi)	In a few places in the text the authors mention the $l2$ loss where the corresponding figures shows $L_2$. Make things consistent. (There are figures also in the Appendix with this inconsistency).
(vii)	Figure 19 it’s not clearly readable. Perhaps the authors can make some changes there.
(viii)	Section Limitation of Fourier Features “Navier-Stokes” is misspelled.
(ix)	Perhaps, for future work, it might be of interest to the authors instead of sampling the coefficients c randomly to use approaches such the “Sampling weights of deep neural networks)”.