PINNsFormer: A Transformer-Based Framework For Physics-Informed Neural Networks
Authors: Zhiyuan Zhao
Xueying Ding
B. Aditya Prakash
Venue: ICLR 2024 poster (2024)
Keywords: Physics-Informed Neural Networks
Transformer
Self-Attention
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)

Abstract: Physics-Informed Neural Networks (PINNs) have emerged as a promising deep learning framework for approximating numerical solutions to partial differential equations (PDEs). However, conventional PINNs, relying on multilayer perceptrons (MLP), neglect the crucial temporal dependencies inherent in practical physics systems and thus fail to propagate the initial condition constraints globally and accurately capture the true solutions under various scenarios. In this paper, we introduce a novel Transformer-based framework, termed PINNsFormer, designed to address this limitation. PINNsFormer can accurately approximate PDE solutions by utilizing multi-head attention mechanisms to capture temporal dependencies. PINNsFormer transforms point-wise inputs into pseudo sequences and replaces point-wise PINNs loss with a sequential loss. Additionally, it incorporates a novel activation function, \texttt{Wavelet}, which anticipates Fourier decomposition through deep neural networks. Empirical results demonstrate that PINNsFormer achieves superior generalization ability and accuracy across various scenarios, including PINNs failure modes and high-dimensional PDEs. Moreover, PINNsFormer offers flexibility in integrating existing learning schemes for PINNs, further enhancing its performance.

Published as a conference paper at ICLR 2024
PINNSFORMER: A TRANSFORMER-BASER FRAME- WORK FOR PHYSICS-INFORMED NEURAL NETWORKS
Zhiyuan Zhao
Xueying Ding
Georgia Institute of Technology Atlanta, GA 30332 leozhaol 997 99w.ggttcc..................... edu
Carnegie Mellon University Piusburgh, PA 15213
xding2@nddrew cmu, .anuu.dde edu
B. Aditya Prakash
Georgia Institute of Technology Atlanta, GA 30332 badityapece. htt.ttapppc..accc......ot gatech.
ABSTRACT
Physics-Informee Neural Networks (PINNs) have emerged as a promising deep learning framework for approximating numerical solutions to partial differential equations (PDEs). However, conventional PINNs, relying on multilayer per- ceptrons (MLP), neglect the crucial temporal dependencies inherent practical physics systems and thus fail to propagate the initial condition constraints globally and accurately capture the true solutions under various scenarios. In this paper, introduce novel transformer-basee framework, termed PINNsFormer, de- signed to address this limitation. PINNsFormer can accurately approximate PDE solutions by utilizing multi-head attention mechanisms to capture temporal de- pendencies. PINNsFormer transforms point-wise inputs into pseudo sequences and replaces point-wist wise PINNs loss with a sequential loss. Additionally, it in- corporates a novel activation function, Wavelet, which anticipates Fourier de- composition through deep neural networks. Empirical results demonstrate that PINNsFormer achieves superior generalization ability and accuracy across vari- ous scenarios, including PINNs failure modes and high-dimensional PDEs. More- over, PINNsFormer offers flexibility in integrating existing learning schemes for PINNs, further enhancing its performance.
INTRODUCTION
Numerically solving partial differential equations (PDEs) has been widely studied in science and engineering. The conventional approaches, such as finite element method (Bathe) (007) or pseudo- spectral method (Fornberg) 1998). suffer from high computational costs in constructing meshes for nigh-diimessioall PDEs. With the development of scientific machine learning, Physics-informed neural networks (PINNs) (Lagaris al. 1998 Raissi al. 2019) have emerged promising novel approach. Conventional PINNs and most variants employ multilayer perceptrons (MLP) as end-to- end frameworks for point-wiss wise predictions, achieving remarkable success in various scenarios.
Nevertheless, recent works have shown that PINNs fail in scenarios when solutions exhibit high- frequency or multiscale features (Raissi) 2018 Fuks Tchelepi 2020)] Krishnapriyan et al., 202T Wang et al. 2022a), though the corresponding analytical solutions are simple. In such cases, PINNs tend to provide overly smooth or naive approximations, deviating from the true solution.
Existing approaches to mitigate these failures typically involve two general strategies. The first strategy, known as data interpolation (Raissi et al., 2017 Zhu et al. 2019 Chen et al., (02T), em- ploys data regularization observed from simulations, or real-world scenarios. These approaches face challenges in acquiring ground truth data. The second strategy employs different training schemes (Mao al. 2020 Krishnapriyan al. 2021 Wang al. 2021 (022a), which potentially impose high computational cost in practice. For instance, Seq2Seq by Krishnapriyan et al. (202I) requires training multiple neural networks sequentially, while other networks suffer from convergence issues

---

Published as a conference paper at ICLR 2024
due to error accumalation. Another method, Neural Tangent Kernel (NTK) (Wang al. 2022a), in- volves constructing kernels K E PxP, where D is the sample size and P is the model parameter, which suffers from scalability issues the sample size model parameter increases.
While most efforts to improve the generalization ability and address failure modes in PINNs have focused on the aforementioned aspects, conventional PINNs, largely relying on MLP- architec- ture, can overlook important temporal dependencies real-world physical systems. Finite Element Methods, for instance, implicitly incorporate temporal dependencies by sequentially propagating the global solution. This propagation relies on the principle that the state at time i + A/ depends on the state time In contrast, PINNs, being point-to-point framework. do not explicitly model temporal dependencies within PDEs. Neglecting temporal dependencies poses challenges in glob- ally propagating initial condition constraints in PINNs. Consequently, PINNs often exhibit failure modes where the approximations remain accurate near the initial condition but subsequently fail into overly smooth or naive approximations.
To address this issue of neglecting temporal dependencies in PINNs, natural idea is employing Transformer-basse models, which are known for capturing long-term dependencies in sequential data through multi-head self-attentions and encoder-decoder attentions (Vaswani et al., 2017). Vari- ants of transformer-bat adapting the Transformer, which is inherently designed for sequential data, to the point-to- point framework of PINNs presents non-trivial challenges. These challenges span both the data represen- tation and the regularization loss within the framework.
models have shown substantial success across various domains. However,
Main Contributions. In this work, we introduce PINNsFormer, a novel sequence-to-ssquenee PDE solver built on the Transformer architecture. To the best of our knowledge, PINNsFormer is the first framework in the realm of PINNs that explicitly focuses on and learns temporal dependencies within PDEs. Our key contributions can be summarized follows:
New Framework:
We propose novel yet intuitive Transformer-basee framework named PINNsFormer. This framework equips PINNs with the capability to capture temporal de- pendencies through the generated pseudo sequences, thereby enhancing the generalization ability and approximation accuracy in effectively solving PDEs.
Novel Activation: We introduce new non-linear activation function Wave l et- Wavelet is designed to anticipate the Fourier Transform for arbitrary target signals, mak- ing it universal approximator for infinite-width neural networks. Wavelet can also be potentially beneficial to various deep learning tasks across different model architectures. Extensive Experiments: We conduct comprehensive evaluations of PINNsFormer for var- ious scenarios. We demonstrate its advantages in optimization and approximation accuracy when addressing failure modes or solving high-dimensional PDEs. We show the flexibility and benefits of PINNsFormer in incorporating different learning schemes of PINNs.
RELATED WORK
Physics-Informed Neural Networks (PINNs). Physics-Informed Neural Networks (PINNs) have emerged as promising approach for tackling scientific and engineering problems. Raissi et al. (2019) introduced the framework that incorporates physical laws into the neural network training to solve PDEs. This work has led to applications across diverse domains, including fluid dynamics, solid mechanics, and quantum mechanics (Carleo et al. 2019 Yang et al. (2020). Researchers have investigated different learning schemes for PINNs (Mao et al. [2200] Wang et al., 2021 2022a), which have yielded substantial improvements in convergence, generalization, and interpretability.
Failure Modes of PINNs. Despite the promise exhibited by PINNs, recent works have indicated certain failure modes inherent to PINNs, particularly when confronted with PDEs featuring high-
frequency or multiscale features (uus & Tchelepi 2020 Raissi 2018 McClenny & Braga-Neto/
2020 Krishnapriyan al. 2021 Zhao et al. 2022 Wang et al. (022a)). This challenge has prompted
investigations from various perspectives, including designing various model architectures, learning
schemes, using data interpolations (Han al. 2018 Lou et al. 2021 Wang et al., 2022 2202a b). A comprehensive understanding of PINNs limitations and the underlying failure modes i8 funda- mental for applications in addressing complicated physical problems.
2

---

Published as a conference paper at ICLR 2024
Pseudo Sequence Generator
Spatio-Temporal Mixer (MLLP)
Output Layer (MILP)
Raw Input
NN output
Encoder
Encoder
Approx.
Decoder
Decoder
NxDecoders
Figure 1: Architecture of proposed PINNsFormer. PINNsFormer generates pseudo sequence based on pointwise input features. It outputs the corresponding sequential approximated solution. The first approximation of the sequence is the desired solution ú( t).
Transformer-Basse Based Models. The Transformer model (Vaswani et al., 2017) has achieved signifi- cant attention due to its ability to capture long-term dependencies, leading to major achievements in natural language processing tasks (Devlin et al. 20188 Radford et al. 2018). Transformers have also been extended to other domains, including computer vision, speech recognition, and time-series analysis (Liu et al.] 2021] Dosovitskiy et al.] 2020) Guulati et al.] [2200] Zhou et al. (02T). Re- searchers have also developed techniques aimed at enhancing the efficiency of Transformers, such as sparse attention and model compression (Child et al. 2019: Sanh et al. 2019).
METHODOLOGY
Preliminaries: Let Ω be an open in Rd bounded by an Rd-1, and temporal input t generally fit the following abstraction:
The PDEs with spatial input
Diu(2 t)] f(æ, t), Vz, te e Ω B Blu(z x t)] = g( t), Vx, e an
where u is the PDE's solution, is the differential operator that regularizes the behavior the sys- tem, and B describes the boundary initial conditions in general. Specifically, {x, t} Ω are resid- ual points, and {a, t} E dΩ are boundary/initial points. Let á be neural network approximations, PINNs describe the framework where empirically regularized by the following constraints:
N. |D|ū(a ()) f(x 0)12 + Ao Σ [ ()) g(x, 1)l
No.
LPTNNS X
(2)
where N₁, N, refer to the residual and boundary/initial points separately, X, Às are the regulariza- tion parameters that balance the emphasis of the loss terms. The neural network ú takes vectorized {æ, f as input and outputs the approximated solution. The goal is then to use machine learning methodologies to train the neural network à that minimizes the loss in Equation
While PINNs focus on point-to-poing predictions, the exploration of tem- poral dependencies in real-world physics systems has been merely neglected. Conventional PINNs methods employ single pair of spatial information and temporal information to approximate the numerical solution u(æ:, t), without accounting for temporal dependencies across previous or subsequent time steps. However, this simplification is only applicable elliptic PDEs, where the relationships between unknown functions and their derivatives do not explicitly involve time. In contrast, hyperbolic and parabolic PDEs incorporate time derivatives, implying that the state one time step can influence states at preceding or subsequent time steps. Consequently, considering temporal dependencies is crucial effectively address these PDEs using PINNs.
Methodology Overview:
In this section, we introduce a novel framework featuring Transformer-base model of PINNs, namely PINNsFormer. Unlike point-to-point predictions, PINNsFormer extends PINNs capabil- ities to sequential predictions. PINNsFormer allows accurately approximating solutions at spe- cific time steps while also learning and regularizing temporal dependencies among incoming states. The framework consists of four components: Pseudo Sequence Generator, Spatio-Temporal Mixer, Encoder-Decode with multi-head attention, and an Output Layer. Additionally, we introduce a novel activation function, named Wavelet, which employs Real Fourier Transform techniques to anticipate solutions to PDEs. The framework diagram is exhibited in Figure !
We provide detailed explanations of each framework component and learning schemes in the following subsections.
3

---

Published as a conference paper at ICLR 2024
3.1 PSEUDO SEQUENCE GENERATOR
While Transformers and Transformer-basee models are designed capture long-term dependencies in sequential data, conventional PINNs utilize non-sequential data as inputs for neural networks. Consequently, to incorporate PINNs with Transformer-base models, it is essential to transform the pointwise spatiotemporal inputs into temporal sequences. Thus, for a given spatial input æ E Rd-1 and temporal input t E R, the Pseudo Sequence Generator performs the following operations:
t
Generator,
{[z, t) [xc At). ................................ (x,t (k-1)
where [-] is the concatenation operation, such that [ar, t E Rd is vectorized, and the generator outputs the pseudo sequence in the shape of Rxd The Pseudo Sequence Generator extrapolates sequential time series by extending a single spatiotemporal input to multiple isometric discrete time steps. k and At are hyperparameters, which intuitively determine how many steps the pseudo sequence needs to 'look ahead' and how 'far' each step should be. In practice, both and At should not be set to very large scales, larger & cause heavy computational and memory overheads, while larger A may undermine the time dependency relationships of neighboring discrete time steps.
3.2 MODEL ARCHITECTURE
In addition the Pscudo Sequence Generator, PINNsFormer consists of three components of its ar- chitecture: Sptio-Tempone Mixer, Encoder-Decode with multi-heat attentions, and Output Layer. The Output Layer is straightforward to interpret as fully-connected MLP appended to the end. We provide detailed insights into the first two components below. Notably, PINNsFormer relies only on linear layers and non-linear activations, avoiding complex operations such convolutional or recurrent layers. This design preserves PINNsFormer's computational efficiency in practice.
Spatio-Temporal Mixer. Most PDEs contain low-dimensional spatial or temporal information. Directly feeding low-dimensional data to encoders may fail to capture the complex relationships between each feature dimension. Hence, is necessary embed original sequential data in higher- dimensional spaces such that more information is encoded into each vector.
Instead of embedding raw data in a figh-dimensionll space where the distance between vectors re- flects the semantic similarity (Vaswani et al.) 2017, Devlin et al. 2018), PINNsFormer constructs a linear projection that maps spatiotemporal inputs onto a higher-dimensions space using fully- connected MIP. The embedded data enriches the capability of information by mixing all raw spa- tiotemporal features together, so-called the linear projection Opatio-Tempor Mixer.
Encoder-Decodee Architecture. PINNsFormer employs an encoder-decoder architecture similar to Transformer. The encoder consists of stack of identical layers, each of which contains an encoder self-attention layer and a feedforward layer. The decoder is slightly different from the vanilla Transformer, where each of the identical layers contains only an encoder-decoder self-attention layer and a feedforward layer. At the decoder level, PINNsFormer uses the same spatiotemporal embeddings as the encoder. Therefore, the decoder does not need to relearn depen- dencies for the same input embeddings. The diagram for the encoder-decoder architecture is shown in Figure
Decoder Duipat
Decoder
Add & Wardler
Add Virelltt
Rool Feeward
Bood Forward
Add & Werelet
Add Wereeet
Muiti-Hlaa Atrestion
Muli-tiead
Inpot Embaddding
Intuitively, the encoder self-attentions allow learning the dependency relationships of all spatiotemporal informa- tion. The decoder encoder-decoder attentions allow se- lectively focusing specific dependencies within the in- put sequence during the decoding process, enabling it to
The architecture of PINNs- Former's 's Encoder-Decode Layers. The decoder is not equipped with self- attentions.
Figure 2:
capture more information than conventional PINNs. We the same embeddings for the encoder and decoder since PINNs focus on approximating the solution of the current state, in contrast to next state prediction in language tasks time series forecastings.
4

---

Published as a conference paper at ICLR 2024
3.3 WAVELET ACTIVATION
While Transformers typically employ LayerNorm and ReLU non-linear activation func- tions (Vaswani et al. 2017; Gehring et al.) 2017 Devlin et al., 2018), these activation functions might not always be suitable in solving PINNs. In particular, employing ReLU activation in PINNs can result in poor performance, whose effectiveness relies heavily on the accurate evaluation of derivatives while ReLU has discontinuous derivative (Haghighat et al., [02T]; de Wolff et al., 202T]. Recent studies utilize Sin activation for specific scenarios mimic the periodic proper- ties of PDEs' solutions (Li et al. 2020; Dagtap et al.) 2020; [Song et al., (022). However, it requires strong prior knowledge of the solution's 's behavior and is limited in its applicability. Tackling this issue, we proposed novel and simple activation function, namely Wavelet, defined as follows:
Wavelet(2r) = wi sin(x)) + w2 cos(2)
Where wi and w2 are registered learnable parameters. The intuition behind Wavelet activation simply follows Real Fourier Transform: While periodic signals can be decomposed into an integral of sines of multiple frequencies, all signals, whether periodic or aperiodic, can be decomposed into an integral of sines and cosines of varying frequencies. is evident that Wavelet can approximate arbitrary functions giving sufficient approximation power, which leads to the following proposition:
Let N be a two-hidden-laye -lyer neural network with infinite width, equipped with Wavelet activation function, then N is universal approximator for any real- target
Proposition 1
Proof sketch: The proof follows the Real Fourier Transform (Fourier Integral Transform). For any given input and its corresponding real-valued target (xx), it has the Fourier Integral:
f(x) = / Folw) cos(wa) du + / 000
(w(w) sin dw
where Fe and F, are the coefficients of Sines and Cosines respectively. Second, by Riemann sum approximation, the integral can be approximated by the infinite sum such that:
N
f(x) ~0 (F((W))) cos(u,,)) + Fs (win) sin(w,,))) = Wa ((([)a)))ee((1)))
12
where W2 and W2 are the weights of N first and second hidden layer. As W1 and W2 are infinite- width, we can divide the piecewise summation into infinitely small intervals, making the approxi- mation arbitrarily close to the true integral. Hence, A is universal approximator for any given f. In practice, most PDE solutions contain only finite number of major frequencies. Using a neural network with finite parameters would also lead to proper approximations of the true solutions.
Although Wavelet activation function is primarily employed by PINNsFormer to improve PINNs in our work. it may have potential applications in other deep-learning tasks. Similar to ReLU, (+), and Tanh activations, which all turn infinite-width two-hidden-lage -layer neural networks into universal approximators (Cybenko 1989; Hornik 1991 Glorot et al. 2011). T). we anticipate that Wavelet can demonstrate its effectiveness in other applications beyond the scope of this work.
3.4 LEARNING SCHEME
While conventional PINNs focus on point-to-poin predictions, adapting PINNs to handle pscudo- sequential inputs has not been explored. In PINNsFormer, each generated point in the sequence, i.e.,
[x,,ta+7 +j is mapped to the corresponding approximation, i.e., (æx,, f, +j jAa for any j E N, < k. This approach allows us compute the rath-order gradients with respect æ or ł independently for any valid n. For instance, for any given input pseudo sequence {[ 4;], [x;, [+) (.....,,,.., 4+
(k- 1) and the corresponding approximations (ū(z), t.), ú(z,, t4 + At). .... a(x,, t, (k
1) y. we can compute the first-order derivatives w.r.t. x and i separately as follows:
(a((,,, + ( i
Jū(x;, 4) Aū(a, 4 + At)
aū(a, 4: + (k 1)
+ a(t++
O(t + At)
att + (k 1))
(5)
0(0(z), l + + jA() 0i(an, 1n)
Oū(a,, 4 + At)
aū(a, 44 + (k - 1))1))
dx;
dx,
dx;
Ox;
5

---

Published as a conference paper at ICLR 2024
This scheme for calculating the gradients of sequential approximations with respect to sequential inputs can be easily extended to higher-order derivatives and is applicable to residual, boundary, and initial points. However, unlike the general PINNs optimization objective in Equation 2 which combines initial and boundary condition objectives, PINNsFormer distinguishes between the two and applies different regularization schemes initial and boundary conditions through its learning scheme. For residual and boundary points, all sequential outputs can be regularized using the PINNs loss. This is because all generated pseudo-timesteps are within the same domain as their original inputs. For example, if [x,, ts] is sampled from the boundary, then [ + j A also lies on the contrast, for initial points, only the
boundary for any j E N+. corresponding to the first element of the sequential outputs. This is because only the first element of the pseudo-sequence exactly matches the initial condition at = 0. All other generated time steps have 1 j&l for any E N which fall outside the initial conditions.
0 condition is regularized,
By these considerations, we adapt the PINNs loss to the sequential version, as described below:
No
Lixs
Diū(a; 4 + jA() - [( 4 +
K.
Nw k-1
Lbx = b:
[ j g to + jAt)l??
(6)
i=1 1=0
No.
1
Ci =
[ 0)] - h(xx, 0)11
Nic
LATNISFOrmer
Ameheer + Aichic + Ane
where Nies = N₂ refers to the residual points as in Equation boundary and initial points, respectively, with Noc + N No. Avess >be. and A regularization weights that balance the importance of the loss terms in PINNsFormer, similar to the PINNs loss.
Nbe: Nie represent the number of
During training, PINNsFormer forwards all residual, boundary. and initial points to obtain their cor- responding sequential approximations. It then optimizes the modified PINNS loss LeiNssFormar in Equation
6 using gradient-based based optimization algorithms such as L-BFGS or Adam, updating the model parameters until convergence. In the testing phase, PINNsFormer forwards any arbitrary pair [x, t] to observe the sequential approximations, where the first element of the sequential approxima- tion corresponds exactly to the desired value of ū(x, t).
3.5
Loss LANDSCAPE ANALYSIS
While achieving theoretical conver- gence or establishing generalization bounds for Transformer-busedd mod- els can be challenging. an alterna- tive approach to assess optimization trajectory is through visualization of the loss landscape. This approach has been employed in the analysis of both Transformers and PINNs (Krish-
100
Visualization of the loss landscape for PINNs (left) and PINNsFormer (right) on a logarithmic scale. The loss landscape of PINNsFormer significantly smoother than conventional PINNs.
Figure 3:
napriyan et al.) 2021 Yao et al. 2020; Park & Kim (022). The loss landscape is constructed by perturbing the trained model along the directions of the first two dominant Hessian cigenvectors. This technique is more informative than random parameter perturbations. Generally, smoother loss landscape with fewer local minima indicates an easier convergence to the global minimum. We visualize the loss landscape for both PINNs and PINNs- Former. The visualizations are presented in Figure
The visualizations clearly reveal that PINNS exhibit
complicated loss landscape than PINNs- Former. To be specific, we estimate the Lipschitz constant for both loss landscapes. We find that TAINNS 17, which is significantly larger than Tap TernsFoorer 32.79. Furthermore, the loss

---

Published as a conference paper at ICLR 2024
landscape of PINNs exhibits several sharp cones near its optimal point, indicating the presence of multiple local minima in close proximity to the convergence point (zero perturbation). The rugged loss landscape and multiple local minima of conventional PINNs suggest that optimizing the objec- tive described in Equation 6 for PINNsFormer offers an easier path to reach the global minimum. This implies that PINNsFormer has advantages in avoiding the failure modes associated with PINNs. The analysis is further validated by empirical experiments, as shown in the following section.
4 EXPERIMENTS
4.1 SETUP
Goal. Our empirical evaluations aim to demonstrate three key advantages of PINNsFormer. First, we show that PINNsFormer improves generalization abilities and mitigates failure modes compared to PINNs and variant architectures. Second, we illustrate the flexibility of PINNsFormer in in- corporating various learning schemes, resulting in superior performance. Third, we provide ev- idence of PINNsFormer's faster convergence and improved generalization capabilities in solving high-dimensional PDEs, which can be challenging for PINNs and their variants.
Experiment Setup. Our empirical evaluations rely on four types of PDEs: convection, ID-reaction, 1D-wave, and Navier-Stokes PDEs, which follow the established setups of preliminary studies for
fair comparisons (Raissi et al. 2019 Krishnapriyan et al., 2021 Wang et al., 2022a)). We include
PINNs, QRes (Bu & Karpatne 2021 and First-Layer Sine (FLS) (Wong et al. 2022) as baselines.
For convection, ID-reaction, and ID-wave PDEs, we uniformly sampled Nrc Nb. = 101 initial and boundary points, as well as uniform grid of 101 x 101 mesh points for the residual domain, resulting in total N₂,, 10201 points. In the case of training PINNsFormer, we reduce the collo- cation points, with N2- = Nox 51 initial and boundary points and a 51 x 51 mesh for residual points. The reduction in fewer training samples serves two purposes: it enhances training efficiency and allows to demonstrate the generalization capabilities of PINNsFormer with limited training data. For testing, we employed a 101 x 101 mesh within the residual domain. For the favicr-Stoker PDE, we sample 2500 points from the 3D mesh within the residual domain for training purposes. The evaluation was performed by testing the predicted pressure at the final time step t = 20.0.
Evaluation. For all baselines and PINNsformer, maintain approximately close numbers of pa- rameters across all models to highlight the advantages of PINNsFormer from its ability to capture temporal dependencies rather than relying solely model overparameterization We train all mod- els using the L--FGS optimizer with Strong Wolfe linear search for 1000 iterations. For simplicity, we set \ves = Ale Abe = for the optimization objective in Equation 6 Detailed hyperparam- eters are provided in Appendix A We also include an ablation study on activation functions and a hyperparameter sensitivity study on the choice of {k, At} in Appendix
In terms of evaluation metrics, we adopted commonly used metrics in related works (Krishnapriyan al. [02T]] Raissi et al.] [019 McClenny & Braga-Neto 2020), including the relative Mean Abso- lute Error (rMAE or relative & error) and the relative Root Mean Square Error (rRMSE or relative lg error). The detailed formulations of the metries are provided in Appendix A
Reproducibility. All models are implemented in PyTorch (Paszke et al. 2019), and are trained separately on single NVIDIA Tesla V000 GPU. All code and demos are included and reproducible at: hittps://qithan http://ttttttttttt com wwwwo///wwwaaaao//o/.oo/ wwwpaaaawwwbpinnssoooeeeee
4.2 MITIGATING FAILURE MODES OF PINNS
Our primary evaluation focuses demonstrating the superior generalization ability of PINNs- Former in comparison to PINNs, particularly on PDEs that are known to challenge PINNs' gen- eralization capabilities. We focus on solving two distinct types of PDEs: the convection equation and the DD-recctio equation. These equations pose significant challenges for conventional MLP- based PINNs, often resulting in what is referred to "PINNs failure modes" (Mojgani et al. 2022 Daw et al. 2022. Krishnapriyan et al., (02I). In these failure modes, optimization gets stuck in local minima, leading overly smooth approximations that deviate from the true solutions.
7

---

Published as a conference paper at ICLR 2024
The objective of our evaluation is showcase the enhanced generalization capabilities of PINNS- Former when compared to standard PINNs and their variations, specifically in addressing PINNs' failure modes. The evaluation results are summarized in Table I with detailed PDE formulations provided in Appendix B We showcase the prediction and absolute error plots of PINNs and PINNS- Former convection equation in Figure all prediction plots available in Appendix C
Convection rMAE
ID-Reaction rMAE
Model
Loss
iRMSE
Loss
rRMSE
PINNs 0.016
0.778
0.840
0.199
0.982
0.981
QRes
0.015
0.746
0,816
0.199
0.979
0.977
FLS
0.012
0.674
0.771
0.199
0.984
0.985
PINNsFormer 3.7e-5
0.023
0.027
3.0e-6
0.015
0.030
Table I: Results for solving convection and 1D-reaction equations. PINNsFormer consistently out- performs all baseline methods in terms of training loss, rMAE, and rRMSE.
The evaluation results demonstrate significant outperformance of PINNsFormer over all baselines for both scenarios. PINNsFormer achieves the lowest training loss and test errors, distinguish- ing PINNsFormer as the only approach capable of mitigating the failure modes. In contrast, all other baseline methods remain stuck at global minima and fail to optimize the objective loss ef- fectively. These results show the clear advantages of PINNsFormer in terms generalization ability and approximation accuracy when compared to conventional PINNs and existing variants.
The additional concern for PINNsFormer is its computational and memory overheads rel- ative PINNs. While MLP-based PINNs are known for efficiency, PINNsFormer, with Transformer-basee architecture in handling se- quential data, naturally incurs higher compu- tational and memory costs. Nonetheless, our empirical evaluation indicates that the overhead is tolerable, benefitting from the reliance on only linear layers, avoiding complicated oper- ators such as convolution or recurrent layers. For instance, when setting the pseudo-sequence length k = 5, we observe an approximate 2.92x computational cost and 2.x5x memory usage (detailed in Appendix A). These overhcads are reasonable in exchange for the substantial per- formance improvements by PINNsFormer.
Aoselute Tror
0.5
Predicted uocti
Figure 4: Prediction (left) and absolute error (right) of PINNs (up) and PINNsFormer (bottom) on convection equation. PINNsFormer shows suc- cess in mitigating the failure mode than PINNs.
4.3 FLEXIBILITY IN INCORPORATING VARIANT LEARNING SCHEMES
While PINNs and their various archi- tectural adaptations may encounter chal- lenges for certain scenarios, prior re- search has explored sophisticated opti- mization schemes to mitigate these is- sues. including learning rate anneal- ing (Wang et al., [202T), augmented La- grangian methods (Lu et al., (02T). and neural tangent kernel approaches (Wang) et al. 2022a). These modified PINNs have shown significant improvement of PINNs under certain scenarios. Notably, when these optimization strategies are applied to PINNs- Former, they can be easily incorporated to achieve further performance improvements. For instance, the Neural Tangent Kernel (NTK) method to PINNs has shown success in solving the ID-wave equation. As such, demonstrate that when combining NTK with PINNsFormer,
ID-Wave
Model
Loss
IMAE
IRMSE
PINNs
1.93e-2
0.326
0.335
PINNsFormer
1.38e-2
0.270
0.283
PINNs + NTK
6.34e-3
0.140
0.149
PINNsFormer + NTK
4.21e-3
0.054
0.058
Table 2: Results for solving the 1D-wave equation, in- corporating the NTK method. PINNsFormer combined with NTK outperforms all other methods on all metrics.
achieve
8

---

Published as a conference paper at ICLR 2024
further outperformance in approximation accuracy. Detailed results are presented in Table 2 and comprehensive PDE formulations are available in Appendix with prediction plots in Appendix
Our evaluation results show both the flexibility and effectiveness of incorporating PINNsFormer with the NTK method. In particular, we observe a sequence of performance improvements, from standard PINNs to PINNsFormer and from PINNs+N7K to PINNsFormer++W7K. Essentially, PINNsFormer explores a variant architecture of PINNs, while many learning schemes are designed from an optimization perspective and are agnostic to neural network architectures. This inherent flexibility allows for versatile combinations of PINNsFormer with various learning schemes, offer- ing practical and customizable solutions for accurate solutions real-worke applications.
Training Loss vs Iterations
4.4 GENERALIZATION ON DIMENSIIMENIONNLL PDES
207
MNNN
Gites
In the previous sections, demonstrated the clear benefits of PINNsFormer in generalizing the solutions for PINNs failure modes. However, those PDEs often have simple analytical so- and
10-2
10-1-
lutions. In practical physics systems, higher-dimensions more complex PDEs need to be solved. Therefore, it's impor- tant to evaluate the generalization ability of PINNsFormer on such high-dimensional PDEs, especially when PINNsFormer is equipped with advanced mechanisms like self-attention.
0
30C
1000
1500
2000
Iar
Figure 5: Training loss vs. Itera- tions of PINNs and PINNsFormer on the Navier- Stokes equation.
We evaluate the performance of PINNsFormer compared to PINNs on Navier-Stokes PDE based on the established setups Raissi et al. (2019). The training loss
shown in Figure [ and the results are shown in Table 3
The detailed formulations of the 2D
Navier-Stokes equation be found in Appendix B and the predictions are plotted in Appendix C
The evaluation results demonstrate clear advan- tages of PINNsFormer over PINNs on high- dimensional PDEs. Firstly, PINNsFormer out- in
Navier-Stoke rMAE
Model
Loss
rRMSE
performs PINNs and their terms of both training loss and validation errors. Firstly, PINNsFormer exhibits sig- nificantly faster convergence during training, which compensates for the higher computa- tional cost per iteration. Secondly, while PINNs and their MLP-variants predict the pressure with good shapes, they exhibit increasing mag- nitude discrepancies as time increases. In contrast, PINNsFormer consistently aligns both the shape and magnitude of predicted pressures across various time intervals. This consistency is attributed to PINNsFormer's s ability to learn temporal dependencies through Transformer based model architec- ture and celf-attention mechanism.
PINNs
6.72c-5
13.08
9.08
QRes
2.24e-4
6.41
4.45
FLS
9.54c-6
3.98
2.77
PINNsFormer
6.66e-6
0.384
0.280
Table 3: Results for solving Navier-Stoke equa- tion, PINNsFormer outperforms all baselines on all metries.
CONCLUSION
In this paper, we introduced PINNsFormer, novel Transformer-based framework of PINNs, aimed at capturing temporal dependencies when approximating solutions to PDEs. We introduced the Pseudo Sequence Generator, a mechanism that translates vectorized inputs into pseudo time quences and incorporated modified Encoder-Decoded layer along with a novel Wavelet ac- tivation. Empirical evaluations demonstrate that PINNsFormer consistently outperforms conven- tional PINNs across various scenarios, including handling PINNs failure modes, addressing high- dimensional PDEs, and integrating with different learning schemes for PINNs. Furthermore, PINNs- Former retains computational simplicity, making it practical choice for real-world applications.
Beyond PINNsFormer, Wavelet activation function can hold promises for the broader machine learning community. We provided sketch proof demonstrating Wavelet's ability to approxi- mate arbitrary target solutions using a wwo-hideen-layer infinite- width neural network, leveraging the Fourier decomposition of these solutions. We encourage further exploration, both theoretically and empirically, of the Wavelet activation function's 's potential. Its applicability extends beyond PINNs and can be leveraged in various architectures and applications.

---

Published as a conference paper at ICLR 2024
Acknowledgements: This paper supported in part by the NSF (Expeditions CCF-1918770, CAREER IIS- USS-008586, Medium 195-885,833, Medium IIS-2106961, PIPP CCF-2200269), CDC MInD program, Meta faculty gift, and funds/computing resources from Georgia Tech and GTRI.
REFERENCES
Klaus-Jürgen Bathe. Finite element method. Wiley encyclopedia of computer science and engineer- ing. pp. 1-12, 2007.
Jie Bu and Anuj Karpatne. Quadratic residual networks: A new class of neural networks for solv- ing forward and inverse problems in physies involving pdes. In Proceedings of the 2021 SIAM International Conference Data Mining (SDM), pp. 675-683 SIAM, 2021.
Giuseppe Carleo, Ignacio Cirac, Kyle Cranmer, Laurent Daudet, Maria Schuld, Naftali Tishby, Leslie Nogt-Maranto, and Lenka Zdeborová. Machine learning and the physical sciences. Re- views of Modern Physics, (144)004500, 2019.
Zhao Chen, Yang Liu, and Hao Sun. Physics-informed learning of governing equations from scarce data. Nature commuicattinns, 12(1):6136, 2021.
Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904. 10549 2019.
George Cybenko. Approximation by superpositions of a sigmoidal function. Mathematics control, signals and systems, 2(4):303-314 1989.
Arka Daw, Jie Bu, Sifan Wang, Paris Perdikaris, and Anuj Karpatne. Rethinking the importance of sampling in physics-informn neural networks. arXiv preprint arXiv:2207.02333 2022.
Taco de Wolff, Hugo Carrillo, Luis Martí, and Nayat Sanchez-Pi. Assessing physics informed neural networks in ocean modelling and climate change applications. In Al: Modeling Oceans and Climate Change Workshop at ICLR 2021, 2021.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:/810.04500 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11 1929, 2020.
Bengt Fornberg. A practical guide to pseudospectral methods. Number 1. Cambridge university press, 1998.
Olga Fuks and Hamdi A Tchelepi. Limitations of physics informed machine learning for nonlinear two-phase transport in porous media. Journal of Machine Learning for Modeling and Computing, 1(1), 2020.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann Dauphin. Convolutional sequence sequence learning. In International conference machine learning, pp. 1243-1252. PMLR, 2017.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp- 315-323. JMLR Workshop and Conference Proceedings, 2011.
Anmol Gulati, James Qin, Chung -Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et Conformer: Convolution-augmentee transformer for speech recognition. arXiv preprint arXiv:2005.081000 2020.
Ehsan Haghighat, Maziar Raissi, Adrian Moure, Hector Gomez, and Ruben Juanes. A physics- informed deep learning framework for inversion and surrogate modeling in solid mechanics. Computer Methods in Applied Mechanics and Engineering, 379:113741,, 2021.
10

---

Published as a conference paper at ICLR 2024
Jiequn Han, Arnulf Jentzen, and Weinan E. Solving high-dimensional partial differential equations using deep learning. Proceedings of the National Academy of Sciences, 115((4):8505-855)) -8510, 2018.
Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural networks, 4 (2):551-257, 1991.
Ameya D Jagtap, Kenji Kawaguchi, and George Em Karniadakis. Adaptive activation functions accelerate convergence in deep and physics-infommed neural networks. Journal of Computational Physics, (004:00936, 2020.
Aditi Krishnapriyan, Amir Gholami, Shandian Zhe, Robert Kirby, and Michael W Mahoney. Char- acterizing possible failure modes in physics-informed neural networks. Advances in Neural In- formation Processing Systems, 34:26548-26560 2021.
Isaac E Lagaris, Aristidis Likas, and Dimitrios Fotiadis. Artificial neural networks for solving ordinary and partial differential equations. IEEE transactions on neural networks, 95)::977-1000, 1998.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, An- drew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895, 2020.
Ze Liu, Yutong Lin, Yue Cao, Han Hu. Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021.
Qin Lou, Xuhui Meng, and George Em Karniadakis. Physics-informed neural networks for solving forward and inverse flow problems via the boltzmann-bgk formulation. Journal of Computational Physics, 447:1 10676. 2021.
Lu Lu, Raphael Pestourie, Wenjie Yao, Zhicheng Wang, Francese Verdugo, and Steven G Johnson. Physics-informed neural networks with hard constraints for inverse design. SIAM Journal on Scientific Computing, 43(6):B1 100-10111111 132, 2021.
Zhiping Mao, Ameya D Jagtap, and George Em Kariaadkki.. Physics-informed neural networks for high-speed flows. Computer Methods in Applied Mechanics and Engineering. 360: 12789, 2020.
Levi McClenny and Ulisses Braga-Neto. Self-adaptive physics-informed neural networks using a soft attention mechanism. arXiv preprint arXiv:2009.045: 2020.
Rambod Mojgani, Maciej Balajewicz, and Pedram Hassanzadeh. Lagrangian pinns: A causality-
conforming solution to failure modes of physics-informed neural networks. arXiv preprint
arXiv:2205.02992, 2022.
Namuk Park and Songkuk Kim. arXiv:2202.06709, 2022.
How do vision transformers work?
arXiv preprint
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances neural information processing systems, 32, 2019.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, al. Improving language under- standing by generative pre-train 2018.
Maziar Raissi. Deep hidden physics models: Deep learning of nonlinear partial differential equa- tions. The Journal of Machine Learning Research, 19(1):32-955, 2018.
Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Physics informed deep learn- ing (part i): Data-driven solutions of nonlinear partial differential equations. arXiv preprint 1711v 105611 2017.
Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686-707, 2019.

---

Published as a conference paper at ICLR 2024
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019.
Chao Song, Tariq Alkhalifah, and Umair Bin Waheed. A versatile framework to solve the helmholtz equation using physics-informed neural networks. Geophysical Journal International, 228(3): 1750-1762, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
Sifan Wang. Yujun Teng, and Paris Perdikaris. Understanding and mitigating gradient flow patholo- gies in wwwspcs-informnneee neural networks. SIAM Journal on Scientific Computing, A3(55: A3081, 2021.
Sifan Wang, Xinling Yu, and Paris Perdikaris. When and why pinns fail train: A neural tangent kemee perspective. Journal of Computational Physics, 449: 149:110768, 2022a.
Yicheng Wang, Xiaotian Han, Chia- Yuan Chang, Daochen Zha, Ulisses Braga-Neto, and Xia Hu. Auto-pinn: Understanding and optimizing physics-informed neural architecture. arXiv preprint 13444, 2022b.
Jian Cheng Wong. Chinchun Ooi, Abhishek Gupta, and Yew-Soon Ong. Learning in sinusoidal spaces with physics-informed neural networks. IEEE Transactions on Artificial Intelligence, 2022.
Liu Yang, Dongkun Zhang, and George Em Karniadakis. Physics-informed generative adversarial networks for stochastic differential equations. SIAM Journal on Scientific Computing, 42(1): A292-A317, 2020.
Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W Mahoney. Pyhessian: Neural networks through the lens of the hessian. In 2020 IEEE international conference on big data (Big data), pp. 581-590. IEEE, 2020.
Zhiyuan Zhao, Xueying Ding, Gopaljee Atulya, Alex Davis, and Aarti Singh. Physics informed machine learning with misspecified priors:) aan analysis of turning operation in lathe machines. In AAAI 2022 Workshop on Al for Design and Manufacturing (ADAM), 2022.
Haoyi Zhou, Shanghang Zhang. Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Beyond efficient transformer for long sequence time-series forecasting, In Proceedings of the AAAI conference artificial intelligence, volume 35, pp. 11106-11115, 2021.
Yinhao Zhu, Nicholas Zabaras, Phaedon-Stelion Koutsourelakis, and Paris Perdikaris. Physics- constrained deep learning for high-dimensional surrogate modeling and uncertainty quantification without labeled data. Journal of Computational Physics, 394:56-81, 2019.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX A: MODEL HYPERPARAMETERS
Model Hyperparameters. We provide detailed set of hyperparameters used obtain the experi- ment results, shown in Table
Model
Hyperparameters
Value
Model Parameters
hidden layer
4
PINNs & FLS
527k
hidden size
512
hidden layer
4
QRes
397k
hidden size
256
5
At # of encoder # of decoder embedding size head hidden size
le-3, 1e-4
1
PINNsFormer
1
454k
32
2
512
Table 4: Hyperparameters for Main Results
Training Overhead. We compare the training overhead of PINNsFormer over PINNs, as PINNs known an efficient framework while Transformer-based models are known for being com- putationally costly. The comparison relies on solving the Convection PDEs, which are detailed in Table Here, we vary the hyperparameter of pseudo-sequence length for validation purposes. In practice, we set k = 5 for all the empirical experiments in this paper.
Training Time
Computational Overhead
GPU Memory (MiB)
Memory Overhead
Model
(sec/epoch)
PINNs
0.80
1311
k=3
2.10
2.62x
2207
1.68x
PINsFormer
A=5
2.34
2.92x
2827
2.15x
k=10
3.10
3.87x
4803
3.66x
Table 5: Overhead of PINNsFormer than PINNs in varying pseudo-sequence length. Both compu- tational and memory overhead are tolerable and grow approximately linearly as increases
Evaluation Metrics. We present the detailed formula of rMAE and rRMSE as the following:
N | (m)) - u( tw)
IMAE
EnTr |u(2 bn)
ΣN [ bn) - u( 12
RMSE
Σn |u(Inn tn
where is the number of testing points, à is the neural network approximation, and u is the ground truth.
B APPENDIX B: PDES SETUPS
We provide detailed PDE setups for convection, reaction-diffusion, and IDrreaction equations.
13

---

Published as a conference paper at ICLR 2024
Convection PDE. The one one-dimensional convection problem is hyperbolic PDE that is commonly used to model transport phenomena. The system has the formulation with periodic boundary condi- tions as follows:
du du
dt + B
= 0, Vr E [0, 2x). L E [0.
dr
IC: (2, 0) sin(r). BC: 8) = u(2 t)
where 3 is the convection coefficient. As 3 increases, the frequency of its solution goes higher, and becomes harder for PINNS approximate. Here, we set 50.
ID-Reaction PDE. The one-dimensional reaction problem is a hyperbolic PDE that commonly used model chemical reactions. The system has the formulation with periodic boundary condi- tions as follows:
du
-pp(1 - u) = 0, Va E [0, 2m]. t [0,
01
(9)
IC: (x2 exp(- (x (x-=)²²
BC: u(2 t)
2( 2(T/4)2
where is the reaction coefficient. Here, we set P = 5. The equation has a simple analytical solution:
h(x) exp(pt) (pt)
Manalytica: e(z))xp((pl ) +
h(x)
(10)
where h(x) is the function of the initial condition.
1D-Wave PDE. The ID-Wave equation is a hyperbolic PDE that is used to describe the propagation of waves one spatial dimension. is often used in physics and engineering to model various wave phenomena, such as sound waves, seismic waves, and electromagnetic waves. The system has the formulation with periodic boundary conditions as follows:
8h
8
Vo E [0, 1), LE [0
dr
1
Ou(x, 0) Øt
(11)
IC: (c: 0) = sin πτ) + (in(BAr),
BC: BC: (0, t) = u(1, =
where the wave speed. Here, we are specifying solution:
The equation has simple analytical
u(x, 1) = sin((T)) cos() sin (3Tx) cos (23xl)
(12)
2D Navier-Stokes PDE. The 2D Navier-Stokes equation a parabolic PDE that consists of a pair of partial differential equations that describe the behavior of incompressible fluid flow in two- dimensional space. They are widely used in fluid dynamics to model the motion of fluids, such as air and water, in various engineering and scientific applications. The system has the formulation as follows:
Ou
+ A1 (u Ou
Du + Oy
Op
8u
8u
+ X
+
0t
dr
dr
0
0.2
(13)
dv
+ A1 (u dv
dv
Op
8.u
+ v
+ A2
01
dr
Oy
Oy
0.12
0
where u(t, 2r, y) and v(t, x) are the r-component and y-component of the velocity field separately, and p(t, x, y) is the pressure. Here, we set ê₁ and X 0.01. The system does not have an explicit analytical solution, while the simulated solution given by Raissi et al. (2019)
C APPENDIX C: ADDITIONAL RESULTS
To investigate the effectiveness of the Wavelet ac- tivation function in PINNsFormer, compare the performance differences using Wavelet than
Ablation Study on Activation Functions.
14

---

Published as a conference paper at ICLR 2024
ReLU, Sigmoid, and Sin activation functions over convection and ID-reaction problems. In par- ticular, we study the effects of using the same activation function in both the feed-forward layer and encoder/decoder layer (marked ReLU, etc.) and changing the activation function of the coder/decoder layer to LayerNorm (as vanilla Transformer does, marked as ReI.U+L,, etc.). The evaluation results shown in Table
Convection rMAE
ID-Reaction rMAE
Activation
Loss
rRMSE
Loss
rRMSE
ReLU
0.5256
1.001
1.001
0.2083
0.994
0.996
Sigmoid
0.1618
1.112
1.223
0.1998
0.991
0.993
Sin 0.3159
1.074
1.141
4.9c-6
0.017
0.032
ReLU+LN
0.7818
1.001
1.002
0.2028
0.992
0.993
Sigmoid+LN
0.0549
0.941
0.967
0.2063
0.992
0.990
Sin+LN 0.3219
1.083
1.156
4.7c-6
0.016
0.033
Wavelet
3.7e-5
0.023
0.027
3.0e-6
0.015
0.030
Wavelet+LN
NaN
NaN
NaN
3.9e-6
0.018
0.037
Table 6: Results for solving convection and 1D-reaction equations using Transformer architecture
with different activation functions. PINNsFormer (with Wavelet activation) consistently outperforms all other activation functions in terms of training loss, rMAE, and rRMSE
The ablation study results show two major conclusions: First, using wavelet activation shows con- stantly better performance than ReLU, Sigmoid, and Sin activations. In particular, Sin activation may show effectiveness in only certain cases, while Wavelet can generalize all cases well. Sec- ond, Introducing LayerNorm activation to the encoder/decoder does not significantly contribute to performance improvement. In contrast. LayerNorm activation may cause convergence issues when coupling with the Wavelet activation function for certain situations.
Hyperparameter Sensitivity Study. To investigate the possible difficulties in picking hyperparam- eters l and A, we compared the performance differences with a mesh choice of these two hyperpa- rameters over the Id-reaction problem. The evaluation results (relative-22 error, with failure modes bolded) are shown in Table 7
At
k=3
k=5
k=7
k=10
Te-1
0.044
0.514
0.743
0.731
le-2
0.029
0.035
0.045
0.049
le-3
0.037
0.037
0.024
0.035
le-4
0.997
0.030
0.029
0.046
le-5
0.977
0.026
0.977
0.021
Table 7: Results for solving ID-reaction equation with various combinations of At and k. PINNs- Former shows the flexibility of wide choice of hyperparameters on certain problems.
The study on hyperparameter sensitivity of At and & exhibits three intuitions: First, given a mesh choice of lo and At, PINNsFormer is not sensitive to wide range of the two hyperparameters. For instance, PINNsFormer successfully mitigates the failure modes for any combinations of k: e [1e 2, le 3, le 4| and AL E [3, 5, 7]. Second, the choice of A/ should not be either too large (i.c.. le-1) or too small (i.c., le-5). Intuitively, either a too-large or a too-small At degrades the temporal dependencies between discrete time steps. Third, increasing the pscudo-sequence length can help mitigate PINNs failure modes (i.c., = + 5 when At le - 4). However, once PINNs successfully mitigate the failure mode, the benefit of further increasing 1: is marginal.
Result Visualizations. We here present the plots of ground truth solutions, neural network pre- dictions, and absolute errors for all evaluations included in the experimental section. The plots on convection, ID-reaction, ID-wave, and 2D Havier-Stoker equations are shown in Figure separately.
15

---

Published as a conference paper at ICLR 2024
(a) ground truth solution
for convection equation
00
08
00
(b) PINNS prediction
(c) FLS prediction
(d) QRes prediction
(c) PINNsFormer predic-
tion
Abookis
Aboshtaa
Abockite Bros
(1) PINNS error
(g) FLS error
(h) QRes error
(i) PINNsFormer error
Figure 6: Ground truth solution, predictions, and absolute errors (up to bottom) of PINNs, FLS, QRes, PINNsFormer (left to right) over convection equation.
Eaact MPCTI
03-
07
04-
24
06-
54
(a) ground truth solution for ID-reaction equation
Prodittdd MbL
Predicted MOLD
Poskted StU.
Predited 10C3
5.00
0224
002
2.0
(b) PINNS prediction
(c) FLS prediction
QRes prediction
PINNsFormer predic-
tion
Atten Errer
Azsduus
Eroor
Abooliia Ena
E12
35
04
24
2xc
(1) PINNs error
gg) FLS error
(h) QRes error
(i) PINNeFormer eтror
Figure 7: Ground truth solution, predictions, and absolute errors (up to bottom) of PINNs, FLS, QRes, PINNsFormer (left right) over ID-reaction equation.
16

---

Published as a conference paper at ICLR 2024
04
02-
04
(a) ground truth solution
for ID-wave equation
Pedidedd Mbed
Predetec
Thedicted alc
Prediteed
43
059
120
PINNs prediction
(c) PINNsFormer predie- (d) PINNs+NTK predic- FornerorK prediction
tion
tion
925
50
022
590 022 0.0 0.5 200
205
62
022
0/
142
630
0J5
L00
200
025
677
100
(1) PINNs error
PINN&Former error
(h) PINNs+NTK error
(i) PINN&Forrmr+4TTK er-
ror
Figure 8: Ground truth solution, predictions, and absolute errors (up to bottom) of PINNs, PINNS- Former, PINNs+NTK, PINNsFoorme=:NTK (left to right) over ID-reaction equation.
(a) solution for 2D
Navier-Stole equation
Predkted pocti
Prodictes
Prodictes
Preskcted
-1-
2.5
25
PINNs prediction
(c) QRes prediction
(d) prediction
(e) PINNsFormer predic- tion
Alsofusd briv
Absortt brie
ALsolide br12
AlrANI Bre
131
138
PINNs
QRes error
(h) FLS error
(i) PIN error
Figure 9: Ground truth solution, predictions, and absolute errors (up to bottom) of PINNs and PINNsFormer (left to right) over 2D Navier-Stokes equation.
17


Reviews:
[{"review_id": "khA229PJXQ", "reviewer": "Reviewer_PNtx", "summary": "This paper proposes PINNsFormer, a novel Transformer-based framework for Physics-Informed Neural Networks (PINNs). PINNs are used to numerically solve partial differential equations (PDEs) but struggle to capture temporal dependencies inherent in physics systems. PINNsFormer addresses this by generating pseudo input sequences from pointwise inputs and using a Transformer encoder-decoder architecture to model temporal dependencies. The main contributions are: 1) A new framework called PINNsFormer that equips PINNs with the ability to capture temporal dependencies through generated pseudo sequences and Transformer architecture. 2) A novel activation function called Wavelet designed to anticipate Fourier decomposition. 3) Extensive experiments showing PINNsFormer outperforms PINNs on problems involving failure modes and high-dimensional PDEs. 4) Demonstration of flexibility to incorporate PINNs optimization schemes for enhanced performance.", "questions": "1. One major advantage of PINNs is that, it leverages automatic differentiation rather than relying on finite difference approximations. On the other hand, the use of discrete pseudo-sequences means temporal dependencies are modeled in the fashion of finite difference approximations rather than pure automatic differentiation. Would it create any difficulties in picking parameters for this differentiation (e.g., the Δt)? \n\nA relevant thought: The impact of the timestep granularity Δt seems worth further analysis. Is there a study on model sensitivity to this parameter? Does the performance degrade if Δt is too small or large? Are there any guidelines for setting Δt?\n\n2. It may be worth trying the causal-attention transformers (decoder-based, e.g. LLM) instead of the encoder-decoder architecture. It does not seem to have particular reasons to adopt current sequence-to-sequence architecture.\n\n3. Other sampling schemes may also play an important role [1], especially for mitigating the temporal propagation failure in PINNs. Since PINNFormer relies on discretization on temporal dimensions, I am curious how PINNFormer can adapt to non-fixed sampling. I believe the transformer architectures have such flexibility.\n\n[1] Daw, Arka & Bu, Jie & Wang, Sifan & Perdikaris, Paris & Karpatne, Anuj. (2022). Rethinking the Importance of Sampling in Physics-informed Neural Networks. 10.48550/arXiv.2207.02338.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. **Novelty**: This is the first work I'm aware of that integrates Transformers with PINNs to capture temporal dependencies,  which is a novel and promising direction. Adapting Transformers designed for sequences to point-wise PINNs is non-trivial, thus the innovations in pseudo-sequence generation and loss formulation are important contributions.\n\n2. **Contributions**: The results on problems like convection and 1D-reaction demonstrate clear benefits in preventing temporal propagation failures that cripple vanilla PINNs. In addition, this work shows that modeling inter-timestep dependencies appears highly effective in maintaining accuracy across the domain rather than just near initial conditions.\n\n3. **Methodology**: The model components are well-motivated - the pseudo-sequence generation and Transformer encoding seem natural yet powerful ways to incorporate temporal modeling into PINNs.\n\n4. **Writing**: The paper is very clearly written, laying out both the background and proposed methodology comprehensively.", "weaknesses": "1. The lack of published code or detailed hyperparameters makes reproducibility difficult. Providing an implementation would strengthen the paper's contributions.\n\n2. While the overall approach is promising, some ablation studies would help determine the impact of different components like the pseudo-sequence generation and Wavelet activation.\n\n3. Since PINNs are notoriously slow in training, the computational overhead of PINNsFormer could be prohibitive for some large-scale applications. Analysis of model complexity and efficiency could help elucidate this issue.\n\n4. The reliance on introducing a discrete timestep risks undermining the automatic differentiation advantage of PINNs. Justification for this design choice could be expanded.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "27YPutzSqL", "reviewer": "Reviewer_ZyRg", "summary": "The manuscript describes a novel architecture for PINNs where a sequence with  even time steps is created.. This forms a sequence that can be used as an input to a transformer. The first layer of the neural network has a special \"wavelet\" non-linearity that can be seen as a spectral type of embedding of the data, with trainable parameters for the amplitudes and the frequency. An additional projection is made before the encoder and decoder. and the final output is generated with fully connected layer.\n\nAn attention mechanism is trained to produce the function value at   $ \\hat u( x , t_i + k \\Delta t )$ from the values   $ \\\\{ \\\\hat u( x , t_i + j \\Delta t ) \\\\}_{j\\in \\\\{1,..,k-1 \\\\}} $ \n\nThe described architecture is tested against a set of baselines with impressive results.", "questions": "See the weakness part.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Impressive results.  Clear representation.", "weaknesses": "The system is using a time discretised set of function values to predict the next time step. This reminds me of the finite difference method, In this case the stencil is 100 elements long, so the optimum stencil would have a very high order in accuracy. \n\nThe reason why very large stencils are not used is that these bear a computational cost, and the same happens in using attention, although most of the computing is parallel.\n\nNow the manuscript does not provide a baseline using a normal, discrete PDE solver, of course,  on the par with the  computational load that the attention mechanism is requiring . Of course, having an analytical, although complicated, solution for a problem has its advantages compared to a set of discrete nodal values.\n\nI would like that the authors would address this in their submission for better rating.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2cwfvioX5W", "reviewer": "Reviewer_mmgn", "summary": "This paper introduces PINNsFormer, a novel transformer-based framework for Physics-Informed Neural Networks (PINNs) to approximate solutions to partial differential equations (PDEs). PINNsFormer addresses the limitation of conventional PINNs in neglecting temporal dependencies within PDEs. Comprehensive experiments show PINNsFormer outperforms PINNs and variants in addressing failure modes and high-dimensional PDEs.", "questions": "see weakness above", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "PINNsFormer addresses a key limitation of PINNs by explicitly learning temporal dependencies, crucial for real-world physics systems. This significantly improves PINNs' generalization ability.\n\nThe proposed pseudo sequence representation and transformer architecture are clever approaches to adapt PINNs for sequential models.\n\nAblation studies provide insights into design choices and integration of existing PINNs schemes.", "weaknesses": "While the Wavelet activation function is theoretically justified to approximate arbitrary solutions, its advantages over other activations like ReLU, sigmoid, etc. require further empirical analysis and validation on practical problems. Conducting detailed empirical studies to evaluate Wavelet against various state-of-the-art activations under different settings can provide better insights into its benefits and limitations. This is important to fully understand its behavior and assess its effectiveness.\n\nThe paper only considers isotropic problems which have constant properties in all directions. However, most real-world physics systems exhibit anisotropic and nonlinear characteristics. Extending PINNsFormer to handle anisotropic problems modeled by direction-dependent PDEs, as well as nonlinear problems involving variable coefficients, would significantly broaden its applicability and demonstrate the approach's versatility. \n\nNo quantitative analysis was performed to evaluate important design choices like the pseudo sequence length and number of levels in coarsening. Without such ablation studies, it is difficult to justify critical hyperparameters and understand their impact on the model's performance as well as computational efficiency. These quantitative studies would provide further insights to validate the architectural design of PINNsFormer.\n\nAlthough various benchmark problems were tested, stronger validation would involve demonstrating the approach's effectiveness in entirely new physical domains beyond the existing test cases. Without such generalization to unseen problem classes, the claims regarding PINNsFormer's broad applicability remain partially unsubstantiated.\n\nWhile efficient on smaller problems, the inherent quadratic complexity of self-attention may pose scalability challenges for extremely large spatiotemporal datasets. Developing techniques to alleviate this computational limitation would enhance the method's practicality when dealing with massive real-world physics simulations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Tf0JrSeqiV", "reviewer": "Reviewer_dren", "summary": "This paper presents a method to utilize the well-known transformer architecture in PINNs. Unlike PINN’s point-to-point processing, the proposed method produces multiple outputs in a forward pass by injecting multiple input coordinates. The authors used a ‘pseudo-sequence generator’, which constructs a sequence of input coordinates consisting of a spatial coordinate and multiple time coordinates. The following transformer module models the dependency between input coordinates to generate the final outputs. They also introduce wavelet activation function that shows the effectiveness. The authors have tested on three different PDEs, and it shows the comparative performance.", "questions": "Questions are embedded in the section above.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1. Using transformers in PINN training is a promising research area. I appreciate the authors’ attempt to incorporate it into PINNs.\n2. The idea of processing multiple coordinates seems interesting and original.", "weaknesses": "1. I respectfully disagree with the author’s argument that the original PINN neglects temporal dependency. PINN takes temporal coordinates and spatial coordinates together and generates the output. By going through multiple layers in MLP, the time coordinate will definitely affect the features from the spatial coordinates. I agree that the suggested method might be able to model time dependency more explicitly. However, the argument that the original PINN is not capable of modeling time dependency is too strong.\n\n2. As a follow-up comment, we sample many random collocation points at each iteration, and MLP can see many time coordinates with spatial coordinates during training. Hence, I believe the original PINN is capable of modeling time dependency.\n\n3. Is delta t fixed? if yes, then it might be a not trivial limitation, considering different time granularity at different time coordinates.\n\n4. If I understand correctly, it seems that Spatio-Temporal Mixer is just one layer MLP. I might have missed something, but the technical details are not properly described in explaining each component. The formal definition of each module would be appreciated.\n\n5. The proposed Wavelet activation function seems to be a simplified version of the positional encoding. In addition, IMHO, the sine activation function, followed by MLP, which attaches weights to each neuron, could do the same thing. cos function can also be easily expressed by a bias term from the previous layer e.g., sin(x) = cos(90-x).\n\n6. My main concern is a weak experimental setup and results. The authors presented the results of simple three PDEs, which already have been tackled by numerous works. And, there are many previous works that achieved better results (lower relative errors) [1]. \nThe authors highlighted ‘very low loss’ values. Loss depends on loss function and hyperparameters, which cannot be a fair metric.\nThe Navier-stokes experiment used in this paper is too simple. Please consider using examples tested in [2] and [3].\n\n[1] An expert guide’s to training physics-informed neural networks, Wang et al., arXiv 2023.\n\n[2] Respecting causality is all you need for training physics-informed neural networks, Wang et al., arXiv 2022.\n\n[3] Separable physics-informed neural networks, Cho et al., NeurIPS 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)