Sub-Sequential Physics-Informed Learning with State Space Model
Authors: Chenhui Xu
Dancheng Liu
Yuting Hu
Jiajie Li
Ruiyang Qin
Qingxiao Zheng
Jinjun Xiong
Venue: ICML 2025 poster (2025)
Keywords: Physics-Informed Neural Networks
State Space Model
Primary area: applications->chemistry_physics_and_earth_sciences
TL;DR: This paper introduces PINNMamba, a novel state-space-based sub-sequence learning framework for physics-informed neural networks.

Abstract: Physics-Informed Neural Networks (PINNs) are a kind of deep-learning-based numerical solvers for partial differential equations (PDEs). Existing PINNs often suffer from failure modes of being unable to propagate patterns of initial conditions. We discover that these failure modes are caused by the simplicity bias of neural networks and the mismatch between PDE's continuity and PINN's discrete sampling. We reveal that the State Space Model (SSM) can be a continuous-discrete articulation allowing initial condition propagation, and that simplicity bias can be eliminated by aligning a sequence of moderate granularity. Accordingly, we propose PINNMamba, a novel framework that introduces sub-sequence modeling with SSM. Experimental results show that PINNMamba can reduce errors by up to 86.3\% compared with state-of-the-art architecture. Our code is available at Supplementary Material.

Sub-Sequential Physics-Informed Learning with State Space Model
Chenhui Xu'
Dancheng Liu
Yuting Hu
Jiajie Li
Ruiyang Qin
Qingxiao Zheng'
Jinjun Xiong'
Abstract
Faast
Proditedd
Ineditted
Physics-Informen Neural Networks (PINNs) are a kind of deep-lcarning-basee numerical solvers for partial differential equations (PDEs). Exist- ing PINNs often suffer from failure modes of be- ing unable propagate patterns of initial con- ditions. We discover that these failure modes are caused by the simplicity bias of neural net- works and the mismatch between PDE's con- tinuity and PINN's discrete sampling. We re- veal that the State Space Model (SSM) can be a continuous-diseree articulation allowing initial condition propagation, and that simplicity bias can be eliminated by aligning a sequence of mod- erate granularity. Accordingly, Mamba, novel framework that introduces sub- sequence modeling with SSM. Experimental re- sults show that PINNMamba can reduce errors by up to 86.3% compared with state-of-the-ann architecture. Our code available at https: // github. cow/ min HuiHuil INNMamba.
(a) Ground Truth
(b) PINN
(c) PINNMamba
Figure PINN gradually distorts convection equation.
approaches like finite element methods (Reddy, 1993).
The mainstream PINNs predominantly employ multilayer perceptrons (MI.Ps) as their backhone architecture. How- ever, despite the universal approximation capability of MLPs, they do not always guarantee the accurate learning of numerical solutions to PDEs in practice. This phenomenon is observed as the failure modes in PINNs, in which case PINN provides a completely wrong approximation (Krish- napriyan et al., 2021). As illustrated in Fig. 1, the failure modes often manifest as a temporal gradual distortion. This distortion arises because MLPs lack the necessary inductive bias effectively capture the temporal dependencies of system, ultimately hindering the accurate propagation of physical patterns informed by the initial conditions.
propose PINN-
1. Introduction
To introduce such an inductive bias, several sequence-to- sequence approaches have been proposed (Krishnapriyan et al., 2021; Zhao al., 2024; Yang et al., 2022; Gao et al., 2022). Specifically, Krishnapriyan training new network at each time step and recursively using its output as the initial condition for the next step. This method incurs significant computational and memory overhead while exhibiting poor generalization. Further- more, Transformer- based approaches (Zhao et al., 2024; Yang et al., 2022; Gao et al., 2022) are proposed to address the timedeeenndnncy issue. Yet, these methods are based discrete sequences, making their model produce incor- rect pattern mutations in some cases due to their ignorance of the basic principle that PINNs approximate continuous dynamical systems. Thus, there is still an open question:
In the past few years, Physics-Informed Neural Net- works (PINNs) (Raissi et al., 2019) have emerged as a novel approach for numerically solving partial differential equa- tions (PDEs). PINN takes a neural network uy (2 (x, whose parameters 0 are trained with physics PDE residual loss, the numerical solution u(z, of the PDE, where and are spatial and temporal coordinates. The core idea behind PINNs is to take advantage of the universal approximation property of neural networks (Hornik et al., 1989) and au- tomatic differentiation implemented by mainstream deep learning frameworks, such PyTorch (Paszke et al., 2019) and Tensorflow (Abadi et al., 2016), that PINNs can achieve potentially more precise and efficient PDE solu- tion approximation compared with traditional numerical
al. (2021) propose
How can we effectively introduce sequentiality PINNs?
University Buffalo, SUNY PUniversity of Notre Dame. Correspondence to: Chenhui Xu <cxu266@ buffalo.edu>.. Jinjun Xiong <jinjun @ buffalo.edu>.
To answer this question, we need to understand the essential difficulties of training PINNs. First, PINNs assume tem- poral continuity, whereas, during their actual training, the aattiotemppral collocation points used to construct the PDE residual loss are sampled discretely. We define this nature of
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Sub-Sequential Physics-Informed Learning with State Space Model
PINN as Continuons-Disered Mismatch. In the absence of well-defined continuous- articulation, the real trajec- tory of physical system is not necessarily recovered correctly Mis-
Former (Zhao et al., 2024) and KAN (Liu et al., 2024c) on multiple hard problems, achieving new state-of-the-art.
Contributions. We make the following contributions:
in the training process, since such Continuous-Diss match would block the propagation of the initial condition.
We reveal that the mismatch between the discrete na- ture of the training collocation points and the continu- ous nature of the function approximated by the PINNs is an important factor that prevents the propagation of the initial condition pattern over time in PINNs.
To respect the inherent Continuous- Discrete Mismatch, we reveal that the State Space Models (SSM) (Kalman. 1960) can be a good continuous-diseree articulation. SSMs para- metrically model discrete temporal sequence continu- ous dynamic system. An SSM's 's discrete form approximates the instantaneous states and rates of change of its continuous form via integrating the system's 's dynamics over each time interval, which more accurately responds to the trajectory of a continuous system. Meanwhile, the SSM unifies the scale of derivatives of different orders, making it easier to be optimized. So far, SSMs have shown their insane capacity in language (Gu & Dao, 2023) and vision (Liu et al., 2024a) tasks, but its potential for solving PDEs remains unexplored. We propose to construct PINNs with SSMs to unleash their excellent properties of continuous-diseree articulation.
We also note that the simplicity bias of neural networks is a key contributing factor to the over-smoothing pat- tern that causes gradual distortion in PINNs.
We propose PINNMamba, which eliminates the discrete-continuity mismatch with SSM and combats simplicity bias with sub-sequential modeling, resulting in state-of-the-art on several PINN benchmarks.
2. Related Works
Physics-Informed Neural Networks. Physics-Informe Neural Networks (Raissi et al., 2019) are class of deep learning models designed to solve problems governed by physical laws described in PDEs. They integrate physics- based constraints directly into the training process in the loss function, allowing them to numerically solve many key physical equations, such as Navier-Stokes equations(Jin et al., 2021), Buler equations (Mao et al., 2020), heat equatuons (Cai et al., 2021). Several advanced learning schemes such as gPINN (Kharazmi et al., 2019), vPINN et al., 2022), and RoPINN(Wu et al., 2024), model archi- tectures such as QRes (Bu & Karpatne, 2021), FLS (Wong et al., 2022), PINNsFormer (Zhao et al., 2024), KAN (Liu et al., 2024c:b) are proposed in terms of convergence, opti- mization, and generalization.
Next, we remark that the simplicity bias (Shah et al., 2020) of neural networks is another crucial contributing factor to PINN training difficulty. The simplicity bias will lead the model to choose the pattern with the simplest hypothesis. This results in an over-smoothed solution when approximat- ing PDEs. Because, for data-free PINNs, there might be a very simple function in the fcasible domain that can make the residual loss zero. For example, for convection equation, u(z,t) leads to zero empirical loss on every collocation point except when f O. While the correct pattern is hard to fight against over-smoothed patterns during training.
A major way to eliminate simplicity bias is to construct agreements over diversity predictions (Teney et al., 2022). Following this principle, we propose a novel sub-sequence alignment approach, which allows the diverse predictions of time-varying SSMs to form such agreements. Sub-sequence modeling adopts medium sequence granularity, forming the time dependency that small sequence fails to capture while avoiding the optimization problem along with the long sequence. Meanwhile, the alignment of the sub-sequence predictions ensures the global pattern propagation and the formation of an agreement that eliminates simplicity bias.
Failure Modes in PINNs. Despite these efforts, PINN still has some inherently intractable failure modes. Krish- napriyan et al. (2021) identify several types of equations that are vulnerable to difficulties in solving by PINNs. These equations are usually manifested by the presence of a pa- rameter in them that makes their pattern behave as a high frequency or a complex state (Cho al., 2024), failing to propagate the initial condition. In such cases, an empirical loss constructed using a collocation point can easily fall into an over-smooth solution (c.g. a(z t) = 0 can make the loss of all collocation points except whose = 0 descend to 0 for I wave equations). Several methods regarding optimiza- tion (Wu al., 2024; Wang et al., 2022a), sampling (Gao et al., 2023; Wu et al., 2023), model architecture (Zhao et al., 2024; Cho et al., 2024: Nguyen et al., 2024b), transfer learning (Xu et al., 2023: Cho et al., 2024) are proposed to mitigate such failure modes. However, the above ap- proaches do not focus on the fact that a PDE system should be modeled as a continuous dynamic, leading to difficulties in generalization over wide range of problems.
In this paper, we introduce a novel learning framework to solve physics PDE's 's numerically, named PINNMamba, which performs time sub-sequences modeling with the Se- lective SSMs (Mamba) (Gu & Dao, 2023). PINNMamba successfully captures the temporal dependence within the PDE when training the continuous dynamical systems with discretized collocation points. To the best of our knowledge, PINNMamba is the first data-free SSM-based model that effectively solve physics PDE. Experiments show that PINN- Mamba outperforms other PINN approaches such as PINNs-
2

---

Sub-Sequential Physics-Informed Learning with State Space Model
State Space Models. The state space model (Kalman, 1960) is mathematical representation of physical system in terms of state variables. Modern SSMs (Gu et al., 2022; Smith et al., 2023; Gu & Dao, 2023) combine the repre- sentational power of neural networks with their own supe- rior long-range dependency capturing and parallel comput- ing capabilities and thus are widely used in many fields, such as language modeling (Fu et al., 2023; Poli et al., 2023; Gu & Dao, 2023; Dao & Gu, 2024), computer vi- sion (Zhu et al., 2024; Liu et al., 2024a), and genomics (Gu & Dao, 2023; Nguyen et al., 2024a). Specifically, Structured SSMs (S4) (Gu et al., 2022) decomposing the structured state matrices as the sum of a low-rank and normal terms to improve the efficiency of state-space-based deep models. Further, Selective SSMs (Mamba) (Gu & Dao, 2023) elim- inates the Linear Time Invariance (Sain & Massey, 1969) of SSMs by introducing gating mechanism, allowing the model to selectively propagate or forget information and greatly enhancing the model performance. In physics, SSMs are used in conjunction with Neural Operator to form a data-driven solution to PDEs (Zheng et al., 2024; Hu et al., 2024). However, these methods are data-driven which lack generalization ability in some scenarios where real data is not available. Unlike these methods, our approach, PINN- Mamba is fully physics-driven, relying only on residuals constructed using PDEs without any training data.
Predicted ubxt)
Absolute Enrr
65
3015
-225
3.000
-0.
2.005
-275
-1.0
2.000
Figure 2. Failure mode of PINN on convection equation, the over- smooth solution brings the losses down to 0 almost everywhere.
Ls((u) = [ax
Σ
[IB(ug(z;,
4:)1: (6)
[(uo) = AF + >I + XgLi(uu),
(7)
where λ5.dT,d are the weights for loss that are adjustable by auto-balancing hyperparameters. ||-|| denotes norm.
State Space Models. An SSM describes and analyzes a continuous dynamic system. It is typically described by:
h((t) = Ah(() + Bx((), u(I) - Ch(() + Dx((),
(8)
(9)
where h(t)) is hidden state of time t, hi(t) is the derivative of h(t). x(I) is the input state of time 4, u(l) is the output state, and A, B, C, D are state transition matrices.
3. Preliminary
In real-world applications, we can only sample in discrete time for building deep SSM model. We usually omit the term Dx(t) in deep SSM models because it can be easily implemented by residual connection (He et al., 2016). So we create discrete time counterpart:
Physics-Informed Neural Networks. The PDE systems that are defined on spatio-temporal set Ω x [0, T] C Rd+1 and described by equation constraints, boundary conditions, and initial conditions can be formulated as:
Flu(z, ()) = 0, () Ω [0,T];
(1)
h Ahg-1 + Bxk:
(10)
u2 Ch..
(11)
I(u(a, 1)) = 0, 0, () E x (0); B(u(x, t)) = 0, ((r E a2 x [0,T],
(2)
(3)
with discretization rules such as zero-order hold (ZOH);
where u: : Rd+1 + RTT is the solution of the PDE, x E Q is the spatial coordinate, an is the boundary of 22 E [] is the temporal coordinate and T is the time horizon. The F, I, B denote the operators defined by PDE equations, initial conditions, and boundary conditions respectively.
= e (AA), B (AA) (exp - 1) AB,
(12)
(13)
where À and B is discrete time state transfer and input matrix, and A is a step size parameter. By parameterizing A using HiPPO matrix (Gu et al., 2020), and parameterizing (A, B, C)) with input-dependency, a time-varying Selective SSM can be constructed (Gu & Dao, 2023). Such a Selective SSM can capture the long-time continual dependencies in dynamic systems. We will argue that this makes SSM good continuous-discrete articulation for modeling PINN
A physics-driven PINN first builds a finite collocation point set X C Ω [o, 7), and its spatio (temporal) boundary 0x C 02 x [0. 2] (Xo 52 (0)), then employs a neural network uo(x, t) which is parameterized by 0 to approximate u(x,t) by optimizing the residual loss as defined in Eq. 7:
Cu() = Ixl
Σ (xx
[F(uo t:11
(4)
4. Why PINNs Present Failure Modes?
Lz (u6)))
1
Σ
(5)
A counterintuitive fact of PINNs is that the failure modes are not devoid of optimizing their residual loss to a very low
Ixol
(xxx)))xx
3

---

Sub-Sequential Physics-Informed Learning with State Space Model
Simple Patten
Complec
the information of the equation determined by the initial conditional differential operator I may act only on a small neighborhood of collocation points with & = 0. The other collocation points in the S2 x (0, T. on the other hand, might fall into local optimum that can make Lr(u)) defined by Eq. 4 to near O. Because the function Wy determined by F and together on the collocation points at l = 0 may not be generalized outside its small neighborhood. The detailed proof of Theorem 4.1 can be found in Appendix A.
++ Lagey 26,
Correct Pamtr
O Svoothe Pattern
au a
(a) Discrete collocation prevents the correct pattern propagate.
(b) Model chooses simplest pattern due Simplicity Bias.
Simplicity Bias. Meanwhile, the simplicity bias of neural networks will make the PINNs always tend to choose the simplest solution in optimizing (=(u)). This implies that PINN will easily fall into an over-smoothed solution. For example, as shown in Fig. 2, the PINN's prediction is O in most regions. The loss of this over-smoothed feasible solution is almost identical to that of the true solution, and the existence of an insurmountable ridge between the two loss basins results in a PINN that is extremely susceptible to falling into local optimums. As in Fig 3. the over-smoothed pattern yields an advantage against the correct pattern.
The correct Pattern determined by the initial conditions faces two resistances in propagation: (a) the difficulty of propa- gating information directly through the gradient among discrete collocation points, and (b) the need to fight against over-smoothed solutions with near-zero loss caused by simplicity bias.
Figure 3.
degree. As shown in Fig. 2, for the convection equation, the converged PINN almost completely crashes in the domain, but its loss maintains near-zero degree at almost every col- location point. This is the result of the combined effects of the simplicity bias (Shah et al., 2020; Pezeshki et al., 2021) of neural networks and the Continuous-Diseree Mismatch of PINNs, as shown in Fig. 3. The simplicity bias is the phenomenon that the model tends to choose the one with the simplest hypothesis among all feasible solutions, which we demonstrate in Fig. 3 (b). Continuous--Dscreed Mismatch refers to the inconsistency between the continuity of the PDE and the discretization of PINN's training process. As shown in Eq. 4 4 6, to construct the empirical loss for the PINN training process, we need to determine a discrete and finite set of collocation points on Ω x [o, T). This is usually done with a grid or uniform sampling. But PDE system is usually continuous and its solutions should be regular enough to satisfy the differential operator F, B, and Z.
Under the effect of difficulty in passing locally correct pat- terns the global due to Comtnnonss-iserete Mismatch and over-smoothing due simplicity bias, PINNs present failure modes. Therefore, to address such failure modes, the key points in designing the PINN models lie in: (1) mech- anism for information propagation in continuous time and (2) a mechanism to eliminate the simplicity bias of models.
5. Combating Failure Mode with State-Space Model and Sub-sequential Alignment
To address the problems in Section 4, we propose (1) a dis- crete saate-space-base encoder that models the sequences of individual collocation points in continuous dynamics, to match with Continuons-Disrreee Mismatch, and propagates the information from the initial condition to subsequent times (Section 5.1). and (2) sub-sequence contrastive alignment mechanism that aligns different outputs of the same collocation point in different sub-sequences, to form an agreement that climinates simplicity bias (Section 5.2).
Madinuous-Diserede Mismatch. Maatinuouu-Discrett Mis- match will cause correct local patterns hard to propagate over the global domain. Because the loss on discrete col- location points does not necessarily respond to the correct pattern on the continuous domain, instead, only responds to its small neighborhood. To show such Santinuous-Discreet Mismatch, we first present the following theorem:
(i,,, (#'x, (x)) C [0, T. Then for differential operator M there exist infinitely many functions Ω + R. parametrized by 0, s.t.
Theorem 4.1. Let x*
5.1. Continuous Time Propagation of Initial Condition Information with State Space Model
As we discussed in Section 4, the -Ditsrrru-- Mis- match of PINNS raises the intrinsic difficulty of modeling, since the time dependency in dynamic PDE system not captured spontaneously by discrete sampling. We argue that such dynamic time dependency can be modeled by SSM. To this end, we first consider the PDE as a spatially infinit-dimmnsioe ODE to simplify the problem. We view the solution uo(2, t) in a function space that, if we let:
(Muoaa) 0 fori lari=,...,,A. N, M(uo(z, t)) #0 for a.. x E xΩx \x*
By Theorem 4.1, enforcing the PDE only at finite set of points does not guarantee a globally correct solution. This can be performed by simply constructing a Bump function in a small neighborhood of points in x" so that it satisfies M(uo( ,t") = for (x",t") E x" This means that
U()) = uo(.
(14)
4

---

Sub-Sequential Physics-Informed Learning with State Space Model
Sub-SSquence Prediction
Collocation Point (K.I)
Lx
Time t Sub-Sequence Prediction
Linear Projection
Output
Multi Layer Perceptron
uisa
Multiplication
SSM
Activation
PINNMamba Enceder
Wavelet
Wavelet
Sub- Soquence Contrastive Loss
Collocation Points
Conv
Spatio-Teune Temporal Embedding
H(N2+201)
aixu+San
Sub-S Tub-Sequence
Time år Sub-Sequenc Prediction
PINNMamba
PINNMamba Encoder
Sub-Sequence Alignment
Figure 4. PINNMamba Overview. PINNMamba takes the sub-sequence as input which is composite of several consecutive collocation points on the time axis. For each sub-sequence, the prediction of the first collocation point is taken as the output of PINNMamba, while the others are used to align the prediction of different sub-sequences, that can propagate information among time coordinates.
be a function x + ug t). by M Mpponnt spatial sampling:
Sequence
Collocation Point
Pseudo Collocation Point
U,(t) == u(z,, t),
(15)
Pseudo Sequenze
u(t = (Ui(t, .M (UM()))))))).,,((()))))))
(16)
Sub Sequence
Long Sequence
Sequential Modeling Continuity with SSM. In continu- ous time, we now model the function u(t) to the dynamic system described by SSM as in Eq. and 9. Here we let x(t) Embed(ar, t), where Embed(.) is the Spatio- Temporal Embedding Fig After temporal discretization uu u(kAt), h4 h(kAl) and Xx= x(k:Al), we get:
Figure 5. Comparison of Sequence Granularity
Thereby, for each collocation point, the numerical value of its solution should be jointly optimized by Eq. 18, 19, and 20, thus receiving the pattern defined by the initial conditions.
k
Uniformed Derivatives Scale. Another benefit that can be got from SSM is, by parameterizing differential state matrix A in Eq.8 with HiPPO matrix (Gu et al., 2020) which con- tains the derivative information, we can align the derivatives of the system with respect to time on a uniform scale. This uniform scale will help to reduce the problem of ruggedness on the loss landscape due gradient vanishing or exploding.
u = Co + CC A
(17)
1=0
Reversibly, by the inverse of the discretization rule defined by Eq. 12, 13, we can restore this temporal dependency to continuous time. This kind of restoration can help achieve PINN's generalization to any moment in [0, T).
Pattern Propagation by Joint Optimization. Combine Eq. 4 with 17, in sequence start with & = 0, the sum of loss of collocation points at time ktt, would be:
Time- Varying SSM. In practice, we use the time-varying Selective SSM (Gu & Dao, 2023), instead of the function defined by Eq. 17 being the SSM on a linear invarant system. The time-varying SSM has two advantages, one is that such input-dependent models typically have stronger representational capabilities (Xu al., 2024), while the other is that it will make diverse predictions that help to eliminate simplicity bias in the model, section 5.2. This time-variance will make (A, B, C)) time- dependent, and therefore, Eq. 17 and 18 need minor adjust- ments. These adjustments won't 't impact the initial condition propagation, and we will discuss them in Appendix B.
M Σ Co(u(a, M |F uz)ll
IM (CA'h + C k ||F
c s
(t-iBx;)) Bxx))
11,, (18)
will discuss in
MM
where 1M [1,1,
RM In Eq. 18, we notice that the ho should satisfy both the initial condition and the equation by jointly optimizing the losses:
5.2. Eliminating Simplicity Bias of Models with Sub-Sequence Contrastive Alignment
1
C)()))
M |F (IM- (Cho)) 11:
(19)
Although SSM can make the information about the initial conditions propagate time coordinates, it still cannot mit-
Lz(10) M [I (Cho)) IF.
(20)
5

---

Sub-Sequential Physics-Informed Learning with State Space Model
igate the simplicity bias of neural networks. The model is still prone falling into over-smoothed local optimum. There are two key points to address this over-smoothness caused by simplicity bias: (1) appropriate sequence gran- ularity to guarantee a smooth optimization process. (2) Mitigating the effect of simplicity bias through the diversity of model prediction paradigms (Pagliardini et al., 2023).
spatio-temporal domain bounded by the PDE. We assume that the grid picks M spatial coordinates and N temporal coordinates, and denote the temporal sampling interval as At = T/(N 1). For a collocation point (z, t),, we con- struct a sequence ((x, () with its ks - 1 temporal successors:
X ((e, 1) = [(x, (1,, AD), ... ,,,,,.,,,,,,,,,, (2,, + (k- 1)-()}-
(21)
Sequence Granularity, A proper sequence granularity en- sures smooth propagation of the initial conditions while making the model easier to optimize. As shown in Fig. 5, there are three ways to define sequence, which are pscudo sequence (Zhao et al., 2024), long sequence (Nguyen et al., 2024a), and the proposed sub-sequence. We propose to use sub-sequence with medium granularity overlapping. The sub-sequential modeling can avoid: (I) the difficulty of crossing the loss barrier that makes the model trapping in the over-smooth local optimum, which is caused by the huge inertia of long sequence; (2) the difficulty of broadcasting information globally on the time coordinate, that caused by construct on small neighborhoods of a collocation point in pseudo sequence. Sub-sequence takes only the first output in the sequence as the output value of the current collocation point. Its successors' values will pass information crossing the time coordinate through subsequences alignment and form diverse predictions to eliminate simplicity bias.
PINNMamba takes such M x N sequences as the in- put of models. For each sequence X X(x, 1), PINNMamba computes a sub-sequence prediction (utg (1r, (z t + At), 8/ (z, t + (k 1) corresponding to every collocation point in the sequence, where ( t i de- note the tentative prediction of collocation point (1,, ( in a sequence start with time t. The (, t) will be taken as the output of collocation point (ar, () and the rest of the sequence will be used to construct the sub-sequence con- trastive alignment loss we will discuss later in Section 5.2. The residual losses of the model w.r.t the sub-sequence will be:
4 (wo) xi
Σ
Σ
(22)
(xx
(uo) = k|xol
Σ
(o, (23)
(xx
Contrastive A Alignment for Information Propagation. As shown in Fig. 5, we construct a sub-sequence for each collo- cation point together with its finite successors, which form overlapping collocation points. By aligning the predictions of these collocation points with a contrastive loss, each col- location point becomes soft relay of the pattern. Thus, it forms the propagation of patterns in the whole time domain.
2 (uo)
Σ (ELX)))EXx
Σ BBu (z)tt+ 1.. (24)
k
Model Architecture. As shown in Fig. 4, PINNMamha employs an encoder-only architecture, which encodes fixed- size input sub-sequence into a sub- uub-seqeence prediction with the same length. First, for each token in the sequence, an MLP-basce Spatio-Tempont Embedding layer first embeds the (z, E) coordinates into high-dimensional representation. The embeddings will be sent to Mamba-based encoder, which consists of several PINNMamba blocks.
Eliminating the Simplicity Bias. Previous work (Teney et al., 2022; Pagliardini et al., 2023) has pointed out that the agreement obtained from diverse predictions is the key to climinating the effects of simplicity bias. We argue that this agreement from diverse predictions is naturally obtained in the sub-seeuence alignment. This is because the fact that, since the SSM we constructed in section 5.1 is time-varying and a collocation point will be at different time coordinates in different aub-sequences, the predictions for this colloca- tion point are naturally diverse. And we force these diverse predictions arrive a consensus by contrastive alignment.
The PINNMamba block employed here consists of two branches: (1) the first a stack of a linear projection layer, Id-convolution layer, Wavelet activation (Zhao et al., 2024), and an SSM layer with parallel scan (Gu & Dao, 2023); (2) the second is a stake of a linear projection layer and a Wavelet activation. The two branches are then con- nected with an element-wiss wise multiplication, followed by another linear projection and residual connection. With input X', the PINNMamba block can be formulated as:
6. PINNMamba
In conjunction with the high-level ideas described in Sec- tion 5, in this section, we present PINNMamba, a novel physics-informed learning framework that effectively com- bats the failure modes in the PINNs.
x = (CMM((WConn(WW"
(25)
X = o(WxX))
(26)
x 1+1 x' + w. o x²),
(27)
where (z)) = W sin(2) + W2 cos(x: is Wavelet activation function (Zhao et al., 2024). in which w1, W2 are learnable. e9 denotes an element- wise multiplication.
lub--equential I/O. As shown in Fig. 4, PINNMamba first samples the grid of collocation points over the entire
6

---

Sub-Sequential Physics-Informed Learning with State Space Model
Poodcted
Prodiced
Esed 2020
-43
PINN
QRcs
PINNsFormer
KAN
PINNMamba
Figure 6. The ground truth solution, prediction (top), and absolute error (bottom) convection equations.
Sub-Sequence Contrastive Alignment. PINNMamba pre- dicts the same collocation multiple times in different sub- sequences. For example, the collocation point (I, f. + k&t) appears on sequences from X (ar + At) to X (z, f + k&t). We align the predictions on these subsequences to make the information defined by the initial conditions propagate over time. To do this, for each subsequence, we design a contrastive loss with the last subsequence for alignment:
strengthening the boundary conditions lead to better convergence. Aalig is set to 1000 for convection and reaction equations, and auto-adapted by } for wave equation. All experiments are implemented in Py 2.1.1 and trained on an NVIDIA H100 GPU. More training details are in Appendix Our code and weights are available at https: / //github. com iumm/miniiiiHiii Hui / INMMmmb..
Metries. To evaluate the performance of the models, we take relative Mcan Absolute Error (rMAE, a.k. & relative error) and relative Root Mean Square Error (rRMSE, a.k.a Es relative error) following common practive (Zhao et al., 2024; Wu et al., 2024). The metrics are formulated as:
1
Calig(uo) = (k x)
t ug
(Iz, ti jDtt
(x j=1
1 At (I + }
ug
(28)
E==1 l (In, (n) (Ins tn]
rMAE (ū)
(30)
N Σm= 1 Iu (I t.
Thus, the empirical loss for PINNMamba is defined as:
[(ue) = >x A + As (ua) Canig (ue).
N 1 l (I:n (in) u (In bn))
(29)
IRMSE (u)
(31)
E (Jns 1r))
7. Experiments
where N is the number of test points, u(x, t) is the ground truth solution, and á(x;, f) is the model's prediction.
Setup. We evaluate the performance of PINNMamba on three standard PDE benchmarks: convection, wave, and reaction equations, all of which are identified as being af- fected by failure modes (Krishnapriyan et al., 2021; Zhao et al., 2024). The details of those PDEs can be found in Appendix C. We compare PINNMamba with four baseline models, vanilla PINN (Raissi et al., 2019), QRes (Bu & Karpatne, 2021), PINNsFormer (Zhao et al., 2024), and KAN (Liu et al., 2024c). For fair comparison, we sample 101 x 101 collocation points with uniformly grid sampling, following previous work (Zhao et al., 2024; Wu et al., 2024). We also evaluate on PINNacle Benchmark (Hao et al., 2024) and Stvier-Stokes equation (Raissi al., 2019).
7.1. Main Results
We present the rMAE and rRMSE for approximating con- vection, reaction and wave equation's 's solution in Table 1. Our model consistently outperforms other model architec- tures, achieving new ofate-of-the-art. Notably, shown in Fig. 6, for the convection equation, PINNMamba allows sufficient propagation of information about the initial con- ditions, whereas on all the other models there is a varying degree of distortion in the time coordinates. As shown in Fig. 8, PINNMamba can further optimize at the boundary, resulting in a lower error than KAN and PINNsFormer for reaction equations. For problems as intrinsically difficult to optimize as the wave, as in Fig. 7, PINNMamba effectively combats simplicity bias and aligns the scales of multi-order differentiation, and thus achieves significantly higher ac- curacy. This illustrates that PINNMamba can be effective against PINN's failure modes. It's also worth noting that,
Training Details. We train PINNMamba and all the base- line models 1000 epochs with L-BFGS optimizer (Liu & Nocedal, 1989). We set the sub-sequence length to 7 for PINNMamba, and keep the original for PINNsFormers. The weights of loss terms [Ax, Λz, As] are set to [1, 10) for all three equations, as we find that
setup
7

---

Sub-Sequential Physics-Informed Learning with State Space Model
Table 1. Results for solving convection, reaction, and wave equations.
Convection
Reaction
Wave
Model
#Params
Loss
rMAE
#RMSE
Loss
rMAE
rRMSE
Loss
rMAE
ARMSE
PINN
527361
0.0239
0.8514
0.8989
0.1991
0,9803
0.9785
0.0320
0.4101
0.4141
QRes
396545
0.0798
0.9035
0.9245
0.1991
0,9826
0.9830
0.0987
0.5349
0.5265
PINNsFormer
453561
0.0068
0,4527
0.5217
3e-6
0.0146
0,0296
0.0216
0,3559
0,3632
KAN
891
0.0250
0.6049
0.6587
7e-6
0.0166
0.0343
0.0067
0.1433
0.1458
PINNMamba
285763
0.0001
0.0188
0.0201
le-6
0.0094
0.0217
0.0002
0.0197
0.0199
Indiceed
Iredited Bocn
Predimed ubud
Inddited LECO
Iradeted sboc
-05
34
15
24
14
M
16
14
34
16
ABtc UNe Oree
Nocke Frve
Abs0 Ever
15
M
04
14
16
02
04
15
00
24
04
38
PINN
QRes
PINNsFormer
KAN
PINNMamba
Figure 7. The ground truth solution, prediction (top), and absolute error (bottom) on wave equations.
Neural Tangent Kernel. Dynamic tuning of losses via Neural Tangent KerneI(NTK) (Wang et al., 2022b) has been shown to have the effect of smoothing out the loss land- scape. PINNMamba also works well with the NTK-adopted loss function. As shown in Table 2, NTK can reduce PIN- NMamba error by 5-25%. The combination of RoPINN and NTK can further improve the overall performance of PINNMamba, which demonstrates the excellent suitability of PINNMamba with other PINN optimization methods.
Table Integrating PINNMamba with advanced training strategics and loss auto-balancing strategy. The rMAE reported here.
Method
Convection
Reaction
Wave
PINNMamba
0.0188
0.0094
0.0197
+gPINN
0.0172
0.0123
0.0264
+vPINN
0.0236
0.0092
0.0169
+RoPINN
0.0102
0.0099
0.0121
+NTK
0.0179
0.0079
0.0147
+NTK+ROPINN
0.0127
0.0072
0.0106
7.3. Loss-Error Consistency Analysis
PINNMamba has the lowest number of parameters (except KAN), while achieving consistently the best performance.
Our other interest is the role of PINNMamba for the elim- ination of simplicity bias. Models affected by simplicity bias that fall into over-smoothing solutions will show in- consistent decreasing trends in loss and error during train- ing. As shown in Fig. 9, in the training process for solving convection equations, the rMAE of PINN doesn't descend as Ly and Lr. This suggests that PINN is trapped in an over-smoothing solution, which is in agreement with our observation in Fig. 6. As comparison, we find that PIN- NMamba's losses descent processes show high degree of consistency with its error descent process. This indicates that PINNMamba does not tend to fall into local opti- mum of oversimplified patterns. Instead, it tends to exhibit patterns that are consistent with the original PDEs.
7.2. Combination with Other Methods
Since PINNMamba mainly focuses on model architecture, it can be integrated with other methods effortlessly. We explore the feasibility and their performance in combination with advanced training paradigm, well as loss balancing.
Training Paradigm. We show the rMAE of PINNMamba when integrated with advanced strategies in Table 2. We observe that gPINN (Yu et al., 2022) and vPINN (Kharazmi al., 2019) erratically deliver some performance gains on some tasks. This is due to the fact that the regularization provided by gPINN and vPINN in the form of loss function through the gradient and variational residuals has little effect on PINNMamba, since SSM itself is sufficiently regularized. RoPINN (Wu et al., 2024) reduces the PINNMamba's 's error on convection and wave equations by about 40%, since it complements the spatial continuity dependency.
7.4. Ablation Study
To verify the validity of the various components of the PIN- NMamba, as shown in Table 3, we evaluate the performance of models subtracting these components from PINNMamba.
8

---

Sub-Sequential Physics-Informed Learning with State Space Model
Predeted
C7
042
M
12
6.0
503
04
-030
-11
PINN
Qees
PINNsFormer
KAN
PINNMamba
Figure The ground truth solution, prediction (top), and absolute error (boltom) on reaction equations.
PINN
PaNNWa
10.
10"1
1800 Blandors 19T
Ledt Baandery int
36-1
-55
45
15
-25
25
-25
25
PINN
PINNsFormer
PINNMamba
Figure 9. Loss and & -Error Curve w.r.t Training Iteration.
Figure 10. Absolute Error of pressure prediction of N-S equation
Table 3. Ablation Studies. The rMAF is reported here.
2024) and Navier-Stokes equations. As shown in Fig. 10, PINNMamba achieves the lowest error the N-S equa- tion. Just like PINNsFormer, PINNMamba also gets out-of- memory on some problems in PINNacle, which we identify as a major limitation of sequence-based methods. We dis- cuss the details of PINNacle experiments in Appendix F.
Method
Convection
Reaction
Wave
PINNMamba
0.0188
0.0094
0.0197
Sub-SSquence Align
0.1436
0.0291
0.0481
-Sub +Long Sequence
0.6492
0.6731
0.3391
-Time Varying SSM
0.0241
0.0179
0.0664
-SSM
0.7785
0,9836
0,3341
-Wavelet +Tanh
0.4531
0.0299
0.3151
8. Conclusion
In this paper, we reveal that the mismatch between discrete training of PINNs and the continuous nature of PDEs, as well simplicity bias are the key of failure modes. In com- bating with such failure modes, we propose PINNMamba, an BSM-based sub- sebbseeuence learning framework. PINN- Mamba successfully eliminates the failure modes, and mean- while becomes the new state-of-the-art PINN architecture.
We remove the sub-sequence alignment, which leads decrease in model performance, indicating the significance of the agreement formed through align- ment in eliminating simplicity bias. After replacing the sub-sequence with long sequence of the entire domain, the model shows failure modes, in line with the sequence granularity analysis in Section 5.2.
Sub-Sequence.
Time- Varee-V SSM. We replace the selective SSM (Gu & Dao, 2023) with a linear time-invarianl structure SSM (Gu al., 2022), and there is some decrease in model perfor- mance, illustrating the role of predictive diversity in elim- inating simplicity bias. And when we remove SSM com- pletely and switch to MLP instead, the model has severe failure modes. This demonstrates that SSM's 's adaptation for Continuous-Dee -iibsrree-- Mismatch allows the initial condition information to propagate sufficiently in time coordinates.
Impact Statement
The development of physics-informed neural networks rep- resents a transformative approach to solving differential equations by integrating physical laws directly into the learn- ing process. This work explores novel advancements in PINN architecture, to improve accuracy and eliminate the potential failure modes. By refining PINN architectures, this study contributes the broader adoption of physics- informed machine learning in fields such as computational fluid dynamics, material science, and engineering simula- tions. The proposed enhancements lead to more robust and scalable models, facilitating real-world applications where conventional PINNs struggle with smoothin.. There is no known negative impact from this study at this time.
In addition, also conducted sensitivity analysis of the choice of sub-sequence length, activation. See Appendix E.
7.5. Experiments on Complex Problems
To further demonstrate the generalization of our method, we tested our model on partial PINNacle Benchmark (Hao al.,

---

Sub-Sequential Physics-Informed Learning with State Space Model
Acknowledgements
Gao, Z., Yan, L.,, and Zhou, T. Failure-informed adaptive sampling for pinns. SIAM Journal on Scientific Comput- ing, 45(4): 2023.
This work is supported, part, by the National Science Foundation and the Institute of Education Sciences under Grant 2229873 (ALAExcettionalEd), National Science Foun- dation under Grant 2235364 (FuSe-TG), and SUNY-IBM Al Collaborative Research Award. Any opinions, findings and conclusions or recommendations expressed in this mate- rial are those of the author(s) and do not necessarily reflect the views of the sponsors.
Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023.
Gu, A., Dao. T., Ermon, S., Rudra, A., and Ré, C. Hippo: Recurrent memory with optimal polynomial projections. Advances in neural information processing systems, 33: 1474-1487, 2020.
References
Gu, A., Goel, K., and Re, C. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations, 2022.
Abadi, M., Barham, P.. Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al. tensor flow: system for large- scale machine learn- ing. In 12th USENIX symposium on operating systems de- sign and implementation (OSDI 16), pp. 265-283, 2016.
Hao, Z., Yao, J., Su, C., Su, H., Wang, Z., Lu, F., Xia, Z., Zhang, Y., Liu, S., Lu, L., et al. Pinnacle: A compre- hensive benchmark of physics-informed neural networks for solving pdes. In Advances in Neural Information Processing Systems, 2024.
Bu, J. and Karpatne, A. Quadratic residual networks: A new class of neural networks for solving forward and inverse problems in physics involving pdes. In Proceedings of the 202/ SIAM International Conference on Data Mining (SDM), pp. 675-683. SIAM, 2021.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-777777777 70007777700 2016.
Cai, S.. Wang. Z., Wang. S., Perdikaris, P., and Karniadakis, G. E. Physies-informed neural networks for heat trans- fer problems. Journal of Heat Transfer, 16666),, 2021.
Hornik, K., Stinchcombe, M., and White, H. Multilayer feedforward networks are universal approximators. Neu- ral networks, 2(5):359-336 -66, 1989.
Cho, W., Jo, M., Lim, H., Lee, K., Lee, D., Hong, S., and neural networks
Park, N. Parameterized physics-informie for parameterized PDEs. In Proceedings of the 41st Inter- national Conference on Machine Learning, Proceedings of Machine Learning Research, pp. 8510-8533. PMLR, 2024.
Hu, Z., Daryakenari, N. A., Shen, Q., Kawaguchi, K., and Karniadakis, G. E. State-space models are accurate and efficient neural operators for dynamical systems. arXiv preprint ar arXiv:2409.03231, 2024.
Jin, X.. Cai, S., Li, H., and Karniadakis, G. E. Nsfnets (navier-stokes flow nets): Physics-informed neural net- works for the incompressible navier-stokes equations. Journal of Computational Physics, 426:109951. 2021.
Dao, T. and Gu, A. Transformers are SSMs: General- ized models and efficient algorithms through structured state space duality. In Proceedings of the 41st Interna- tional Conference on Machine Learning, Proceedings of Machine Learning Research, pp. 10041-10071. PMLR, 2024.
Kalman, R. E. A new approach to linear filtering and pre- diction problems. 1960.
Fan, E. Extended tanh-function method and applications to nonlinear equations. Physics Letters A, 277(4-))2212 218, 2000.
Kharazmi, E., Zhang, Z., and Karniadakis, G. E. Variational physics-informed neural networks for solving partial dif- ferential equations. arXiv preprint arXiv:1912.00899 2019.
Fu, D. Y., Dao, T., Saab, K. K., Thomas, A. W., Rudra, A., and Re, C. Hungry hungry hippos: Towards language modeling with state space models. In The Eleventh Inter- national Conference on Learning Representations, 2023.
Krishnapriyan, A., Gholami, A., Zhe, S., Kirby, R., and Mahoney, M. W. Characterizing possible failure modes in physics-informed neural networks. Advances in neural information processing systems, 34:26548 24.25588-26660, 2021.
Gao, Z., Shi, X., Wang, H., Zhu, Y., Wang, Y. B., Li, M., and Yeung, D.-Y. Earthformer: Exploring space-time trans- formers for earth system forecasting. Advances in Neural Information Processing Systems, 35:25390 254033, 2022.
Liu, D. C. and Nocedal, J. On the limited memory bfgs method for large scale optimization. Mathematical pro- gramming, 45(1):503-55 -288, 1989.
10

---

Sub-Sequential Physics-Informed Learning with State Space Model
Liu, Y., Tian, Y., Zhao, Y., Yu, H., Xie, L., Wang. Y., Ye, Q., Jiao, J., and Liu, Y. VMamba: Visual state space model. In Advances in Neural Information Processing Systems, 2024a.
Raissi, M., Perdikaris, P., and Karniadakis, G. E. Physics- informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686-707, 2019.
Liu, Z., Ma, P., Wang, Y., Matusik, W., and Tegmark, M. Kan 2.0: Kolmogorov-armold networks meet science. arXiv preprint vrXiv:2408.10205, 2024b.
Reddy, J. An Introduction to the Finite Element Method. McGraw-Hie Hill, 1993.
Liu, Z. Wang, Y., Vaidya, S., Ruehle, F, Halver- son, J., Soljačić, M., Hou, T. Y., and Tegmark, M. Kan: Kolmogorov-arnole networks. arXiv preprint 1rXiv:2404.19756, 2024c.
Sain, M. and Massey, J. Invertibility of linear time-invarians dynamical systems. IEEE Transactions automatic control, 14(2):141-149) 1969.
Shah, H., Tamuly, K., Raghunathan, A., Jain, P., and Netra- palli, P. The pitfalls of simplicity bias in neural networks. Advances in Neural Information Processing Systems, 33: 9573-9585, 2020.
Mao, 7., Jagtap, A. D., and Karniadakis, G. E. Physics- informed neural networks for high-speed flows. Com- puter Methods in Applied Mechanics and Engineering, 360: 360:112789, 2020.
Nair, V. and Hinton, G. E. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICMI-10), pp. 807 814, 2010.
Smith, J. T., Warrington, A., and Linderman, S. Simplified state space layers for sequence modeling. In The Eleventh International Conference on Learning Representations, 2023.
Nguyen, E., Poli, M., Durrant, M. G., Kang, B., Katrekar, D., Li, D. B., Bartie, L. J., Thomas, A. W., King, S. H., Brixi, G., et al. Sequence modeling and design from molecular to genome scale with evo. Science, 386(6723); eado9336, 2024a.
Teney, D., Abbasnejad. E., Lucey, S., and Van den Hengel, A. Evading the simplicity bias: Training a diverse set of models discovers solutions with superior ood gener- alization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16761- 16772. 2022.
Nguyen, P. C., Cheng, X., Azarfar, S., Seshadri, P., Nguyen, Y. T., Kim, M.. Choi, S., Udaykumar, H., and Back, S. PARCv2: Physics-awand recurrent convolutional neural networks for spatiotemporal dynamics modeling. In Pro- ceedings of the 41 International Conference on Machine Learning, Proceedings of Machine Learning Research, pp. 31633-36766669 PMLR, 2024b.
Wang. C., Li, S., He, D., and Wang, L. Is 1'2 physics in- formed loss always suitable for training physics informed neural network? Advances in Neural Information Pro- cessing Systems, 35:8278-8290, 2022a.
Wang. S., Yu, X., and Perdikaris, P. When and why pinns fail to train: A neural tangent kernel perspective. Journal of Computational Physics, 449:110768, 2022b.
Pagliardini, M., Jaggi, M., Fleuret, F., and Karimireddy, S. P. Agree to disagree: Diversity through disagreement for better transferability. In International Conference on Learning Representationss 2023.
Wong, J. C., Ooi, C. C., Gupta, A., and Ong, Y.SS. Learning in sinusoidal spaces with physics-informed works. IEFE Transactions on Artificial Intelligence, 5(3): 985-1000, 2022.
informed neural net-
Paszke, A., Gross, S., Massa, F,, Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, 7., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Wu, C., Zhu, M., Tan, Q., Kartha, Y., and Lu, L. A compre- hensive study of non-adaptive and residual- adaptive sampling for physics-informe neural networks. Com- puter Methods Applied Mechanics and Engineering, 403:115671, 2023.
Pezeshki, M., Kaba, O., Bengio, Y., Courville, A. C., Pre- cup, D., and Lajoie, G. Gradient starvation: A learning proelivity in neural networks. Advances in Neural Infor- mation Processing Systems, 34: 1225-1272. 2021.
Wu, H., Luo, H., Ma, Y., Wang, J., and Long, M. RoPINN: Region optimized physics-ine innsree- neural networks. In Advances in Neural Information Processing Systems, 2024.
Poli, M., Massaroli, S., Nguyen, E., Fu, D. Y., Dao, T., Baccus, S., Bengio, Y., Ermon, S., and Ré, C. Hyena hierarchy: Towards larger convolutional language models. In International Conference on Machine Learning, pp- 18043-28075 PMLR, 2023.
Xu, C., Cao, B. T., Yuan, Y., and Meschke, G. Transfer learn- ing based physies-informen neural networks for solving

---

Sub-Sequential Physics-Informed Learning with State Space Model
inverse problems in engineering structures under differ- ent loading scenarios. Computer Methods in Applied Mechanics and Engineering, 405:1 15852, 2023.
Xu, C.,, Yu, F., Li, M., Zheng, Z.., Xu, Z.,, Xiong, J., and Chen, X. Infinite-dimensionall feature interaction. In Ad- vances Neural Information Processing Systems, 2024.
Yang. T.-Y., Rosca, J., Narasimhan, K., and Ramadge, Learning physics constrained dynamies using autoen- coders. Advances in Neural Information Processing Sys- tems, 35:17157-17172. 2022.
Yu, J., Lu, L., Meng. X., and Karniadakis, G. Gradient- enhanced physics-informed neural networks for forward and inverse pde problems. Computer Methods in Applied Mechanics and Engineering, 393: 1 14823, 2022.
Zhao, Z., Ding, X., and Prakash, B. Pinnsformer: A transformer-based framework for physics-informet neu- ral networks. International Conference on Learning Representations, 2024.
Zheng, J., LiweiNo, Xu, N., Zhu, J., Xiaoxul .in, and Zhang, X. Alias-free mamba neural operator. Advances in Neural Information Processing Systems, 2024.
Zhu, L., Liao, B., Zhang, Q., Wang. X., Liu, W., and Wang, X. Vision mamba: Efficient visual representation learning with bidirectional state space model. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, pp. 62429- 62442. PMLR, 2024.
12

---

Sub-Sequential Physics-Informed Learning with State Space Model
A. Proof of Theorem 4.1
start with a function v such that M(r) is non-zero almost everywhere. Such a function exists because M is non-zero differential operator. For example, M is the Laplacian, non non-harmonic function can be chosen.
Lemma A.1 (Existence of Base Function). Let M be non-degenerate differential operator on x [0, where L C R" a domain. There exists a function v E C (St [0, T)) such that:
M(U) + 0 for almost every (x. t) E x [0.T].
Proof. Since M is non-degenerate (i.e., not identically zero), there exists at least one function C ( [0, TT) and point (zo. E Q x [0, T such that:
M[w)(ro: to) # O.
By continuity of M(w) (assuming smooth coefficients for M), there is an open neighborhood U C Ω x [0, around (zo, to) where M(w) #
Construct smooth bump function & E C (22 T)) with: o on smaller neighborhood V C U, and á outside
U. Define vo Φ w. Then M(ro) = ((dw) is non-zero on V and smooth everywhere. Let ((ra, 4x) k=1 be countable dense subset of S2 [J. For each k, repeat the above construction obtain function E C* x [0, T)) such that: M(vx) + 0 in neighborhood Us of (Ia, (x), supp(uk) UR. and the supports } are pairwise disjoint.
Define the function:
EkUK
where > O are chosen such that the series converges in C (Ω x [0, (c.g.. c = 2 max(lelllc+.
The set U Uk is open and dense in x T). Since M(v) * on this dense open set, the zero set Z = {(x t) M(v)(r, 0} is contained the complement of U Uk, which is nowhere dense and hence has Lebesgue measure
zero. Therefore:
M(U) #0 0 for almost every (z, t) E e [0.
Lemma A.2 (Local Correction Functions). Let M be a non-degenerate differential operator on Ω x [0,T), and let AN
x" = ((f (ii,tt))................ (z } C Q [0,T]] There exist smooth functions {w C O (Ω2 x [0,T]) and radii c,....C > such that for each i:
Compact Support: supp(we) (1,,))
2. Non-Vanishing Action: M(w:)(af, (7) + 0,
3. Disjoint Supports: Be, (x, tt) Be, tz) f +
Proof. Let dmin = min dist ((a),)), (x), t)) be the minimal distance between distinct points in x". For all i, choose radii &; > such that:
dmin
2
This ensures the balls (2), t7) are pairwise disjoint.
For each (af, 17), since M is non-degenerate, there exists smooth function fa E C (2 x [0, T]) such that:
MI ( 0
This is because, when M is non-degenerate, its action cannot vanish on all smooth functions at (x7, 4). For instance, if M contains a derivative Ozg, take ft = near :)).
Then for each i, construct a smooth bump function o; E C x [0, T)) satisfying:
Φ; on B Be, (n], 17),
13

---

Sub-Sequential Physics-Informed Learning with State Space Model
2. Φ: = 0 outside Be, (z)
3. 0 < 0; I 1 everywhere.
Therefore, define the localized function:
Wy = 0; fi-
By construction:
1. supp(w:)) C. (2) my = fi on (2,/2(27 (4), so
M(w:)(a7, -
(M())(a7,4)) 40.
Since < Lnis the distance between any two balls B,, (17, (7) and Be, (2f, 17) is at least dmin - -2, > 0. Thus, the supports of w; and w are disjoint for * j.
Therefore, the functions (w; | satisfy all required conditions.
We now state the oieedimensional case of Theorem 4.1 here:
Lemma A.3 (One-Dimensional Case of Theorem 4.1). Let x" ((",,1)).... (x)) C Q [0, Then for differential operator M there exist infinitely many functions ug O - IR parametrized by 0, s.t.
M(uo(ot, =0 for L, M(uo(z,t)) #0 for a. 2x [0, xx
Proof. Define the corrected function:
Up
W;Wi;
where w; is the local correction function defined in Lemma A.2, ar E R are scalars chosen such that:
MI ()) = =Muu)at; ()) + a(s(s) (1) = 0.
Since ((w,)(xf,t)) (7) * O. we can solve for a;;
M(z)(22,t1)
a, M(wi)(zj,
Outside the union of supports U B,, (z), we have:
M(ue)= = M + a(w( = M(U),
since 0 outside ()), By construction, .M(e) * 0 almost everywhere.
The parameters translated, or reshaped (e.g., Gaussian vs. polynomial) while retaining the properties Local Correction in Lemma A.2 and varying e;: For each i, choose e, from continuum 60, where 6, ensures disjointness.
CN, O1,-.. ..y) can be varied infinitely by varying wy: The hump functions wy can be scaled,
Thus, the family {wo} } is uncountably infinite.
The X by definition has Lebesgue measure zero in Ω [0, The corrections MN 0; are confined to the measure-zero set U (x, t?). Therefore:
M(us) for a.e. (x,t) x xx
14

---

Sub-Sequential Physics-Informed Learning with State Space Model
We now generalize Lemma A.3 to m-dimeenion,, get Theorem 4.1. Theorem A.4 (Theorem 4.1). Let x" = exist infinitely many functions ug Ω + R parametrized by 8, s.t.
x ) C St x [0,T], Then for differential operator M there iv
M(uo(z),t)) = for = I,,.,,
M(ua(z, # 0 a.r a.e. E Exx {
Proof. It trivial generalize the Lemma A.3 to the case ug Ω - Rm, by constructing:
N m
u0
0
as Wi,j+
where (arg) E R. Adjust oij such that:
M(uo)(2), = M(v) an M(u,) 0.
This gives a linear system for a, which is solvable because the wiy are linearly independent.
B. Linear Time- Varying System
To adjust the given Linear Time-Invariant system a Linear Time- Time-Varying system, we replace the constant matrices A, B, and C with their time-varying counterparts A(k), B(k). and C(k). The state transition matrix A in the LTI system becomes the product of time-varying matrices from time to & - 1. The resulting time-varying output equation is:
Ug C(k)) + C(R) Σ (kk, ())))),(i)x;
(32)
where (k: is the state transition matrix from time i to k. defined as:
+ (k: i) A(k 1)A(k 2) ... A(i) if > i,
(33)
I
if
and the term @ 0)h, represents the free response due to the initial condition hg-
The summation Σ (k: Bi)x, includes contributions from all inputs up to time k, with (kk, B) capturing the time-varying dynamics.
To adjust the Eq. 18 to Time- Varying system The state transition term A - becomes the time-ordered product Φ i). and the output uą now explicitly depends on time-varying dynamics. The adjusted equation becomes:
k(tt( M (1ar 2
1 M
F 1. uy C(K))(k;))h) + C(k) Σ t B())x)
(34)
This modification ensures consistency with the Time- Sane-Varying system's time-dependent parameters while preserving the structure of the original loss function.
15

---

Sub-Sequential Physics-Informed Learning with State Space Model
C. PDEs Setups
C.1. 1-D Convection
The 1-D convection equation, also known as the I-D advection equation, is a partial differential equation that models the transport of a scalar quantity u(2, t) (such as temperature, concentration, or momentum) due fluid motion a constant velocity It fundamental equation in fluid dynamics and transport phenomena. The equation is given by:
du Ot
+ 3 Tu
dr 0, Var E [0, [0. 1],
u(x 0) sin Vo [0, 2m]: u(0,4 t) u(2n, l), Vt [0. 11,
(35)
where 3 is the constant convection (advection) speed. As 3 increases, the equation will be harder for PINN to approximate. It is well-known equation with failure mode for PINN. We set 50 following common practice (Zhao et al., 2024; Wu et al., 2024).
The 1-D convection equation's 's analytical solution is given by:
Mana (I, t) = sin(x - Bt).
(36)
C.2. 1-D Reaction
The I-D reaction equation is a partial differential equation that models how chemical species reacts over time and (optionally) varies along single spatial dimension. The equation is given by:
du
0t pa(1 u) = 0, Vz 20. [0,1],
u(x, 0) = expl (x- - x)²
2(T ) Vo E [0, 2m],
(37)
u(0, () = u02,, l), 1I E [0, 1],
where is the growth rate coefficient. As increases, the equation will be harder for PINN to approximate. It a well-known equation with failure mode for PINN. We set p = 5 following common practice (Zhao et al., 2024; Wu et al., 2024).
The 1-D reaction equation's analytical solution is given by:
exp(
(a- (x-m)) exp(pt)
2(5/4)2
Uana
(38)
exp(
(--1 (exp(pt) +1
2
C.3. 1-D Wave
The I-D wave equation is a partial differential equation that describes how a wave propagates through a medium, such as a vibrating string. We consider such an equation given by:
du pu = 0, Vx [0, 1]. [0, 1],
4
01 Or
1
u(x 0) sin sin(T
す sin (BT)), Vo [0, 1].
(39)
Ou(o, 0)
0, Vx E [0, 1),
u(0,t) t) = u (1 t) = 0, vt [0,1].
where 3 is a wave frequency coefficient. We set / as 3 following common practice (Zhao et al., 2024; Wu et al., 2024). The wave equation contains second-order derivative terms the equation and first-order derivative terms in the initial condition, which is considered to hard optimize (Wu al., 2024). This example illustrates that PINNMamba can better capture
16

---

Sub-Sequential Physics-Informed Learning with State Space Model
the time continuum because its differentiation for time is directly defined by the matrix, whose differential scale is uniform for multiple orders.
The 1-D wave equation's analytical solution is given by:
Mana (I, 1) = (ix( cos() + sin(( c2s(2
(40)
C.4. 2-D Navier-Stokes
The 2-D Navier-Stokes equation describes the motion of fluid in two spatial dimensions æ and y. fundamental in fluid dynamics and is used to model incompressible fluid flows. We consider such equation given by:
Uu + >1 (u
du
Ou
Op
Ouu u Ou,
+ U
+
X2
+
Ot
Oz
dy
dr
dr
dv2
dv
X1 (u Ov
+ U δv.
Op
8u
22
+
(41)
Øt
Dr
Oy
dy
dr
dv
where u(2, y, t). vxx, y. and p(z, t) are the x-coordinate velocity field, y-coordinate velocity field, and pressure field, respectively. We set Åı 1 and A2 = 0.01 following common practice (Zhao et al., 2024; Raissi et al., 2019).
The 2-dimensional Navier-Stokes equation doesn't 't have an analytical solution that can be described by existing mathematical symbols, we take Raissi et al. (2019)'s 's finite-element numerical simulation as ground truth.
C.5. PINNNacle
PINNacle (Hao et al., 2024) contains 16 hard PDE problems, which can be classified as Burges, Poisson, Heat, Navier-Stokes Wave, Chaotic, and other High-dimensional problems. We only test PINNmamba on problems, because solving the remaining problems with a sequence-based PINN model will cause an out-of-memory issue, even on the most advanced NVIDIA H100 GPU. Please refer to the original paper of PINNacle (Hao et al., 2024) for the details of the benchmark.
D. Training Details
Hyperparameters. We provide the training hyperparameters of the main experiments in Table 4.
Table 4. Hyperparameters for main results.
Model
Hyperparameter Type
Value
network depth
4
PINN
network width
512
network depth
4
QRes
network width
256
network width
[2,5,5,1]
KAN
grid grid_epsilon
1.0
of encoder # of decoder embedding size attention head MLP hidden width
32
PINNsFormer
2
512
sequence length
sequence interval At
lc44
of encoder embedding size
32
A. B, C width MLP hidden width
8
PINNMamba
512
sequence length
sequence interval At
le-2
17

---

Sub-Sequential Physics-Informed Learning with State Space Model
Computation Overhead. We report the training time and memory consumption of baseline models and PINNMamba the convection equation in 5.
Table 5. Memory overhead and training time on H100 GPU for solving convection equation.
Method
Memory Overhead
Training Time
PINN
1605 MB
0.28 s/it
QRes
1561 MB
0.38
PINNsFormer
8703 MB
1.82 s/it
KAN
1095 MB
2.73 s/it
PINNMamba
7899 MB
1.99 s/it
E. Sensitivity Analysis
PINNMamba can be further improved by selection in this section.
tuning, we test the sub-sequence length, interval and activation
Sub-sequence Length. We test the effect of different sub-sequence lengths model performance. As shown in Table 6, test the length of 3, 5, 9, 21. Length k = 7 achieves the best performance on reaction and wave equations, while k: = 5 achieves the best performance on convection equation.
Table Results with different Sub-Sequence Length of PINNmamba.
Convection
Reaction
Wave
Longth
rMAE
rRMSE
rMAE
rRMSE
rMAE
rRMSE
0.6698
0.7271
0,0150
0.0331
0.5288
0,533926
0.0092
0.0099
0.0131
0,0286
0.0278
0.0303
0.0188
0.0201
0.0094
0.0217
0.0197
0.0199
0.0410
0.0444
0,0105
0.0246
0.0343
0.0374
1.0263
1.0596
0.0884
0.1588
0.0458
0.0493
Sub-Sequence Interval. We test the effect of different sub-sequence intervals on model performance. As shown in Table
test the intervals of 2e 3, 5e 3, le le 1. The interval Al le achieves the best performance convection and wave equations, while At 5e 3 achieves the best performance on reaction. Note that, when At 1e cannot build the sub-sequence contrastive alignment.
Table 7. Results with different Sub Sequence Interval of PINNmamba. & is set to 7.
Convection
Reaction
Wave
Interval
rMAE
rRMSE
rMAE
rRMSE
:MAE
rRMSE
2e-3
0.0249
0.0257
0.0739
0.1389
0.1693
0.1903
5e-3
0.0243
0.0287
0.0083
0.0185
0.2492
0.2690
le-2
0.0188
0.0201
0.0094
0.0217
0.0197
0.0199
le-1
1.2169
1,3480
0.4324
0.5034
0.0666
0.0703
Activation Function. We test the activation function's effect on the performance of PINNMamba. We report the results of ReLU (Nair& & Hinton, 2010), Tanh (Fan, 2000), and Wavelet (Zhao et al., 2024) in Table 8.
Table 8. Results with different activation function in PINNmamba.
Convection
Reaction
Wave
Activation
:MAE
rRMSE
rMAE
rRMSE
rMAE
rRMSE
ReLU
0.4695
0.4722
0.0865
0.1583
0.4139
0.4203
Tanh
0.4531
0.4601
0.0299
0.0568
0.3515
0.3539
Wavelet
0.0188
0.0201
0.0094
0.0217
0.0197
0.0199
18

---

Sub-Sequential Physics-Informed Learning with State Space Model
Fredcted
Pinditeepoo
01
25
15
600
140
-55
14
15
-25
30
75
-25
08
080
-25
09
25
50
PINN
QRes
PINNsFormet
PINNMamba
Figure 11. The ground truth solution, prediction (top), and absolute error (bottom) on Navier-Stokes equations.
F. Complex Problem Results
F.1. 2D Navier-Stokes Equations
Although PINN can already handle Navier-Stokes equations well, we still tested the performance PINN Mamba on Navier-Stokes equations to check the generalization performance of method on high-dimensional problems. As shown in Fig. 11, method achieves good results Navier-Stokes pressure prediction. Since there is initial condition information for the N-S equation for pressure, took the data from the only collocation point for pattern alignment.
E2. PINNacle Benchmark
Like PINNsFormer, PINNMamba is sequence model. The sequence model suffers from Out-of-Memory problems when dealing with some of the problems in the PINNacle Benchmark (Hao et al., 2024), even when running the advanced Nvidia H100 GPU. We report here the results of the sub-problems for which results can be obtained in Table 9, PINNMamba can solve the Out-of-Memory problem by distributed training over multiple cards, which we leave as follow-up work.
Table 9. Results on PINNacle. Baseline results from RoPINN paper (Wu al., 2024). OOM means Out-of-Memroy.
PINN
PINNsFormer
PINNMamha
Equation
rMAE
iRMSE
iMAR
rRMSE
rMAE
rRMSE
Burgers 1d-C
1.le-2
3.3e-2
9.3e-3
1.4e-2
3.7e-3
I.le-
Burgers 2d-C
4.5e-1
5.2e-1
OOM
0OM
OOM
OOM
Poisson 2d-C
7.5c-1
6.8c-1
7.2c-1
6.6e-1
6.2
5.7c-1
Poisson 2d-CG
5,4c-1
6.6c-1
5.4c-1
6.3c-1
1.2c-1
1.4c-1
Poisson 3d-CG
4.2c-1
5.0c-1
OOM
OOM
00M
OOM
Poisson 2d-MS
7.8e-1
6.4e-1
1.3e+0
1.le40
7.2e-1
6.0e-1
Heat 2d-VC
1.2e40
9.8e-1
OOM
OOM
OOM
OOM
Heat 2d-MS
4.7e-2
6.9e-2
OOM
OOM
OOM
OOM
Heat 2d-CG
2.7e-2
2.3e-2
OOM
OOM
OOM
OOM
NS 2d-C
6.1e-2
5.1e-2
OOM
OOM
OOM
OOM
NS 2d-CG
1.8e-1
1.1e-1
1.0e-1
7.0e-2
1.le-2
7.8e-3
Wave ld-C
5.5e-1
5.5c-
5.0c-1
5.1e-1
1.0c-1
1.0c-
Wave 2d-CG
2.3c+0
1.6c+0
00M
OOM
OOM
OOM
Chaotic GS
2.1c-2
9.4c-2
00M
OOM
OOM
OOM
High-dim PNd
1.2e-3
1.1e-3
0OM
OOM
OOM
OOM
High-dim HNd
1.2e-2
5.3e-3
0OM
0OM
OOM
OOM
19


Reviews:
[{"review_id": "UG8tX7bB19", "reviewer": "Reviewer_PwV2", "summary": "The paper introduces PINNMamba, a framework that integrates state space models (SSMs) into physics-informed neural networks (PINNs) to address failure modes in solving partial differential equations (PDEs). It identifies two key issues: the continuous-discrete mismatch, which disrupts initial condition propagation, and simplicity bias, which leads to over-smoothed solutions. PINNMamba mitigates these by using SSMs for continuous-discrete articulation and sub-sequence modeling to enhance pattern propagation. Experimental results show that PINNMamba outperforms existing PINN architectures in solving PDEs with improved accuracy and generalization.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the constructive comments from reviewer PwV2 and the time spent on reviewing this paper. We address the questions and clarify the issues accordingly as described below.\n\n>**[W1]**: The authors placed related works in the appendix, which I find unusual and not ideal. Integrating it into the main text would provide better context for their contributions.\n\n**[Response to W1]**: We highly agree with your opinion. We will place the related works section in the main text in the next release since another page will be allowed in camera-ready according to ICML author guidance.\n\n>**[Un RW]**: ... Including a discussion of such methods (NN/Graph-based PDE solvers) would provide a more comprehensive view of the broader landscape of neural network-based PDE solvers.\n\n**[Response]**: We will add the following paragraph in the related works section.\n\n**Learning-Based PDE Solvers.** In addition to Physics-Informed Neural Networks (PINNs), other neural-network-based approaches have emerged for solving PDEs, each offering unique advantages. Graph Neural Networks (GNNs) like MeshGraphNet [1] excel at handling irregular domains by treating computational meshes as graphs, making them particularly effective for complex geometries. Neural operators [2], including Fourier Neural Operators (FNOs) [3] and Graph Neural Operators (GNOs) [4], learn mappings between function spaces, enabling generalization across different PDE parameters without retraining. Hybrid approaches combine neural networks with traditional numerical methods, such as neural finite element techniques [5], to enhance solver efficiency. While PINNs uniquely enable data-free solutions through direct physics constraint enforcement, GNNs, and neural operators provide complementary capabilities - GNNs for mesh-based problems and neural operators for parametric systems. These diverse approaches collectively demonstrate the expanding toolkit of neural-network-based PDE solvers across scientific computing applications.\n\n\n[1] Pfaff, Tobias, et al. \"Learning mesh-based simulation with graph networks.\" in ICLR. 2021.\n\n[2] Kovachki, Nikola, et al. \"Neural operator: Learning maps between function spaces with applications to pdes.\" in JMLR. 2023.\n\n[3] Li, Zongyi, et al. \"Fourier Neural Operator for Parametric Partial Differential Equations.\" in ICLR. 2021.\n\n[4] Li,  et al. \"Multipole graph neural operator for parametric partial differential equations.\" in NuerIPS. 2020.\n\n[5] Hennigh, Oliver, et al. \"NVIDIA SimNet™: An AI-accelerated multi-physics simulation framework.\" International conference on computational science. 2021.\n\n>**[W2]**: reduce the length of the methods, introduction and remove some results to the appendix.\n\n**[Response to W2]**: We will streamline these sections in the next release depending on space.\n\n>**[Q]**: About the memory usage.\n\n**[Response to Q]**: The main reason for the large Memory Usage is that, initially, we set the sequence length to 7 and MLP width to 512 in our main experiment, which was made with the consideration of making a fair comparison with the then SOTA model PINNsFormer. High memory usage is one of the main drawbacks of sequence modeling on PINNs, as the gradient information needs to be preserved for every point in the sequence.\n\nHowever, we find in a follow-up that our model is robust w.r.t sequence length and MLP width. Our model with a smaller sequence length and MLP width can also eliminate failure modes. This further enhances the generalizability of our approach. To scale to more complex problems, we suggest reducing the sequence length or reducing the MLP width.\n\nOur follow-up experiments show that the model is not very sensitive to sequence length and that a sequence length of 3 makes PINNMamba very effective in combating failure modes. (This is contrary to our results in Table 6 because we found that we had incorrectly set the computational precision in the length 3 and 5 setting in our original sensitivity analysis, resulting in severe performance degradation. We will fix this in the next release.) Adjusting the sequence length to 3 leads to a reduction in the memory overhead of the model by about 57%.\n\nIn addition, we found that a large MLP width (512) is not necessary. Setting the width to 32 is sufficient to make PINNMamba effective. This saves up to 53% of memory usage.\nThe combination of the two \nadjustment can lead to a reduction in memory consumption from 7899MB to 1900MB on the convection problem.\n\nWe added the following experiments. It is worth noting that the model successfully eliminates failure modes under all these settings.\n\nWhen length is set to 1, PINNMamba degrades to PINN and failure modes present again.\n\nLength|MLP width| rMAE| rMRSE | Memory | Time/iter|\n|---|---|---|---|---|---|\n|3|32|0.0140|0.0167|1900MB|0.79s|\n|5|32|0.0284|0.0321|2586MB|1.16s|\n|7|32|0.0240|0.0269|5932MB|1.35s|\n|3|512|0.0102|0.0126|4042MB|1.10s|\n|5|512|0.0059|0.0068|6020MB|1.59s|\n|7|512|0.0188|0.0201|7899MB|1.99s|", "claims_and_evidence": "The paper's claims are supported by theoretical analysis, empirical experiments, and comparative evaluations. PINNMamba's ability to mitigate continuous-discrete mismatch and simplicity bias is justified through theory and demonstrated by improved accuracy over baselines. The effectiveness of sub-sequence modeling and state space models (SSMs) is validated through ablation studies and performance metrics.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are appropriate for the problem and align with prior works. The use of state space models (SSMs) and sub-sequence modeling effectively addresses known limitations in PINNs, and the evaluation on standard PDE benchmarks ensures comparability with existing approaches. The inclusion of ablation studies and comparisons with multiple baselines further supports the validity of the methodology.", "theoretical_claims": "I do not specialize in PINNs, so I can only perform a general check of the theoretical claims. The proofs appear logically structured and follow standard mathematical reasoning, but I rely on other reviewers with more expertise in this area to verify their correctness in detail.", "experimental_designs_or_analyses": "The experimental designs and analyses appear sound. The paper evaluates PINNMamba on standard PDE benchmarks, compares it against multiple baselines, and includes ablation studies to validate key design choices. The metrics used, such as relative MAE and RMSE, are appropriate for assessing model accuracy. I did not identify any major issues.", "relation_to_broader_scientific_literature": "The paper builds on prior work in PINNs and addresses known failure modes by incorporating state space models (SSMs) and sub-sequence modeling. It aligns with existing research on improving PINN stability and accuracy, particularly studies that explore sequential modeling and optimization strategies to mitigate simplicity bias and continuous-discrete mismatch.", "essential_references_not_discussed": "The paper provides a strong review of PINNs but could benefit from discussing other machine learning approaches for solving PDEs, particularly graph neural network (GNN)-based methods like MeshGraphNet. Including a discussion of such methods would provide a more comprehensive view of the broader landscape of neural network-based PDE solvers.", "other_strengths_and_weaknesses": "The authors placed related works in the appendix, which I find unusual and not ideal. Integrating it into the main text would provide better context for their contributions. They could reduce the length of the methods and introduction sections to make space. Additionally, some results, such as Table 2 (which studies the effect of training strategies), could be moved to the appendix to streamline the main presentation.", "other_comments_or_suggestions": "- Reorganize Related Works: Move the related works section from the appendix into the main text for better context on contributions.\n- Condense Writing in Methods/Intro: The methods and introduction sections could be more concise to improve readability. Some results, such as Table 2, could be moved to the appendix.", "questions_for_authors": "The paper has a major limitation in memory usage, which the authors acknowledge. However, a deeper analysis of the computational bottleneck is needed to study which operations caused the issue. Additionally, would reducing the sequence length or modifying memory-intensive operations help mitigate these issues? Given that PINNsFormer also encounters OOM errors, is there a fundamental limitation in sequence-based PINNs and would this prevent the method to scale up to large scale simulations?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GrYcSIhMXo", "reviewer": "Reviewer_yZBS", "summary": "This paper proposes PINNMamba, enabling PINNs with state-space model's continuous-discrete capability to address the limitations (simplicity bias and continuous-discrete mismatch) of existing PINNs.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the constructive comments from reviewer yZBS and the time spent on reviewing this paper. We address the questions and clarify the issues accordingly as described below.\n\n>**[W1]**: The proposed approach is computationally expensive, as shown in Table 5 of the supplementary material. The memory overhead is the second largest, and the training time is seven times slower than that of PINN.\n\n>**[Q1]**: From the sensitivity analysis, the proposed approach is very sensitive to the sub-sequence length, especially comparing the length of 3 and 5 (about 70 times smaller). Are there any explanations for such a big gap?\n\n**[Re W&Q1]** We answer these two questions together. We start by fixing a bug in Table 6. In our follow-up experiments, we found an error in the experimental data for sequence lengths 3 and 5 in Table 6. The reason for this error is that we accidentally reduced the numerical precision of the calculations when performing these two sets of experiments, which led to an unexpected performance degradation of PINNMamba when using these two sets of parameters. So the conclusion of the original submission about the sensitivity of sequence length needs to be corrected, and we will fix this in the next release. Our latest experimental data are shown in the table below.\n\nConvection:\n\nLength|MLP width| rMAE| rRMSE | Mem. | Time/iter|\n|---|---|---|---|---|---|\n|3|512|0.0102|0.0126|4042MB|1.10s|\n|5|512|0.0059|0.0068|6020MB|1.59s|\n|7|512|0.0188|0.0201|7899MB|1.99s|\n\nReaction:\n\nLength|MLP width| rMAE| rRMSE | Mem. | Time/iter|\n|---|---|---|---|---|---|\n|3|512|0.0164|0.0352|4042MB|0.90s|\n|5|512|0.0109|0.0244|6020MB|1.25s|\n|7|512|0.0094|0.0217|7899MB|1.56s|\n\nThis group of experiments shows that our model can reduce computational and memory overhead by reducing the sequence length. There is a slight accuracy degradation on the reaction problem, but the performance is even better on the convection problem. Nevertheless, they successfully address the failure modes in all cases. \n\nOn the other hand, we found that setting the MLP layer's width to 512 is unnecessary. Reducing the MLP width from 512 to 32 yields a model that still solves the failure modes. \n\nConvection:\nLength|MLP width| rMAE| rRMSE | Mem. | Time/iter|\n|---|---|---|---|---|---|\n|3|32|0.0140|0.0167|1900MB|0.79s|\n|5|32|0.0284|0.0321|2586MB|1.16s|\n|7|32|0.0240|0.0269|3310MB|1.35s|\n\nReaction:\nLength|MLP width| rMAE| rRMSE | Mem. | Time/iter|\n|---|---|---|---|---|---|\n|3|32|0.0042|0.0085|1900MB|0.62s|\n|5|32|0.0032|0.0069|2586MB|0.82s|\n|7|32|0.0060|0.0126|3310MB|1.01s|\n\nIf we set sequence length to 3 and MLP width to 32, the memory usage is 1900MB, only slightly larger than 1605MB of vanilla PINN. Given the addressing of failure modes and an rMAE that is only 1/60th of vanilla PINN's, we consider this slight increase in memory and computational overhead to be insignificant. Our model is robust w.r.t sequence length and MLP width. Model with smaller sequence length and MLP width can also eliminate failure modes. This further enhances the generalizability of our approach.\n\nThe gap between 3 and 5 sequence length is in fact not large, but when it is 1, it degrades to PINN and failure modes present again. We sincerely apologize for this confusing. \n\n>**[Q2]**: The current ablation study only removes one component at a time, making it difficult to see the effect of other combinations of proposed components.\n\n**[Re Q2]**:\nWe added the following experiments.\n\nModel|Convection rMAE|Convection rRMSE| Reaction rMAE | Reaction rRMSE | Wave rMAE| Wave rRMSE|\n|-|-|-|-|-|-|-|\n|PINNMamba|0.0188 |0.0201|0.0094| 0.0217|0.0197|0.0199|\n|-Sub Seq Align & Time Varying SSM|0.1534|0.1572|0.0333|0.0351|0.0701| 0.0702|\n|-Sub Seq Align & SSM|0.9833|0.9836|0.9801|0.9821|0.4211| 0.4431|\n|-Sub Seq Align & Wavelet|0.5000|0.5021|0.0345|0.0399|0.3519| 0.3573|\n|-Time Varying SSM & Wavelet|0.3921|0.3971|0.0287|0.0287|0.2919| 0.2873|\n|-SSM & Wavelet|1.0263|1.0348|0.9987|0.9988|0.5137| 0.5222|\n|-Sub Seq Align & Time Varying SSM & Wavelet|0.3472|0.3521|0.0487|0.0534|0.3437| 0.3492|\n|-Sub Seq Align & SSM & Wavelet|1.2263|1.2748|0.9821|0.9833|0.5421| 0.5453|\n\nThese experimental data show that elimination of continuous-discrete mismatch with SSM is the most important factor, illustrating the significance of the PINNMamba model. Also, Sub Sequence Alignment and Time Varying SSM play an important role in eliminating simplicity bias, and the use of wavelet activation function leads to better numerical precision.\n\n>**[Q3]**: ... it would be interesting to see the performance comparison using a similarly sized model to PINN (i.e., with a smaller embedding size).\n\n**[Re Q3]**: For the convection problem, we further reduce the width of MLP and Mamba to 8. This reduces the memory overhead of the model to 1300MB, and the average optimization time per iteration to 0.23s (less than PINN). The rMAE is 0.0262, the rRMSE is 0.0346. It addresses the failure modes and outperforms all baselines.", "claims_and_evidence": "1. Mainstream PINNs predominantly use MLPs and suffer from the inability to accurately propagate physical patterns informed by initial conditions. Evidenced by a toy example in Fig. 1.\n\n2. PINNs have issues of continuous-discrete mismatch and simplicity bias. Evidenced by Section 3.", "methods_and_evaluation_criteria": "The authors propose to use SSM to address the continuous-discrete mismatch and sub-sequential alignment to improve the simplicity bias limitation.", "theoretical_claims": "Theorem 3.1 is proposed to show the continuous-discrete mismatch failure mode of PINN by demonstrating that there exist infinitely many functions when given a discrete collection of points.", "experimental_designs_or_analyses": "The authors evaluated the approach on three public benchmarks (convection, wave, and reaction equations) and compared it with other approaches such as PINN, QRes, PINNsFormer, and Kan.", "relation_to_broader_scientific_literature": "Addressing the failure modes in PINNs can potentially be applied to a wide range of scientific and engineering disciplines, such as computational fluid dynamics.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n1. The paper is well-written and easy to follow.\n2. From the results presented by the authors, the proposed approach consistently outperforms other approaches in the three tested benchmarks.\n\nWeaknesses:\n1. The proposed approach is computationally expensive, as shown in Table 5 of the supplementary material. The memory overhead is the second largest, and the training time is seven times slower than that of PINN.", "other_comments_or_suggestions": "See the questions section.", "questions_for_authors": "1. From the sensitivity analysis, the proposed approach is very sensitive to the sub-sequence length, especially comparing the length of 3 and 5 (about 70 times smaller). Are there any explanations for such a big gap?\n\n2. The current ablation study only removes one component at a time, making it difficult to see the effect of other combinations of proposed components.\n\n3. Given that the computational overhead of the proposed PINNMamba encoder is significantly higher than PINN, it would be interesting to see the performance comparison using a similarly sized model to PINN (i.e., with a smaller embedding size).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LRTZqVTMZk", "reviewer": "Reviewer_Ce7f", "summary": "The authors find the reason of failure modes of PINNs is the dismatch between continous nature of PDE and the discrete nature of sampled observations, and the simplicity bias of PINNs fails fixing this gap. To address this, they propose to use Mamba's sequence modeling ability and enhance it with alignment.\n\n### **After rebuttal**\n\nThanks to the authors for their feedback. I think my concerns have been addressed, and I would like to keep my score 3 unchanged.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the constructive comments from reviewer Ce7f and the time spent on reviewing this paper. We address the questions and clarify the issues accordingly as described below.\n\n>**[W1]**: Claims to resolve the continuous-discrete mismatch but still relies on a discretized version of Mamba..\n\n**[Response to W1]**: We need to clarify that we do not claim to solve the continuous-discrete mismatch. In fact, we think that the computer's finite precision inherently makes it impossible to describe continuous systems (including continuous-time SSM).\n\nInstead, we can only try to mitigate the effect of this mismatch. A discretized SSM is a description of the behavior of a continuous SSM. The point here is that even discrete-time SSM **describes the dynamic in a continuous/differential manner** since there is a set of rules for a direct mapping between a discretized SSM and a continuous SSM. Although it is actually an approximation of continuous dynamics, this ability to directly approximate/describe continuous dynamics is not available with traditional neural network models such as MLP and Transformer. \n\nIndeed, SSM may not be the ultimate solution to this problem, but we have still taken a huge step towards solving continuous-discrete mismatch. We hope that our work will inspire more researchers to achieve a better solution to this problem.\n\n>**[W2]**: Comparsion with other methods.\n\n**[Response to W2]**: Since our approach is an on model architecture, we have mainly listed multiple model architectures as baselines. But we do value your opinion and we will include the following results in the next release. Since the experimental settings of some of the papers differ from ours and the source code is not published, we report those works that we successfully reproduced.\n\nModel|Convection rMAE|Convection rRMSE| Reaction rMAE | Reaction rRMSE | Wave rMAE| Wave rRMSE|\n|-|-|-|-|-|-|-|\n|RoPINN (Wu. 2024 NeurIPS) (Optimization)|0.635| 0.720|0.056 | 0.095|0.063 |0.064|\n|P2INN (Cho. 2024 ICML) (Model Archi)|0.1023| 0.1035|0.0098|0.0227|0.2134|0.2157|\n|DCGD (Hwang. 2024 NeurIPS) (Optimization)|0.0232|0.0246|0.9780|0.9800|OOM|OOM|\n|R3 (Daw. 2023 ICML)(Sampling)|0.0267|0.0277\n|PINNMamba|0.0188 |0.0201|0.0094| 0.0217|0.0197|0.0199|\n\n>**[C&S1]**: Theorem is not novel. There is also no theory justifying why PINNMamba outperforms PINN.\n\n**[Response to C&S1]**: Theorem 3.1 is very intuitive, so we also tried to find a direct proof of Theorem 3.1 when preparing the manuscript, but failed. We would welcome some relevant references from the reviewers.\n\nAs for a theory justifying why PINNMamba outperforms PINN. We agree that an in-depth theoretical analysis of the proposed method is helpful to better understand our model. But to be honest, currently, we could not find a good theorem to prove rigorously such mechanisms in deep networks, since theoretically analyzing a complex system like PINNMamba is very difficult. However, we still have some intuitive and empirical analysis to show the methodology and philosophy behind our continuous-discrete mismatch and simplicity bias perspective. We hope such analysis can help readers to better understand our motivation and provide some guidance to design better models in future research.\n\n>**[C&S2]**: The proposal cannot fundamentally resolve simplicity bias.\n\n**[Response to C&S2]**: As previous work[1,2,3] has pointed out, simplicity bias is an intrinsic problem for neural networks. Currently, there is no single way to completely solve this problem, only to evade or mitigate it.\n\n[1] H Shah. The pitfalls.. NeurIPS 2020\n\n[2] D Teney. Evading the simplicity bias.. CVPR 2022\n\n[3]R Tiwari. Overcoming simplicity bias.. ICML 2023\n\n>**[Q1]**: Novelty: How is this work fundamentally different from PINNsFormer, given that it seemly replaces Transformers with Mamba...\n\n**[Response to Q1]**: Our work is not a simple block-wise replacement. Although both works are based on model architecture, they are quite different in the following points. \n\n1. Macro Architecture: PINNsFormer employs an Encoder-Decoder architecture, which we found unnecessary for PINN. So in PINNMamba, we use an Encoder-Only archi for better performance and efficiency.\n\n2. Sequence Modeling: As shown in Fig.5, PINNsFormer uses a pseudo-sequence, which is not a real collection points sequence, instead, a regional extrapolation, which cannot propagate information. We build collection point sub-sequences and an alignment to realize info propagation across time.\n\n3. Understanding: Our work provides a deeper understanding of PINN failure modes that it is caused by C-D mismatch and simplicity bias. We believe it can inspire more valuable work in the future.\n\n>**[Q2]**: typo\n\n**[Response to Q2]**: It should be $(x,t+k\\Delta t)$. Will fix.\n\n>**[Q3]**: Memory Usage\n\n**[Response to Q3]**: We greatly optimized memory usage in a follow-up. The OOM is no longer an issue when using a reduced model size. See response to reviewer yZBS.", "claims_and_evidence": "Yes. The experimental results demostrate the claims of good performance.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "I checked the proof.", "experimental_designs_or_analyses": "Yes. The experimental designs and alalyses are sound.", "relation_to_broader_scientific_literature": "The paper is based on related work on PINN failure modes (Krishnapriyan et al., 2021), state-space models (Gu & Dao, 2023), and sequence modeling for PDEs (Zhao et al., 2024). However, it lacks thorough comparisons with adaptive sampling (Wu et al., 2023; Gao et al., 2023) and optimization strategies like RoPINN (Wu et al., 2024) and NTK-based methods (Wang et al., 2022), which also address PINN failure modes.", "essential_references_not_discussed": "Based on my limited knowledge, I am unable to identify any missing essential related works.", "other_strengths_and_weaknesses": "Strengths: \n\n1. Proposes PINNMamba, integrating Selective SSM (Mamba) with PINNs to improve temporal information propagation and mitigate failure modes.\n2. Demonstrates significant performance gains on benchmark PDEs, reducing errors compared to previous PINN architectures.\n3. This paper is very clearly written.\n\nWeaknesses:\n1. Claims to resolve the continuous-discrete mismatch but still relies on a discretized version of Mamba, which remains dependent on time-step selection. Therefore, the continuous-discrete mismatch still persists, making this an ad-hoc rather than a fundamental solution.\n2. Although the authors mentioned various methods for addressing failure modes in the related works, including optimization techniques (Wu et al., 2024; Wang et al., 2022a), adaptive sampling strategies (Gao et al., 2023; Wu et al., 2023), model architectures (Zhao et al., 2024; Cho et al., 2024; Nguyen et al., 2024b), and transfer learning approaches (Xu et al., 2023; Cho et al., 2024), they did not provide a thorough comparison with these methods in their analysis or experiments.", "other_comments_or_suggestions": "1. The theoretical result Theorem 3.1 is interesting. It formalizes the well-known issue that optimizing PDE constraints at discrete points does not ensure global correctness. Though interesting, it is not novel. There is also no theory justifying why PINNMamba outperforms PINN. \n\n2. The proposal cannot fundamentally resolve simplicity bias. While sub-sequence contrastive alignment mitigates the issue by enforcing consistency across overlapping sub-sequences, it does not eliminate the tendency of neural networks to favor simpler solutions.", "questions_for_authors": "1. Novelty: How is this work fundamentally different from PINNsFormer (Zhao et al., 2024), given that it seemly replaces Transformers with Mamba, a common modification in recent models?\n\n2. Writing: Is there a typo in Line 326 \"$(x,k+\\Delta t)$\"?\n\n3. In Page 17, the authors say \"because solving the remaining problems with a sequence-based PINN model will cause an out-of-memory issue\". Does this mean PINNMamba needs more memory than traditional solutions?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PV1mBVUkao", "reviewer": "Reviewer_PWyp", "summary": "This paper addresses the two fundamental challenges in training PINN, continuous-discrete mismatch and simplicity bias. The proposed PINNMamba employs 1) State Space Model (SSM) to effectively capture continuous information in discrete temporal sequences and 2) sub-sequence contrastive alignment not to make the model trapping in the over-smooth local optimum (simple but incorrect solution).", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the constructive comments from reviewer PWyp and the time spent on reviewing this paper. We address the questions and clarify the issues accordingly as described below.\n\n>**[C&E:]** I have concerns on the novelty .... with a toy example in Figure 1 does not sound a challenging problem.\n\n**[Re:]** To understand our work's contribution, one must first understand the current status of research on **PINN failure modes**, which is the core problem we are addressing. PINN failure mode is not as simple as a regular learnability problem. It is a problem unique to PINN and was first identified by Krishnapriyan et al in 2021 NeurIPS. PINN failure modes refer to a phenomenon where the residual loss of a PINN is extremely low, but its average error (even on training collection points) is extremely high. This phenomenon is distinguished from concepts like local minimal (PINN failure modes loss can be as small as ground truth), overfitting, and gradient vanishing, in which the core problem is optimization or generalization. The essential difficulty of this problem is that ground truth is unknowable, and the only known fully optimized PDE-based governing loss is unreliable. In real-world applications, this can result in serious misestimation of the system. \n\nThe central contribution of our paper is a unique understanding of this unique phenomenon. The fact that the continuous-discrete mismatch and simplicity bias are well known does not mean that their explanatory role for PINN failure modes is well established. The ideas in the traditional spatiotemporal modeling do not necessarily apply to PINN, since training a PINN is a data-free and ground-truth-unknown problem. A great deal of discussion has arisen in the research community regarding failure modes for PINNs, but to the best of our knowledge, including PINNsFormer, which introduced the Transformer to PINNs, has failed to intrinsically understand and address the PINN failure modes. The distortions in the propagation of the initial conditions we propose are empirically observable, as shown in Fig.1, 2, 6, 7, and 8. We propose that continuous-discrete mismatch and simplicity bias are the core causes of these propagation failures, which is a completely novel understanding of the subject. \n\nWe also need to point out that the convection problem shown in Fig.1 is a well-known challenging failure mode case, proposed by [1] and used by several related works. \n\n>**[M&EC:]** technical novelty is limited.\n\n**[Re:]**: Our contribution lies not only in this simple and effective SSM-based approach but also in the deep understanding of PINN failure modes. Our proposed SSM-based approach is an embodiment of this understanding. Without this understanding, such an approach that can address failure modes in a wide range of equations would never have been produced. In general, we think the machine learning community always appreciates and prefers simple and effective methods.\n\nSub-sequence contrastive alignment differs from temporal contrastive learning in that, instead of a model learning contrastively for multiple similar data/frame features, it aims to allow different stages of time-varying SSM to form a consensus on the prediction of spatiotemporal collection points. Sub-sequence contrastive alignment emphasizes the ability of the model to inherit information at subsequent times and the ability to form an SMM-step-width agreement, while temporal contrastive learning focuses on representation learning of similar data features, which are not available in PINN.\n\n>**[TC:]**  Theorem 3.1 ... does not directly indicate that continuous-discrete mismatch is the key challenge in propagating initial condition in training PINN.\n\n**[Re:]** What we need to show through Theorem 3.1 is that continuous-discrete mismatch may lead to disconnections in the pattern propagation from the initial condition if only the losses at discrete collection points are optimized. This is because a pattern defined by a point may only act in its small neighborhood, and this neighborhood may not contain any other collection points, which can therefore cause the failure of propagation.\nAlso, a concurrent work, ProPINN [2] observed a lower gradient correlation phenomenon, which is empirical evidence for our proposal that continuous-discrete mismatch is the cause of propagation failure.\n\n>**[ED & A:]** Detailed Ablation and Discussion.\n\n**[Re:]** We add some ablation studies to discuss the combinatorial effect. See response to reviewer yZBS. Due to the rebuttal limit, we can't include a more detailed analytical discussion of these ablations but will do so in the next release.\n\nGiven these explanations, we sincerely invite you to reconsider the rating of our work.\n\n[1] Krishnapriyan, et al. \"Characterizing possible failure modes in physics-informed neural networks.\" NeurIPS 2021\n\n[2] Wu, et al. \"ProPINN: Demystifying Propagation Failures in Physics-Informed Neural Networks.\" arXiv:2502.00803", "claims_and_evidence": "I have concerns on the novelty of problem statement. While I agree that the continuous-discrete mismatch and simplicity bias are important to enhance the learnability of PINN, those two challenges are not considered a novel problem. As we know, there has been numerous literatures to model continuous time series from discrete one, shifting the focus of spatio-temporal modeling methods from conventional RNNs to neural ODEs and PDEs. Ultimately, they try to model continuous time series from discrete sampling. Also, I cannot directly connect if this would block the propagation of the “initial condition” (L57), as there could be more causes blocking its propagation. Simplicity bias is also an empirically very well-known problem, when comparing model and data complexity.  Hence, the author’s main problem statement “How can we effectively introduce sequentiality to PINNs?” with a toy example in Figure 1 does not sound a challenging problem.", "methods_and_evaluation_criteria": "While the motivation to deploy SSM into PINN model makes sense, its technical novelty is limited. It simply adds a SSM module between MLP modules, which could be considered a simple variation from common PINN designs. Sub-sequence contrastive alignment is straightforward and also makes sense. However, again, I doubt if we can say if it significantly differs from well-known temporal contrastive learning. Learning from sub sequences is widely used to efficiently learn long sequences with better stability. Overall, I understand the author’s motivation to use SSM and sub-sequence contrastive alignment, yet find their limited technical novelties.", "theoretical_claims": "I appreciate the author formulate Theorem 3.1. and its proof in Appendix A. However, it is not clearly connected to the proposed claim (i.e., continuous-discrete mismatch). There could be of course more solutions that satisfy discrete samples, as they are partial observations from original continuous source. However, I think it does not directly indicate that continuous-discrete mismatch is the key challenge in propagating initial condition in training PINN.", "experimental_designs_or_analyses": "More detailed experiments are required to validate the proposed methods are effective to address the continuous-discrete mismatch and simplicity bias.  For example, in section 6.4. Ablation Study, PINNMamba without Sub-Sequence Align (in Table 3) demonstrates the higher loss than the original PINNMamba. This number does not directly indicate the simplicity bias is relieved, and related discussion is missing in the section. At the same time, I think \"relieved continuous-discrete mismatch and simplicity bias\" are challening to generally validate with few empirical results, and hence it would be nice to present relevant analytical studies.", "relation_to_broader_scientific_literature": "Physics-informed neural networks (PINN) have a wide range of application, as PDEs are the fundamental component of scientific researches and applications. This paper focuses on enhancing the learnability of PINN, particularly when the discrete training data degrades its training process and the model easily traps in a naive local minimum. Such scenarios can often happen in practical setups.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes PINNMamba, a novel architecture that integrates state space models (SSMs) with physics-informed neural networks (PINNs) to address two major limitations: the continuous-discrete mismatch and simplicity bias in existing PINNs. The approach introduces sub-sequence alignment and time-varying SSMs to improve information propagation and robustness. While the technical novelty is somewhat incremental—drawing on known modeling tools like Mamba and contrastive alignment—the conceptual contribution is well-articulated and backed by compelling empirical evidence. The authors offer detailed theoretical and empirical analysis, and their ablation studies and follow-up experiments demonstrate the effectiveness and scalability of PINNMamba under varying settings. Several reviewers raised concerns regarding novelty, computational cost, and fairness of comparisons, but the rebuttal addressed these points with additional results and clarifications. The work contributes a practical and well-motivated method with strong results across multiple PDE benchmarks, and despite some limitations, it merits a weak accept.
Decision: Accept (poster)