HyResPINNs: A Hybrid Residual Physics-Informed Neural Network Architecture Designed to Balance Expressiveness and Trainability
Authors: Madison Cooley
Mike Kirby
Shandian Zhe
Varun Shankar
Venue: Accepted by TMLR ()

Abstract: Physics-informed neural networks (PINNs) have emerged as a powerful approach for solving partial differential equations (PDEs) by training neural networks with loss functions that incorporate physical constraints. In this work, we introduce HyResPINNs, a two-level convex-gated architecture designed to maximize approximation expressiveness for a fixed number of degrees of freedom (DoF). The first level involves a trainable, per-block combination of smooth basis functions with trainable sparsity, and deep neural networks; the second involves the ability to gate entire blocks (much like in ResNets or Highway Nets), allowing for expressivity along the depth dimension of the architecture. Our empirical evaluation on a diverse set of challenging PDE problems demonstrates that HyResPINNs consistently achieve superior accuracy to baseline methods while remaining competitive relative to training times. These results highlight the potential of HyResPINNs to combine desirable features from traditional scientific computing methods and modern machine learning, paving the way for more robust and expressive approaches to physics-informed modeling.

Published in Transactions on Machine Learning Research (12/2025)
HyResPINNS: A Hybrid Residual Physics-Informed Neural Network Architecture Designed to Balance Expressiveness and Trainability
Madison Cooley
mccoley Ssci utah. edu
Scientific Computing Institute University of Utah
Salt Lake City, UT
Mike Kirby
kirby Ssci utah. edu
Scientific Compuling Institute University of Utah Salt Lake City, UT
Shandian Zhe
zhe@utah edu
Kahlert School of Computing University of Ulah Sult Lake City, UT
Varun Shankar
Shankar@cs utah. edu
Kahlert School of Computing University of Utah
Salt Lake City, UT
Reviewed on OpenReview: https: // openreview. net/ forum? id= etWWkjkkgu
Abstract
Physics-informed neural networks (PINNs) have emerged powerful approach for solv- ing partial differential equations (PDEs) by training neural networks with loss functions that incorporate physical constraints. In this work, we introduce HyResPINNs, a two-level convex-gated architecture designed to maximize approximation expressiveness for fixed number of degrees of freedom (DoF). The first level involves trainable, per- combina- tion of smooth basis functions with trainable sparsity, and deep neural networks; the second involves the ability gate entire blocks (much like in ResNets Highway Nets), allowing for expressivity along the depth dimension of the architecture. Our empirical evaluation on diverse set of challenging PDE problems demonstrates that HyResPINNs consistently achieve superior accuracy to baseline methods while remaining competitive relative to train- ing times. These results highlight the potential of HyResPINNs to combine desirable features from traditional scientific computing methods and modorn machine learning, paving the way for more robust. and expressive approaches to physics-informed modeling.
1 Introduction
Partial differential equations (PDEs) underpin the simulation of complex physical phenomena across physics and engineering, from turbulent. flows to electromagnetic waves and quantum systems (Evans) 2010 LeV- eque 2007). However, the interplay of nonlinearities, intricate geometries, and temporal dynamics often ren- ders analytic solutions intractable, motivating need for robust and adaptable numerical methods (Strang et al. 1974 LeVeque 2007). While classical solvers such finite element, finite difference, and spectral methods have established foundation for computational science, recent. advances in machine learning

---

Published in Transactions on Machine Learning Research (12/2025)
have led to physics-informed neural networks (PINNs). which offer flexible, data-driven approach to PDE approximation (Rasssi et al. 2019a Karniadakis et al. 2021
Despite these advances, a central challenge remains: how can we balance expressivity, adaptivity, and effi- ciency to obtain accurate, tractable solutions for increasingly complex problems? The impact of these three factors differs fundamentally between classical and machine learning (ML) paradigms, creating complemen- tary strengths and weaknesses. In this work, we decompose expressivity into three interrelated components: function space richness the breadth and diversity of approximable functions; adaptivity the ability to allocate expressive power selectively (whether across input regions or within the model's internal structure); and efficiency per DoF-the contribution of cach paramcter to the overall approximation. These criteria jointly determine a solver's ability to capture complex solution structures while mainlaining computational tractability. The central motivation of this study is to combine the controllable adaptivity and efficiency of classical methods with the richness and flexibility of neural networks, unifying their advantages within a hybrid framework.
Classical numerical methods offer rigorous convergence guarantees and explicit, controllable adaptivity allowing targeted mesh or basis refinement ( dAinsworth & Oden, 1997). Such refinement is often guided by prior knowledge of solution smoothness or domain geometry (Brenner & Scott 2008 Strang et al. 1974). leading to high efficiency per DoF, where each additional parameter meaningfully improves accuracy. How- ever, their function space richness is constrained to the chosen basis or kernel, limiting the diversity of representable functions without significant model redesign. Adaptive extensions such as hp-adaptive finite elements (Schwab) 1998) or greedy kernel selection (Schaback Wendland| 2 can expand or restruc- ture the function space, but operate through discrete, manually designed rules within fixed basis family. In contrast, ML-based neural architectures adjust capacity continuously during training, reweighting and recombining learned features without altering the underlying discretization. Further, maintaining accuracy in complex scenarios, such as nonsmooth features, irregular geometries, or nigh-dimensional interactions, escalates the roquired DoFs, resulting in incrcased computational costs (Babuska & Suri, 1994). Thus, while classical methods excel in efficiency and controllable adaptivity, they are less inherently flexible in representing highly varied solution behaviors.
Conversely, PINNs offer a mesh-free, data-driven approach that embeds physical laws directly into the neural network training. PINN approaches are promising due to their rich function spaces, enabling the approximation of complex, high-dimensional problems without the need for hand-crafted bases (Hormik (et al.] 1989,, and without requiring a priori knowledge of solution structure. The overparameterization of neural architectures enables this flexibility, but concurrently reduces the efficiency per DoF such that many parameters contribute marginally to the final accuracy (Adoock & Dexter [2021). Unlike classical methods, where DoF' efficiency is linked to convergence theory, its quantification in neural architectures remains largely empirical. Morcover, their adaptivity is largely implicit, governed by global optimization rather than targeted refinement, making it challenging to resolve localized features, sharp gradients, multiscale phenomena (Wang et al., 2021b). While adaptive PINN methods exist, achieving architecture- level, localized adaptivity comparable to classical numerical methods remains an open challenge.
A growing body of research seeks to combine the complementary strengths of classical and ML paradigms for PDE approximation. One class enriches neural networks with fixed structured bases-sute as Fourier, polynomial, or radial basis functions (RBPs) to improve efficiency per DoF or representation quality (Cooley et al.. 2025b: Wang et al., 2021b 2023b) While some adaptive basis methods exist, such trainable modes in random Fourier feature embeddings (Wang et al. 2021b), most basis-enrichment strategies fix the basis set in advance and introduce adaptivity through pruming or regularization. This imposes a predefined representat ional structure that the model can only reduce rather than expand. In contrast to the incremental refinement strategies of classical methods, this separation of adaptivity from expressivity can limit efficiency per DoF, particularly early in training, when many basis functions contribute little to the solution, or when pruning strategies are insufficient. Another class draws on mesh-refinement ideas, reallocating model effort toward difficult regions via adaptive sampling (Zeng et al.) 2022), domain decomposition (Jagtap & Karniadakis 2020), or architecture adjustments (Luo et al.) 2025 These approaches reallocate expressive power locally either by concentrating the training signal in regions of high residual (adaptive sampling) or by statically partitioning the domain so that subregions have dedicated parameters (domain decomposition and
2

---

Published in Transactions on Machine Learning Research (12/2025)
mesh-inspired designs). As result, function space richness remains unchanged, and adaptivity is achieved only indirectly through where and how the fixed capacity is applied-leaving gap for methods that can dynamically expand and redistribute capacity while blending complementary function spaces throughout training.
We address this gap with HyResPINNS- a hybrid deep residual architecture that unifies function space richness, adaptivity, and efficiency per DoF in single framework. Each residual block is a hybridization of two complementary function spaces: trainable RBF neural network (RBFNN) that leverages compactly- supported RBEss for localized, structured refinement, and deep neural network (DNN) for flexible, compo- sitional expressivity. Their outputs are merged via convex combination controlled by trainable, inter- pretable gating parameter, allowing the model to adaptively shift between global and local modes over the course of training. A second gating parameter modulates the residual (skip) connection, starting at identity and progressively "opening" to activate additional depth only when ncoded, reducing early overparameteriza- tion. Stacking these hybrid blocks- is, composing multiple blocks soquentially enriches the function space while concentrating refinement. in specific solution regions, enabling fine-grained adaptivity and efficient parameter use without reliance on static bases, coarse pruning, fixed architectural capacity.
Our key contributions are as follows:
Expanded function space richness through hybridized composition. We combine the global, compositional expressivity of deep neural networks with the localized approximation power of compactly-supporter RBFs, enabling accurate representation of both smooth global structures and highly localized or discontinuous features across diverse PDE problems.
Fine-grained adaptivity via global-loca blending and progressive depth activation. Our architecture dynamically reallocates representational power during training adjusting the balance between global and local modes and activating additional depth only when needed- to target re- finement where it is most. beneficial.
Improved efficiency per DoF through targeted capacity growth and localized enrich- ment. HyResPINNs achieve competitive or superior accuracy with fewer, more effectively utilized parameters compared to static overparameterized neural PDE solvers by starting with minimal active capacity and selectively expanding it in response to solution complexity.
Through these contributions, we establish HyResPINNs as practical approach for learning solutions to PDEs, providing a flexible framework that generalizes well across different problem domains. The remainder
of this paper organized as follows. In Section 2 we discuss related works and in Section B we review
the relevant background. Then, in Section we describe the mathematical formulation and architectural
details of HyResPINNs. Next, in Section we present numerical experiments comparing HyResPINNs to
suite of baseline neural PDE solvers, on series of challenging PDE problems. We sunmarize results
and discuss future work in Section 0
2 Related Work
In this section, we outline in more depth the approaches of classical numerical solvers and traditional PINNs. Then we review prior work hybrid architectures that combine classical and MI. paradigms, residual and stacked designs for progressive refinement, and radial basis function (RBF) methods and related kernel approaches. We organize this discussion to highlight how each class of methods addresses or falls short the three critical components of expressive PDE solvers: (i) function space richness, (ii) adaptivity, and (iii) efficiency per degree of freedom. By situating HyResPINNs in relation to these prior works, we clarify both the limitations that motivate our framework and the distinct design principles that set it apart.
Classical Numerical Solvers. Classical numerical methods such as finite element (FEM) Strang et al. 1974), finite difference (FDM) (Leeeque 2007). and spectral approaches (Shen et al. 2011) have long pro- vided foundation for PDE solvers. These approaches employ carefully engineered discretizations basis
3

---

Published in Transactions on Machine Learning Research (12/2025)
functions informed by a priori knowledge of solution smoothness and domain geometry (Brenner & Scott, 2008 Strang et al. 1974). They offer provable convergence rates and explicit adaptivity, allowing refinement to be directly targeted to specific regions of complexity. For instance, they can concentrate additional Dol's in regions with singularities or sharp gradients by refining the approximation through mesh, basis, or com- bined strategies (Ainsworth & Oden 1997 Melenk 1997). However, the required DoF can escalate rapidly in the presence of nonsmooth features, irregular goometries, or high-dimensional interactions (Schwab) 1998;
Babuska & Suri 1994 Demkowicz al. 2002
Bungartz & Griebel 2004), resulting in significant computa-
tional costs.
Physics-Informed Neural Networks (PINNs). PINNs Raissi et al. 2019a) approximate PDE so- lutions by incorporating physical laws into the neural network's s loss function. PINNs achieve expressivity through learned, mesh-free representations, with function space richness emerging from network architecture,
activation functions, and optimization processes (Hornik et al. 1989: Cybenko 1989). Universal approxi- mation thcorems guarantee that, under certain architcctural assumptions, neural networks can approximate any function arbitrarily well, suggesting their theoretical suitability for modeling complex PDEs. However, realizing this expressivity in practice is often difficult, with performance closely tied to the specific network architecture and optimization strategy (DeVore et al. 2020 Adcock & Dexter, 2021 Marwah et al. 2021 Standard PINNs exhibit implicit, global adaptivity: the network parameters are updated in response to the overall loss computed across the full domain, rather than through explicit, spatially localized refinement mechanisms typical in classical methods. Further, PINNs often perform poorly in regimes with steep gra- dients or multiscale fcatures (Wang et al., (022bb. Recent approaches to ameliorating optimization issucs include domain decomposition (X-PINNs) (Jagtap & Karniadakis 2020), gradient-enhanced training (G- PINNs) (Yu et al., 2022), ; or discretely-trained PINNs using RBF-FD approximations in place of automatic differentiation (DT-PINNs) (Sharma & Shankar 2022).
Hybrid Architectures. Efforts to hybridize PINNs with classical ideas often fall into two broad categories. The first are basis-enrichment methods which embed structured bases directly into neural architectures. Embedding known spectral or polynomial structure can yield high efficiency per DoF: each parameter has a largeted, interpretable role, and explicit representation of low- or high-frequency modes can offset the spectral bias of standard neural networks (Tancik et al. 2020 Cooley et al. [222aa). RBF-based methods, in contrast, target localized approximation and can provide improved computational efficiency when compactly supported
kernels limit the region of influence (Wang et al. 2023b Widrow & Lehr 2002 Passhauer 2007 While
some adaptive basis methods exist, such as trainable modes in random Fourier feature embeddings (Wang et al. 2021b), most ausis-enrichment strategies fix the basis set in advance and introduce adaptivity through pruning or regularization. Across these mothods, fixed, overcomplete basis sets are common, with adaptivity introduced only through pruning or regularization, which can decouple adaptivity from expressivity and limit efficiency per DoF' when pruning is coarse.
Adaptivity-focusedd hybrids, in contrast, modify PINNs through adaptive sampling, domain decomposition, or architecture-level refinements inspired by mesh adaptation. These approaches reallocate expressive power locally-cither by concentrating the training signal in regions of high residual (adaptive sampling) or by statically partitioning the domain so that subregions have dedicated parameters (domain decomposition and adaptive sampling (Zeng) et al. 2022; Lu et al., (021]. loss reweighting (MeClenny & Braga-Netoo 2020), and domain decomposi- tion (Shukla al.J [202] Jagtap & Karniadakis] 2020) have improved performance in challenging regions, but these strategies still operate primarily the data or loss level; the underlying parameter updates remain global, and truly local, mesh- like refinement within the network architecture remains largely unaddressed. tradcoff at a time.
mesh- inssh-ispiredd designs). Recent efforts to introduce local adaptivity-inchute
Overall, most hybrids address only one axis of the
Residual and Stacked Architectures. Deep residual learning and stacked architectures have been re- cently introduced in scientific machine learning to overcome optimization challenges and enhance the expres- sivity of DNNs for solving PDEs. In particular, by enabling the training of deeper models and facilitating the learning of multi-scale phenomena through incremental refinement or additive corrections (He et al., 2016: Noorizadegan et al.) 2024b). Residual connections allow deeper networks by alleviating gradient degrada-
4

---

Published in Transactions on Machine Learning Research (12/2025)
tion, while stacked or multi-stage approaches iteratively refine solutions. These designs can increase function space richness without dramatically widening single models.
For instance, PirateNets (Wang et al.) 2024) use multiple residual blocks with learned skip weights that modulate each block's influence. However, these residual weights are unconstrained and can grow arbitrarily large; complicating the interpretation of each block's non-linear transformation and potentially destabilizing training. Stacked PINNs (Howard et al.) 2025) introduce a multifidelitt paradigm in which the output of each trained network serves a low-fidelity input for the next, iteratively building up model accuracy while reducing the need for excessively large single networks. This framework incrementally increases depth by stacking new PINNs, but it does so through sequential training process that requires manual intervention at each stage. Both methods introduce partial adaptivity PirateNets modulate block contributions; stacked PINNs increase depth over stages- but neither couples adaptivity, efficiency, and richness in a principled way. Crucially, neither integrates complementary function spaces (e.g., local RBFs with global DNNs) to exploit different inductive biases within a unified refinement framework.
Radial Basis Function Networks and Kernel Methods. Radial basis functions (RBF) methods have long been used in scientific computing as a flexible, mesh-free alternative to grid-based discretizations, especially for scattered or irregular domains (Wu et al.] 2012] Fasshauer 2007). Many RBFss RBFss-spcciillly those with compact support or rapidly decaying influence enable local control in function approximation, as each basis function primarily affects specific region of the domain. This locality allows RBFs accurately
represent local solution features with minimal disturbance to other regions (Wu et al. 2012 Widrow & Lehr 2002 Chen et al. 2023). As a result, RBFs are particularly advantageous in problems where local refinement is critical, whereas global bases such as Fourier or polynomial functions are often more effective for smooth, globally distributed features but can struggle near sharp transitions due to the Gibbs phenomenon.
Despite their strengths for localized approximation, traditional RBFNNs, particularly those with fixed cen- ters, encounter the curse of dimensionality when approximating general smooth functions in high dimen- sions (Wu et al. 2012). The number of required basis functions can increase exponentially with input dimension. This contrasts with three-layer multilayer perceptrons, where the nunber of hidden units grows polynomially due to their global parameterization (Barron 2002 Wu et al.) 2012). This limitation highlights a broader challenge in approximating complicated functions: balancing local accuracy with efficient global representation. While RBFs offer locality, the field has also explored how other paradigms, such as neural networks and kernel methods, expand representational capacity. Specifically, theoretical and practical con- nections have been established between neural networks and kernel methods (Arora et al. [001 Li et al. 2020a), inspiring feature lifting architectures such random Fourier embeddings (Wang al. 2021b Li et al. 2020a) that aim to elficiently expand representational capacity. Crucially, while RBPNN- models are va lued for their mesh-free construction and strong locality, they generally lack the hierarchical, compo- sitional structure and global parameter sharing that characterize DNNs a key advantage shared by many feature lifting methods in expanding representational capacity efficiently. Yet most retain a bias toward either predominantly local (RBF-like) or predominantly global (DNN-like) representation.
Positioning of HyResPINNS. HyResPINNs directly address the limitations identified above by em- bedding trainable, compact-kernel RBFNNs and DNNs within a unified residual block architecture. Two interpretable gating parameters per block control (i) the convex combination between local (RBF) and global (DNN) modes, and (ii) the activation of the residual (skip) path, allowing progressive depth ac- tivation during training. This enables HyResPINNS to dynamically grow and redistribute capacity while blending complementary function spaces- coupling function space richness, adaptivity, and efficiency per DoF' in a single architecture. Unlike prior hybrids, this approach jointly targets all three axes, enabling fine-grained, architecturo-level refinement without manual basis selection, coarse pruning, or static overpa- rameterization. HeResPINNs exlend beyond prior residual or bybrid frameworks by introducing dual gating to simultaneously manage local local-global feature blending and progressive depth activation. The inclusion of compactly-suppe RBFs supplics explicit locality, addressing the lack of spatial adaptivity in residual and stacked architectures, while the residual gating mechanism ensures that model complexity grows smoothly with training.
5

---

Published in Transactions on Machine Learning Research (12/2025)
Operator Learning Frameworks. A parallel line of research learns mappings between function spaces malber than individual PDE solutions, including DeepONets [Goswami et al. 2023), Fourier Newral Operalors (FNOs) (Li et ol., 2020aa,, Graph Neural Operators (GNOs) (Li et al, (0206), and GeoFNOs [Li et al,, 2023). These models typically rely on data-driven training to learning operator mappings, which find solutions to a class of PDEs. In contrast, HyResPINNs solve single inslances of PDEs; however, the archilectural concepts introduced this work could extend to the operator learning settings and represent. and interesting future line of work on hybrid operalor networks.
3 Background
In this section, we review the core components underlying HyResPINNs: physics-informed neural networks (PINNs) the baseline neural PDE framework, residual network architectures for progressive refinement, and radial basis function neural networks (RBFNNs) for localized approximation.
3.1 Physics-Informed Neural Networks.
Given a spatio-temporal domain D [0, 7] C Rl+d where Q bounded spatial domain in Rd with boundary 0N, the general form of parabolic PDE is:
Ut + F(u, Vu,...)= =
(1)
where F is linear nonlinear differential operator, and denotes the unknown solution. This PDE is subject to an initial condition x00,x)= g(x) for E ft and boundary conditions B(u(t, x)) = 0 for E [0, and x E M. Here, f and g are given functions, and B denotes an abstract boundary operator.
The general formulation of PINNs (Raissi et al. 2019a) aim to approximate the unknown solution ult, x) using a representation model ug(t, x), such that ut(,, u(t, x) and 0 denotes the set of all trainable parameters of the network. The parameters 0, are optimized to minimize the composite loss function:
L((0) = Aichic + Abe ApTrr
(2)
where Lies Libes and Ly represent the loss components associated with the initial conditions, boundary conditions, and the residual of the PDE, respectively; and dies Abey and X, are corresponding weighting coefficients that balance the contribution of each loss term. These terms can be set static weights adapted during training using various techniques such as NTK weighting (Wang et al. 2021b), gradient annealing Wang et al. 2021a among others.
Each loss term in Equation 2 is computed as the mean squared error of the initial condition, boundary, and PDE residuals. Specifically, each loss term is defined as:
Asa
Nic
Lie =
Σ
|oo(0, x(,) - g(x{{))
(3)
Nic
i=1
Nbc
Abe
Lbe
13 BBuu (ther x
(4)
Nbc
i=1
Nr
Ous (Et. x + Fluel(t, x;) f(t
L,
((),x xt)
(5)
Nr
Øt
The training data points (x(
Nic i: Nr. at each iteration of gradient descent algorithm.
(ther x Nh and (t), xf can be fixed mosh or points randomly sampled
Variants of PINNs extend this basic formulation to improve stability, scalability, or compatibility with specific PDE structures. Specifically, Karniadakis and collaborators have extended these methods to conser-
vative PINNs (cPINNs) (Jagtap et al. (2020), variational PINNs (vPINNS) (Kharazmi al. 2019 parareal
PINNs (pPINNs) (Meng et al. 2020), stochastic PINNs (sPINNs) (Zhang et al. 2019). fractional PINNs

---

Published in Transactions on Machine Learning Research (12/2025)
(fPINNs) Pang al. 2018). LesPINNs (Yag et al. 2019) non-local PINNs (nPINNs) Pang al. 2020 and eXtended PINNs (xPINNs) (Jagtap & Karniadakis 2020). While differing in implementation details, all variants rely on the ability of neural networks to represent complex solution spaces without manually efficiently directing model capacity toward localized, high-
designed bases, and all face the challenge complexity regions. In this work, we will focus on application of the original collocation PINNs approach; however, the work presented herein can be applied many if not all of these variants.
3.2 Residual Networks
DNNs often encounter optimization challenges in deep architectures such as vanishing or exploding, resulting in training performance degradation. Residual Networks (ResNets) address these issues by introducing skip connections (He al.] 2016), which allow information flow directly from an carlier layer to a later layer, creating an identity mapping that ensures information is preserved even as the network deepens Noorizade- gan et al. 2024a). Formally, standard residual block expressed as:
y == F(x, W:) +x
(6)
where x represents the input to the block, F is the learned residual mapping, and is the block output.
These connections improve gradient flow, enable stable training of deeper models, and allow the network learn incremental refinements rather than full transformations from scratch (Noorizadegan et al.] 2024a In the PDE context, this incremental refinement aligns naturally with multi-scale and hierarchical solution structures, making residual architectures useful foundation for adaptivity in depth. Recent work has explored variants with trainable skip weights, allowing the effective network depth to evolve during training, though often without constraints that ensure interpretability or stability WWng et al. 2024
Name Gaussian Multiquadric
Formula (br)
Support
Smoothness
Notes
exp Vr2 c
Global
C
controls width
Global
C
controls shape; non-decaying Long-range; sets shape
Inverse MQ
Global
C
7,44
Matérn-5/2
v5r/e+ (r/e))) e-
voree e
Global
C²
the lengthscale scts support radius
Wendland C
(1 (35(g) + 18(5) 3), r<p
Compact
C
Table 1: Common radial basis functions vbrr for RBFNNs. The distance r(x, x²;wy) may be isotropic, anisotropic diagonal, or fully anisotropic (Mahalanobis), with wy denoting shape parameters (e.g., a, L, p) for the j-th RBF.
3.3 Radial Basis Function Neural Networks
A. Radial Basis Function Neural Network (RBENN) is typically a shallow network with single hidden layer of RBF units followed by a lincar output layer. The output of the network is a linear combination of the kernel activations:
[(x) f(x) = Σ oj v(r(xx xf; wy)),
(7)
j=1
where x5 are center points, Gj are coefficients, and sb chosen radial basis function kernel (see Table for examples). The influence of each RBF is determined by distance metric between the input and the j-th center xf. with additional shape parameters w/ (o.g., width or lengthscale). One metric is the isotropic Euclidean distance:
Xx Ill
(8)
Jj
7

---

Published in Transactions on Machine Learning Research (12/2025)
which applies uniform scale in all directions. Anisotropic variants allow direction-dependent scaling, such 8s:
(9)
for axis-aligned ellipsoidal support (per-dlmeensio widths )zs), more generally the Mahalanobis distance:
V (x-x T xf),
(10)
using a symmetric positive-defintte matrix A, for arbitrary orientation.
The specific choice of & determines each basis function's support, smoothness, and decay-for example, Gaussians are globally supported and C smooth, while Wendland functions are compactly-supportee and less smooth. The shape parameters (a, e, A, etc.) control locality and ionectional influence, and are often initialized to reasonable heuristics (e.g., based on data spread or uniformly), but can be further learned from data during training. RBFs with rapid docay or compact support intrinsically localize their inffuence, enabling accurale modeling of fine-scale features while minimizing impact on distant regions (Wu et al. 2012).
Summary. PINNs provide flexible, mesh-free function approximations: ResNets offer a mechanism for incremental refinement and stable training at depth; and RBFNNs give localized, interpretable basis functions well matched to targeted refinements. Together, these components form a foundation for architectures that can represent both global and local solution structures while controlling where and how model capacity is used.
4 Hybrid Residual PINNs (HyResPINNs)
We propose HyResPINNs, a deep learning framework that integrates radial basis function neural networks (RBFNNs) and deep neural networks (DNNs) within a hybrid residual architecture. This hybridization is designed to enhance function approximation for PDEs by combining the adaptive locality potential of RBFs with the expressive, hicrarchical modeling capacity of DNNs.
4.1 Hybrid Residual Block: Dual Gating for Adaptive Representation
The core compouent HyResPINNs is the hybrid residual block (see Figure I which combines two rep- resentational paradigms: an RBFNN component (Fn) and DNN component (Fx). These are combined using two trainable gating parameters one internal each block and one external across blocks to flexibly balance local and global contributions and to enable dynamic depth adaptation. We note that the degree of locality globality realized in practice depends on the kernel choice and initialization strategies in the RBFNN component, and the chosen activation function in the DNN component.
4.1.1 Internal Gating: Balancing Local and Global Features
Within each block, the oulputs of the RBFNN and DNN components are merged via trainable scalar parameter (9), transformed by a tempered sigmoid (Papernot et al. 2020) Φ(2) (z/T),, where > 0 is a hyperparameter controlling gate sharpness and enforces the weights in [0, 1]. This ensures a convex combination:
px) (p o ((aa)))))))) (x) + (010)++11000 (1-)(a()) 1 (x)
(11)
where F() (x) and 1 (x) are both projected to a common output dimension and a = tanh ensures bounded, nonlinear transformations. Low values r yield smooth mixing; high values drives near binary selection between the component outputs. This mechanism enables the network to dynamically interpolate between localized adaptation provided by RBFs especially with compact support or rapid decay-and and global, compositional trends from DNNs
8

---

Published in Transactions on Machine Learning Research (12/2025)
(x)) E Rx
Hybrid Residual Block (1)
(1- p(a())) x (x)
o(a(1) F (x)
DNN
Weight Layer
RBFNN
w E R N.XD
K RNN
Weight Layer
x E Nxd Or X E X
Figure 1: Illustration of the hybrid residual block (?), which combines contributions from RBFNN and DNN com- ponents. The input, x E Nxd for the first block (f= 1), or E RR- for inner blocks (E > 1), is first processed by the RBFNN through the kernel (()) E NxN= followed by a linear weight layer W E R NeXp to produce (x). (x). In parallel, the DNN branch applies a sequence of weight layers and nonlincarities ō to yield F (x). The two branches are adaptively fused through trainable gating parameter a producing the weighted sum of (z)) () 1 (ar + (1 (1-6(s )(a")) p (zr), which is then passed forward through the residual connection. This design allows the model to balance kernel-basse representations from the RBFNN with expressive features from the DNN, enhancing flexibility.
6(3(1))
((3(2))
Hybrid Block 1
Hybrid Block
Hybrid Block
DNN
Figure 2: Illustration of the HyResPINN architecture using L hybrid residual blocks. Each block combines outputs from an RBFNN component and DNN component (see Figure [}. The trainable skip connections ((3(5), modulate the residual flow of information across blocks. These gated skip parameters enable the network to adaptively balance local kernel-based features and global neural representations different depths, improving flexibility during training, The final output is obtained by passing the aggregated representation through terminal DNN layer to produce the final prediction y.
The p Parameter and Kernel Method Connection. The block output dimension p defines the feature lifting space where both components operate. This aligns with kernel methods, where data is explicitly mapped into a higher-dimennsioall feature space to enhance separability or approximation capabilities. As discussed in Section 2 theoretical and practical connections between neural networks and kernel methods inspire such architectures, aiming to efficiently expand representational capacity. In our design, the RBFNN and DNN components leverage this principle, enabling the subsequent layers operate on more expressive features.

---

Published in Transactions on Machine Learning Research (12/2025)
4.1.2 External Gating: Adaptive Residual Depth
A second gating parameter, ((0) is likewise passed through a tempeerature-connrlllled sigmoid to modulate the residual connection:
H = -((()) + +1 (1-6(80))((-1)
(12)
where H((-1) is the previous block's output (or the original input for I 1). This adaptive gating enables the network to learn how much each block should transform the representation, thus supporting dynamic depth during training.
4.1.3 Regularization and Initialization
We regularize galing parameters to avoid over-reliance on single pathway similar to (Howard et al. 2025
L = Aiohic + AbcLbe + Cr + Ag (a +
(13)
such that Ng is weight modulating the strength of the regularization.
We initialize d(3 0 for all I, biasing the network toward the residual (identity) mapping at the start (Hauser 2019), and ()) 0.5 to initially balance RBENN and DNN contributions. DNN weights and RBF coefficients use Xavier (Glorot & Bengio, 2010)) initialization, while the RBF shape parameters are initialized using a quantile distance heuristic between the center points in each block. The input center points are initialized using the Poisson sampling method of (Shankar 2017), and internal block center points are initialized using k-means clustering of the subsoquent block's output following (Wurzberger Schwenker 2024).
4.2 RBFNN Block Component
For imput x E Nxd, the 1-th RBFNN constructs:
K K = 10(0) (r (r(x,,x; xf: w/²))
=x6(0)
(14)
Here, 450) denotes the selected RBF (see Table 1 and r(s55 w/) is distance metric parameterized by (1) trainable shape parameters w/', which control the width or anisotropy of each basis. The kernel matrix is multiplied by a trainable weight matrix W E RNxxp, yielding the final RBFNN output:
FR (x) (x) - ( E NxP.
(15)
Fach hybrid block maintains its own set of RBF center points (x)) allowing block-specific local refinements that adapt independently across network depth.
Wendland C2 Kernel Selection. In this work, we primarily use the (2 Wendland kernel defined
as:
Ca (1- (35r" + 18r + 3),
(16)
where (+)+ = m ensures compact support and x; is the center of the i-th RBF. The variable r denotes the distance metric, which may be either isotropic (Fquation 8 or anisotropic (Fquation g or Equation IO We implement the Wendland kernels with unit support (p from Table [ such that the kernel evaluates to zero whenever 2 1, ensuring strict locality and sparse kernel matrices. Fixing p 1 lets the anisotropy control the effective radii directly, simplifying the parameterization without sacrificing expressivity. For the fully anisotropic case in Equation 10. we parameterize cach matrix Aj in Cholesky form, A₁ = M7 M,, where Mj is a trainable lower-triangular matrix with positive diagonal entries. This guarantees that Aú remains symmetric positive-definite during training, which ensures well-defined distances, avoids degeneracies, and is practically efficient (Wurzberger Schwenker 2024). Empirically, ablation studies confirmed that the C2 Wendland RBF achieves the lowest approximation error compared to alternative kernels, justifying its use in our architecture.
10

---

Published in Transactions on Machine Learning Research (12/2025)
4.3 DNN Block Component
Each DNN block component computes:
(x) = NN RNNxp
(17)
where ((0) represents the set of trainable parameters in the network. The final output is projected to the p-dimensional feature space shared by the RBFNN component. By design, this block can use any stan- dard or advanced DNN architecture; in some experiments, we employ Modulated Multi-Layer Perceptrons (ModMLPs) with random weight factorization (Wang et al. 2021a), which are an exlension to standard fully connected works inspired by neural attention mechanismss. These modifications incur relatively small com- putational and memory overhead while leading to significant improvements in predictive accuracy (Wang et al. 2
4.4 Full Model Composition
The complete HyResPINN model (see Figure 2 consists of L stacked hybrid residual blocks. After the final block, a standard DNN head further refines the representation into the final prediction:
u DNN out (H())).
(18)
In some examples, we initialize the final layer of the final DNN using the physics-informed approach of (Wang al. 2024). least-squares fit to the PDE residuals at collocation points before gradient training which embeds physics priors and accelerating convergence.
Summary. By fusing RBFNNs with DNNs in residual, gated architecture, HyResPINNs achieve function space richness, fine-grained adaptivity, and improved parameter efficiency. Our experiments demonstrate that this approach generalizes robustly across canonical and irregular PDE benchmarks, outperforming state-of the- neural PDE solvers in both accuracy and parameter efficiency.
Problem
PINN
ResPINN
ExpertPINN StackedPINN PirateNet.
HyResPINN
Allen Cahn (Dirchlet BCs)
2.65e=1
6.82e-2
9.66e-3
6.11e-2
1.10e-2
2.39e-3
Allen-Cahn (Periodic BCs)
5.26e-1
2.70e-3
3.86e-3
5.87e-3
2.62c-5
1.19c-5
DarcyFlow (2D Dirichlet BCs)
7.50c-4
4.60e-4
5.00c-4
9.00c-4
8.71e-5
5.44c-5
(2D Neumann BCs)
2.00e-3
1.40e-3
1.20e-4
4.10e-3
1.70c-4
6.00e-5
(3D Dirichlet BCs)
2.20e-3
1.20e-2
2.10e-2
5.40e-2
3.90e-2
1.20e-3
(3D Neumann BCs)
8.50e-3
6. 6.103
1.10e-3
3.90e-2
1.30e-3
1.10e-3
DarcyFlow (rough)
6.85e-5
2.69e-5
1.10c-4
1.50c-4
5.44e-5
1.05e-5
Kuramoto- Sivashinsky
Kuramoto-SSvashinsky
1.42o-1
1.29e-2
1.57-33
1.86e-
2.00e-3
8.12e-4
Korteweg De Vries
8.17e-1
5.72e-1
6.86e-4
2.98e-1
5.61e-4
4.09e-4
Grey Scott (u)
7.54e11
9.87e-3
4.79e-3
1.21e-3
3.61e-3
6.43e-3
Grey-Scott (v)
9.56e-1
1.10e-2
8.98e-3
6.90e-2
9.39e-3
1.31e-2
Table 2: Relative l2 test error results across a range of PDE benchmarks under differrnt boundary conditions. For the Grey-Scott PDE, errors are reported separately for both components (u and v). As a summary table, the lowest. error for each problem (i.e., each row) is highlighted in bold and second-lowest errors are underlined. Note that in some cases, competing methods arrive at similar results.
5 Experimental Results and Discussion
We evaluate HyResPINNs on a suite of PDE benchmark problems designed to test its function space richness, adaptivity, and efficiency per DoF. These benchmarks have been selected either model solution features that present significant challenges for computational models are recognized as established benchmark problems. First, to evaluate localized adaptivity, we consider two 1D non-linear hyperbolic Allen-Cahn
11

---

Published in Transactions on Machine Learning Research (12/2025)
equations, one with Dirichlet boundary conditions in Section 5. and periodic boundary conditions in Section 5.2 each containing sharp solution features. In Section 5.3 we test 2D and 3D Darey flow problems on annulus and extruded anmulus domains to evaluate generalization to non-standard geometries using various boundary conditions. Then, Section 5.4 details a 2D Darcy flow problem with rough, discontimuous coefficients to examine robustness under discontinuities. To further explore the function space richness of our methods, we evaluate the chaotic and multi-scale systems given by the Kuramoto-Sine -Svashiisyy equation in Section 5.5 the Korteweg-de Vries equation in Sections 5.6 and the Grey Scott equation in Section 5.7 We assess predictive accuracy and efficiency per DoF throughout all experiments, comparing against state- of-the-art baselines. Table summarizes the top results from each test.
Baseline Methods We compare HyResPINNS against suite of baseline methods. These include 1) the standard PINN as originally formulated in (Raissi et al. 2019b) (PINN); 2) a PINN based on the architectural guidelines proposed in (Wang et al. 2023a) (ExpertPINNs); 3) PINNs with residual connections (ResPINNs); 4) PirateNets (Wang et al. 2024); and 5) the stacked PINN approach of (Howard et al. 2025) (StackodPINNss) This baseline suite was selected to capture the ineremental steps leading to the proposed architecture. Each comparison isolates one primary architectural mechanism ResPINNs isolate residual skip connections; ExpertPINNs isolate Fourier feature embeddings and high-dimensional residual skip connections; PirateNets isolate adaptive skip connections within block structure; StackedPINNs isolate convex combinations between two types of approximators.
Collectively, these baselines incrementally probe the three axes of expressivily, adaptivily, and efficiency per DoF. Standard PINNs rely on a fixed DNN representation with limited expressivity and no adaptive structure. ResPINNS extend this baseline by incorporating residual skip connections, improving training stability and adaptive depth utilization, while moderately improving efficiency. ExpertPINNs enhance expressivity through Fourier feature embeddings, while PirateNets focus on adaptivity by learning dynamic skip connections that regulate residual information flow. StackedPINNs increase expressivity through convex combinations of multiple sub-networks. Together, these methods progressively explore the awwpsiiiittyddaaiiittt-ttiei within single architecture implemented each approach following the architectural details provided to ensure a fair comparison.
space that HyResPINNs unify local and global representations through dual learnable gates. We
We follow similar experimental design procedures those described in (Wang et al, 2022a 2023a 2024). We employ mini-batch gradient descent for most experiments, with collocation points randomly sampled during each training iteration. In the Darcy Flow experiments, we use full-batch gradient descent, with collocation point sets generated using the efficient Poisson sampling technique from (Shankar 2017). For training, we use the Adam optimizer Kingmna & Ba (2015), and follow the learning rate schedule of (Wang et al. 2023a) which starts with a linear warm-up phase of 5, 000 iterations, starting from zero and gradually increasing to 10-3, followed by an exponential decay at rate of 0.9. We also employ learning rate annealing algorithm (Wang et al. 2023a) to balance each loss term and causal training (Wang et al. 2022a: 2023a to mitigate causality violation in time-dependent PDEs. Further, we apply exact periodic or Dirichlet boundary conditions (Dong & Ni 2021) when applicable. We use the hyperbolic tangent activation functions in all DNNs and initialize cach network's paramoters using the Glorot normal scheme (Glorot & Bengio 2010). We ran five random trials for each lest and report the mean values achieved in each plot and table. The code for our methods and all compared baseline approaches is written in Jax 0.6.q We train all models an Nvidia H100 GPU running CentOS 7.2.
Experimental Setup
Comparison Metrics We measure the quality of cach model's predicted solution ug using the relative l error, defined as Eg = l|u high-fidelity solver). Here |l refers to the discrete l norm evaluated over set of equispaced test points in the domain, which serves as an approximation of the contimuous (2( norm. We quantify model efficiency as a function of accuracy, parameter count, and computational cost. Formally, we define the efficiency per DoF (denoted 7/DO)) as:
||ue-u"|le where u² denotes the reference solution (either exact or computed using a
M/D Eg N Nparams T Terain
(19)
http:///gthuuuccc///acaccccs///p
12

---

Published in Transactions on Machine Learning Research (12/2025)
Reference Solution
Prediction (RBFNN + DNN)
Absolute Error (RBFNN + DNN)
1.00
1.00
1.00
0.75
0.75
0.75
0.50
0.50
0.50
0.25
0.25
0.25
0.03
0.00
0.00
0.00
0.25
-0.25
-0.25
-0.50
-0.50
-0.50
0.01
-0.75
-2
00.5
-2
-0.7
-1.00
-1.00
-1.00
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
t
t
t
Prediction (DNN Only)
Absolute Error (DNN Only)
1.00
1.00
0.75
0.75
1.9
0.50
0.50
0.25
1
0.25
-0.25
-0.25
1.5
-0550
-0550
-0.75
0.75
-1.00
11.00
0.0
0.0
0.2
0.4
0.6
0.B
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Prediction (RBFNN Only)
Absolute Error (RBFNN Only)
1.00
1.00
0.75
0.75
1.75
0.50
0.50
1.50
0.25
0.25
1.25
x
0.00
0
X
0.00
-0.25
-0225
0.50
-0.75
0.75
0.75
-1.00
11.00
0.00
0.4
(a) Comparison of the three variants: HyResPINN (RBFNN+DNN), DNN-only, and RBFNN-only's predicted solutions (middle) and the absolute (right) compared to the true solution (left).
RBFNN-only support map
Hybrid (RBFNN DNN) support Distribution of Learned Sigma Norms
1.00
RBFNN-onhy
Hybrid (RDFNN DNN)
0.50
0.50
0.25
0.25
Count
0.00
0.00
0.25
0.25
-050
0.50
10-
-0.75
0.75
-1.00-
0.0
0.2
0.4
0.6
0.8
1.0
-1.00
0.0
0.4
0.6
1.0
0
0.75
0.75
1.25
1.75
2.25
RBFNN Sigma Norm (Width)
(b) (Left) RBF Support Map. Hybrid model exhibits stronger kernel coverage sharp solution features. (Right) Sigma Norms. Distribution of learned RBF widths. Hybrid learns broader, smoother kernels.
Figure 3: Allen Caln equation (Dirichlet BCs): Comparison of model outputs and internal representations across variants.
13

---

Published in Transactions on Machine Learning Research (12/2025)
where Npprame is the number of trainable parameters (the model's degrees of freedom), and Tuain represents the mean computational cost of training, measured as the average number of epochs completed per second under identical hardware and batch configurations. Intuitively, a model with higher MDo achieves more accurate solutions relative to its complexity and computational cost, resulting in higher efficioncy per DoF.
To further analyze efficiency per DoF, we decompose T/DaP into two complementary metrics that isolate different aspects of model performance:
1
params
Eg -NNparams
params
Moomp
Eg -Trrain
(20)
Here, Tparams (parameter efficiency) quantifies the accuracy achieved per degree of freedom. Conversely, Mcmmp (computational efficiency) measures accuracy per unit training time, indicating how effectively the model architecture and optimization routine utilize computational resources.
Intuitively, a model can minimize the error &g to an arbitrary extent by increasing the mumber of trainable parameters N Nprrams extending training time Tarain- However, doing so diminishes its overall efficiency. In practice, two models with identical accuracy can exhibit markedly different efficiency values: one achieving low error compactly and quickly, the other reaching the same accuracy only through overparameterivaaioo or prolonged optimization. Thus, high accuracy alone does not imply superior performance model that achieves slightly higher error yct trains faster or with fewer degrecs of freedom may, in effect, be more efficient. Together, these fine-grained metrics provide a comprehensive view of efficiency per DoF, clarifying how different design choices balance expressivity, computational cost, and solution accuracy.
5.1
1D
Allen- Cahn with Dirichlet Boundary Conditions
We start by considering the one-dimensional Allen Cahn equation defined on æ E [-1, 1], t E [0, 1] given as:
du 82u 06
8 dx2
5u + 5u,
where u(2z is the solution and 0.001 is the diffussio coefficient. We impose the homogeneous Dirichlet boundary conditions:
u(-1,t) 00 u(1,t)= = 0, for all t E [0,
and the initial condition is:
((x 0) = -2sin
8C0-000 (2 (2e
- ---5(e+0.))
This nonlinear, reaction-diffusion PDE's solution exhibits both smooth, global variation and sharp, localized peaks (centered at x -.5).. This problem designed to evaluate model's capacity to accurately capture these localized features while simultaneously preserving the global solution structure.
We first perform ablation studies to systematically investigate the individual contributions of the RBFNN and DNN components within the HyResPINN residual blocks. Next, to measure the DoF efficiency, we benchmark the complete HyResPINN model against baseline solvers with comparable model complexity, quantified by parameter count and network depth. Appendix A details ablation studies mcasuring the accuracy and computational cost of the HyResPINN architecture under varying RBF kernel types, numbers of center points, and deepening residual blocks. For the remainder of the experiments, we fix the RBF kernel to Wendland C the configuration our ablation studies identify the most effective concerning accuracy, computational efficiency, while aligning with our goal of inherent locality.
Expressivity: Local and Global Representations. We first examine the function space richnoss of HyRcsPINNs by isolating local (RBFNN), global (DNN), and hybrid (RBFNN+DNN) representations, testing whether their combination enables more expressive solution approximation than either alone. To isolate the contri- butions of each hybrid component, we conduct ablation experiments comparing RBFNN-only, DNN-only, and full HyResPINN configurations. In the RBFNN-only configuration, the DNN outputs are fixed to zero, isolating localized basis function representation. In the DNN-only configuration, the RBFNN outputs are
14

---

Published in Transactions on Machine Learning Research (12/2025)
fixed to zero, isolating global neural representation. In the full HyResPINN, both components active and combined via learned gating. All models are trained with Adam for 100,000 iterations, with batch size of 1024 collocation points randomly sample each iteration. The DNNs use three layers of 64 neurons, RBFNNs use 64 centers, and HyRosPINNs use both and one block.
As shown in Figure 3al the full hybrid model achieves the lowest relative error (2.46-3) -3), accurately resolving both the steep gradients and the global structure. The DNN-only baseline captures the overall shape but
fails to resolve sharp peaks, resulting in localized errors above 1.0 (relative error 2.65e 1). The RBFNN-only baseline captures localized peaks but struggles to maintain smoothness elsewhere (4.6e - 3). Neither baseline achieves the accuracy or robustness of the hybrid. These results directly confirm that hybrid local-global representation enables superior function approximation on challenging, multiscale PDEs.
0.010
0.50
0.008
0.49
0.48
0.006
0.004
0.46
0.45
0.002
0.44
0.43
0.000
50000
100000
150000
200000
0
50000
100000 150000
200000
Iterations
Iterations
((p))
((B²)
(B2))
((ag)
((a,)
((az)
Figure 4: Allen Cahn equation (Dirichlet BCs): Evolution of the learned gating parameters during training. (Teft) Evolution of residual gating parameters ((B,), which modulate the strength of residual information flow across blocks. (Right) Evolution of hybrid gating parameters Φ(a)), which balance the DNN and RBF components within each block.
Adapticity: Solution Refinement. We next examine how HyResPINNs achieve fine-grained adaptivity in the width of RBF kernels. Kernel support maps (Figure 3b left), computed by summing RBF activations across the domain, reveal that the hybrid model develops broad, overlapping support, densely covering regions with sharp features. In contrast, the RBFNN-only model forms narrowly focused kernels with little overlap. Furthermore, the distribution of learned RBF widths (Figure 3b] right) demonstrates that HyResPINNs favor wider, smoother kernels compared to RBFNNs alone. This broader support reflects the model's ability to blend basis functions, promoting both smooth transitions and precise local refinement- key ingredients of adaptive expressivily.
Additionally, we conducted an ablation study analyzing the impacts of different parameter initialization strategics; we report this in Appendix [A.3] Overall, the results indicated that the model is largely robust to initializations of both parameters apart from when ø(a) is initialized to 1 and Φ(3) is initialized to 1. Therefore, opt to initialize x(3) to O such that the start is the identity and initialize ω(a) to 0.5 so the components start with equal contribution. Figure | shows the evolution of the gating parameters during training. The o(3) gating parameters increase steadily during training, indicating the progressive strengthening of residual information flow across the blocks. The hybrid gating parameters Φ(ar) initially a stronger reliance on the RBFNN components- before stabilizing at values that indicate moderate preference for the RBFNN.
Efficiency: Accuracy and Converyence per Degree of Freedom. We evaluate the empirical efficiency of HyRosPINNS relative to competing PINN basclines under comparable parameter budgets, focusing on both the error convergence behavior and overall accuracy. As shown in Figure 5] HyResPINNs achieve lower mean relative le error and faster convergence than the baseline models. Table 3 further quantifies these trends using the empirical efficiency metrics defined in Equation [9 and Equation 20 Although HyRosPINNs do not exhibit the lowest computational cost (training fewer iterations per second than some alternatives), they achieve the highest parameter efficiency (Jparams 2.358e 2), and highest computational efficiency
15

---

Published in Transactions on Machine Learning Research (12/2025)
Mean Relative Error
10a
10a
Mean PDE Loss
Mean ICS Loss
10-1
20
10-1
10-1
10
10-
10-2
10-3
10-2
10-2
10-4
10-1
50000
150000
200000
50000
150000
200000
150000 200000
Iterations
Iterations
Iterations
PINN
StackedPINN
ResPINN
PiratePINN
ExpertPINN
HyResPINN
Figure 5: Allen-Cahn equation (Dirichlet BCs): (Left) Mean relative & errors, (middle) PDE loss, (right) initial condition loss, across all baselines of similar parameter numbers as a function of training iteration.
Model
params
T Train
Rel & Error
7arrams
7comp
M/DF
PINN
17,025
8264 (232)
4.98e-02 (0.00986)
1.179e-3
2.430e-3
1.427e-7
StackedPINN
17,359
5705 (7)
2.83e-02 (0.04217)
2.036c-3
6.194c-3
3.568c-7
ResPINN 17,157
6734 (136)
7.18e-02 (0.03379)
8.118e-4
2.068e-3
1.205e-7
PiratePINN 17,926
4623 (26)
1.70e-02 (0.00848)
3.281e-3
1.272e-2
7.098e-7
ExpertPINN 17,793
5394 (52)
6.80e-03 (0.00025)
8.265e-3
2.726e-2
1.532e-6
HyResPINN 17,667
5549 (9)
2.40c-03 (0.00021) 2.358c-2
7.508c-2
4.250e-6
Table 3: Allen-Col Cahn equation (Dirichlet BCs): Comparison of model complexity and efficiency across PINN variants. Reported are the total number of trainable parameters (Nparams); the computational cost, ex- pressed as the average number of training iterations completed per second (Tirain) with standard deviation in parentheses; the mean relative l2 error with standard deviation in parentheses; and the corresponding efficiency metrics Tparams, 7omp:: and IDDoF. Higher efficiency values indicate models that achieve lower rela- tives crrors relative their reprosentational and computational costs.
HyResPinn Prediction
ResNet Prediction
PireateNet Prediction
ExpertPian Prediction
StackedPinn Prediction
+0.5
-0.5
-1.2
-1.0
-3.0
-30
-20
Absolute Error
Absolute Error
Absolute Error
Absolute Error
Absolute Error
0.00
5000
003
084
015
5005
002
005
65
Figure 6: Allen-Cahn equation (Dirichlet BCs): Predicted solutions (top row) and corresponding absolute errors (bottom row) for various baseline models. Each column shows the model's prediction across the full domain, followed by the absolute errors relative the reference solution.
16

---

Published in Transactions on Machine Learning Research (12/2025)
(ecmm = 7.508e 2). resulting in the highest overall composite efficiency (MDoF = 4.250e 6). This indicates that HyResPINNs attain lower errors relative to their representational complexity and runtime cost. Quali- tative comparisons in Figure further show that only HyResPINNs robustly capture both sharp and smooth solution features, whereas most bascline methods fail to recover the sharp solution features, exhibiting large absolute errors near x = +..5.
Summary. These analyses highlight the benefit of hybridization: the RBENN component provides localized adaptivity, while the DNN component ensures global coherence. HyResPINN's lower error and more adap- tive internal structure demonstrate that coordinated local-glober representation is essential for robust and expressive PDE learning.
5.2 1D Allen- Cahn with Periodic Boundary Conditions
Next, we present results for 1D the Allen-Cahn problem with periodic boundary conditions, well-known challenge for conventional PINN models and a widely studied benchmark in recent literature (Wight &
Zhao 2020 Wang et al. 2022a Daw et al. 2022p. For simplicity, we consider the one-dimensional case with
a periodic boundary condition with E [0, 1], and æ E [-1,
U - 0.0001ugr + 5u 5u = 0,
u(0, x) = x² cos (OT))
u(t, -1) = u(t, 1), ur (t, -1) = ur (t,
This problem is particularly difficult for two reasons. First, the Allen-Cahn PDE requires PINNs to ap- proximate solutions with sharp transitions in space and time, posing challenges for standard neural network architectures. Second, there is an inherent incompatibility between the initial and boundary conditions, lead- ing to a weak discontinuity at time = O. Traditional numerical methods would likely produce oscillations near the discontinuity due to the mismatch between the initial and boundary conditions. However, unlike traditional solvers, PINNs do not rely discretization and instead learn their model parameters through optimization. This optimization process may mitigate spurious oscillations, as it relies on stochastic updates and continuous function approximations. Specifically, implicit regularization from both mini-batch training and stochastic gradient descent (SGD) may contribute to stabilizing optimization and reducing any spurious oscillations (Sekhari et al. 2021 Full experimental details are provided in Appendix
Expressivily: Capturing Periodic and Disconlinuous Solutions. Similar to the previous section, we compare HyResPINNs to DNN-only and RBFNN-only variants. Figure demonstrates the advantages of the hybrid residual block structure in capturing both smooth and sharp features in the ID Allen-Cahn solution. The absolute error plots show that HyResPINN exhibits significantly lower errors than standard DNN-only and RBFNN-only, validating its improved accuracy.
Model
N params
T train
Rel l Error
maraans
7somp
iDDF
PINN
724,353
1947 (6)
4.54c-03 (0.00010)
3.038c-4
1.130c-1
1.560c-7 7
ExpertPINN
727,170
1309 (10)
3.86e-05 (0.00004)
3.563e-2
1.979e1
2.721e-5
PiratePINN
727,171
1355 (4)
2.62e-05 (0.00001)
5.249e-2
2.816e1
3.873e-5
ResPINN 790,145
2114 (11)
1.53e-03
(0.00016)
8.285e-4
3.096e-1
3.919e-7
StackedPINN
793,738
833 (3)
5.87e-03 (0.00012)
2.146e-4
2.044e-1
2.576e-7
HyResPINN 759,686
827 (2)
1.19c-05 (0.00001) 1.106e-1
1.016e2
1.337e-4
Table 4: Allen-Cahn equation (Periodic BCs): Comparison of model complexity and efficiency across PINN variants. Reported are the total number of trainable parameters (Nparams); the computational cost, ex- pressed as the average number of training iterations completed per second (Timin) with standard deviation in parentheses; the mean relative la error with standard deviation in parentheses; and the corresponding efficiency metrics Mparams, Nomp:: and DDDF Higher efficiency values indicate models that achieve lower rela- tives errors relative to their representational and computational costs.
Adaptivity: Depth Ablation and Solution Refinement. To test architectural adaptivity, we vary the number of residual blocks. D that increasing the number of residual blocks in HyResPINNs leads to
17

---

Published in Transactions on Machine Learning Research (12/2025)
HyResPINN
PINN
RBFNet
1.0
0.0
-0.5
0.5
1.0
0.0
0.0
0.5
1.0
0.0
0.5
1.0
Error
Abs. Error
Abs.
1.0
0.0
-0.5
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.25
0.50
0.99
M
-0.2
0.66
Exact
HyResPINN
PINN ROFPINN
-1.0
x
Figure Allen- Cahn equation (Periodic BCs): Comparison of the predicted solutions for the Allen-Cahh equation using HyResPINN, standard PINN, and RBF PINN models. The top row shows the predicted solutions for HyResPINN (left), standard PINN (center), and RBF PINN (right). The second row shows the absolute error between the predicted and true solutions. The bottom row shows the predicted solutions for time steps (t = 0.25, 0.5, 0.99) compared to the true solution.
18

---

Published in Transactions on Machine Learning Research (12/2025)
Mean Relative 12 Error
10
10
10-3
4
8
10
12
Number of Hidden Layers
ResPINN
PiratePINN StackedPINN
PINN
HyResPINN
ExpertPINN
Figure 8: Allen- Cohn equation (Periodic BCs): Comparison of the mean relative l error using various
methods as function of the number of hidden layers.
0.51
0.4
0.50
0.49
0.3
0.48
0.2
0.1
80000
160000
240000
80000
160000 Iterations
240000
Iterations
(Ba)
(B²)
(B2))
((ao)
((a,)
((az)
Figure 9: Allen-Cahn equation (Periodic BCs): Evolution of the learned gating parameters during training. (Left) Evolution of residual gating parameters ((B), which modulate the strength of residual information flow across blocks. (Right) Evolution of hybrid gating parameters ((a)), which balance the DNN and RBF components within each block.
consistently lower errors, outperforming the competing methods. Residual learning allows deeper networks refine predictions iteratively, progressively improving both global structure and sharp transitions. The competing methods struggle maintain accuracy as depth incrcases, whereas HyResPINNs remains robust, demonst.rtiing the effectiveness of deep hybrid residual learning. Specifically, the standard PINNs prediction error increases the network becomes deeper, eventually failing to capture any part of the solution. ResNets errors also increase with increased depth, though at slower rate than the standard PINN. This is likely due to the static skip connection in the standard residual architecture design, preventing the adaptive learning of residual connections.
Figure g illustrates how the learned gating parameters evolve during training of HyRosPINNs. The residual gating parameters o(8) increase steadily during training, indicating the progressive strengthening of resid- ual information flow across the blocks. The hybrid gating parameters ((a)) initially decrease-reflee reflecting
19

---

Published in Transactions on Machine Learning Research (12/2025)
10
Mean Relative 1, Error
10 1
10
10
10
10
0
50000
100000
150000 Iterations
200000
250000
300000
ResPINN
PiratePINN PINN
EXpertPINN StackedPINN
HyResPINN
Figure 10: Allen- Cahn equation (Periodic BCs): Comparison of mean relative & error across various meth- ods, plotted against the number of training iterations.
stronger reliance on the RBFNN components before stabilizing at values that indicate a moderate prefer- ence for the RBFNN.
Efficciecy:: Accuracy and Convergence per Degree of Freedom. Next, we assess the empirical efficiency of HyResPINNs relative to competing PINN basclines under comparable parameter budgets. As shown in Figure 10 HyResPINNs initially converge at similar rate to PiratePINNs, but ultimately achieve a slightly lower mean relative & error than all other baselines. Table | further summarizes the efficiency trade-offs defined in Equation 19 and Equation 20 While HyRcsPINNs exhibit the highest computational cost among the baselines (Terain 827), they achieve the lowest mean relative l error of 1.19e 5. Consequently, HyResPINNS attain the highest parameter efficiency (7params = 1.106e - 1), and the highest computational efficiency (Mommp 1.016e2), with the overall composite efficiency reaching T/DoF 1.337e 4. Although Rommp is only moderately higher than that of ExpertPINN (Ncmmp = 1.979e1), this improvement combined with the lower error demonstrates that HyResPINNs achieve superior accuracy-efficiency balance without requiring additional model capacity.
Summary. These results demonstrate that the hybrid, gated architecture of HyResPINNs enables improved accuracy and efficiency compared conventional PINN variants. Despite comparable parameter counts, HyResPINNs achieve lower errors, underscoring the benefits of hybridization and the enhanced representa- tional depth provided by the adaptive residual connections.
5.3 Darcy Flow with Smooth Coefficients
We next test the performance HyResPINN on the Darcy Flow equation with smooth coefficients. Darcy Flow serves as a fundamental model for various physical processes, including porous media flow, heat transfer, and semiconductor doping. In most applications, the flux Vo is the variable of primary interest. The Darey Flow problem is an elliptic boundary value problem given by:
-V vV in Ω $ = u on Tp n V = on In
(21)
where TD and Ny denote Dirichlet and Neumann parts of the boundary I, respectively, / is symmetric positive definite tensor describing a material property and f, and g are given data.
20

---

Published in Transactions on Machine Learning Research (12/2025)
1.0
0.0
0.0
-0.4
0.88
-0.5
00.5
-0.5
-1.0
0.0
0.4
0.8
0.4
-0.8
0.8
Figure 11: Smooth Darcy Flow equation:
True solutions of the 2D and 3D problems.
We manufactured solutions in both 2D and 3D domains, ensuring that the exact solution contains global structure and cross-dimensional correlations. Specifically, the exact solutions are given as:
u(I,,) sin(w)) cos()ry),
(22)
u(z, y. 2) sin(TT) sin sin(
(23)
for the 2D and 3D cases, respectively. Figure displays the target solutions for both cases.
To rigorously evaluate data efficiency, adaptivity, and convergence, generated static training datasets for all experiments, fixing the set of collocation points within each run. This design enables controlled studies of error reduction as the number of training points increascs, while climinating the variability introduced by stochastic sampling. We systematiiaaly varied the number of collocation points to perform convergence studics, and solved each problem under both Dirichlet and Neumann boundary conditions in both 2D and 3D domains. This approach allows for a fair, direct comparison of accuracy, data efficiency, and robustness across problem dimensions and boundary types. All baseline models were matched in parameter count and trained under identical conditions for consistent benchmarking.
Expressivily: Accurate Solution of Smooth Multidimenssinaal PDEs. HyResPINN accurately reconstructs these solutions and achieves low mean relative & errors across all tested training set sizes (see Figure 12 solid lines), indicating its capacity to represent smooth solutions in higher dimensions. Notably, HyResPINN's predictions for both the solution and the physically relovant flux quantity remain stable and accurate even as dimensionality increases.
Adaptivity: Robustness to Boundary Conditions and Domain Complerity. HyResPINN maintains high ac- curacy across both types of boundary conditions in 2D and 3D, whereas bascline methods often exhibit significant. degradation or instability, particularly for Neumann constraints (see Figure 12 where dashed lines denote flux errors). The model also scales from the 2D annulus to the more complex 3D cylindri- cal domain, demonstrating flexibility in handling increased geometric and dimensional complexity without additional tuning.
Figure illustrates the evolution of the learnod gating parameters for the 2D Dirichlet boundary condition case. The residual gating parameters d(3) increase steadily during training, indicating the progressive strengthening of residual information flow across the blocks. The hybrid gating parameters Φ(a) initially docrease-reflecting stronger reliance on the RBFNN components-bone a moderate preference for the RBFNN. Similar gating dynamics were observed for the 2D Neumann case and the 3D cases, as reported in Appendix
stabilizing at values that indicate
Efficiency per Degree of Freedom: Data Efficiency and Convergence with Statie Datasets. Table and Figure 12 show that HyResPINNs achieve lower or comparable solution and flux errors competing models across nearly all configurations, except in the smallest-data regime, where ResPINNs attain the lowest mean
21

---

Published in Transactions on Machine Learning Research (12/2025)
Dirichlet Boundaries
Neumann Boundaries
Mean Relative Ia Error
Mean Relative Ia Error
10-2
10-3
10
10-4
10a
104
10a
10*
Training Set Size
Training Set Size
Dirichlet Boundaries
Neumann Boundaries
10-1
Mean Relative 1, Error
10-1
Mean Relative F2
10-2
10-2
10-3.
10-3
104
10a
Training Set Size
Training Set Size
PINN
PiratePINN StackedPINN
ExpertPINN HyResPINN
ResPINN
Figure 12: 2D and 3D Smooth Darey Flow equations: Comparison of the mean relative & errors across each baseline method for the 2D Darcy Flow problem (top) and the 3D Darcy flow problem (bottom) plotted against the number of training collocation points using both Dirichlet and Neumann boundary conditions. Solid lines show the solution error, while dashed lines show the x-directional flux errors.
0.50
0.012
0.011
0.49
0.010
(33
0.48
0.009
0.47
0.008
0.007
0.46
25000
50000
75000
100000
0
25000
50000
75000
100000
Iterations
Iterations
(BB))
((B))
((B2)
$(ap)
(la,)
((az)
Figure 13: 2D Smooth Darcy-Flot equation (Dirichlet BCs): Evolution of the learned gating parameters during training. (Left) Evolution of residual gating parameters @(Bi), which modulate the strength of residual information flow across blocks. (Right) Evolution of hybrid gating parameters o(a:), which balance the DNN and RBF components within each block. Similar convergence trends were observed for the 2D Neumann case and the 3D cases (see Appendix
22

---

Published in Transactions on Machine Learning Research (12/2025)
Model
N params
Trrin
Rel & Error
M params
DoF
2D Smooth Darcy Flow
Dirichlet BCs
PINN
14,913
196 (7)
7.54e-04 (0.00010)
8.889e-2
6.760e0
4.533e-4
ExpertPINN
15,298
114 (5)
5.09c-04 (0.00003)
1.285e-1
1.730e1
1.131e-3
PiratePINN
15,302
115 (3)
8.71e-05
(0.00001)
7.503e-1
9.989e1
6.528e-3
ResPINN
12,737
196 (12)
4.62e-04
(0.00009)
1 1.699e-1
1.104el
8.667e-4
StackedPINN
13,642
122 (3)
9.00c-04 (0.00007)
8.145e-2
9.111e0
6.679e-4
HyResPINN
14,248
125 (2)
5.44e-05 (0.00002)
1.290e0
1.471e2
1.032--2
Neumann BCs
PINN
14,913
192 (6)
2.05e-03 (0.00040)
3-27le-2
2.537e0
1.701e4
ExpertPINN
15,298
112 (5)
1.29c-04 (0.00003)
5.083e-1
6.921c1
4.524c-3
PiratePINN
15,302
112 (3)
1.71e-04 (0.00003)
3.822e-1
5.205e1
3.401e-3
ResPINN
12,737
192 (11)
1.40e-03 (0.00007)
5.600e-2
3.709e0
2.912e44
StackedPINN
13,642
120 (2)
4.10c-03 (0.00009)
1.788c-2
2.024c0
1.484c4
HyResPINN
14,248
123 (2)
6.00e-05 (0.00004) 1.170e0
1.350e2
9.475e-3
3D Smooth Darcy Flow
Dirichlet BCs
PINN
14,945
189 (6) 2.05e-03 (0.00010)
3.257e-2
2.580e0
1.726e-4
ExpertPINN
15,266
111 (4)
2.10c-02 (0.00034)
3.118c-3
4.284c-1
2.806c-5
PiratePINN
15,270
110 (3)
3.90e-02 (0.00029)
1.679e-3
2.333e-1
1.528e-5
ResPINN
12,753
189 (11)
1.21e-02 (0.00076)
6.501e-3
4.394e-
3.445e-5
StackedPINN
13,834
119 (2)
5.49c-02 (0.00040)
1.317c-3
1.530c-1
1.106c-5
HyResPINN
14,344
115 (2)
1.25e-03 (0.00002)
5.575e-2
6.958e0
4.85le-4
Neumann BCs
PINN
14,945
185 (6)
8.25e-03 (0.00023)
8.106e-3
6.542e-1
4.377e-5
ExpertPINN
15,266
110 (4)
1.00c-03 (0.00034)
6.547e-2
9.095e0
5.957e-4
PiratePINN
15,270
109 (3)
1.31c-03
(0.00019)
5.010e-2
7.038e0
4.609e-4
ResPINN 12,753
185 (10)
6.10e-03 (0.00071)
1.285e-2
8.850e-1
6.939e-5
StackedPINN
13,834
119 (2)
3.90e-02 (0.00023)
1.853e-3
2.154e-1
1.557e-5
HyResPINN 14,344
115 (2)
1.00e-03 (0.00003) 6.969e-2
8.697e0
6.063e-4
Table 5: 2D and 3D Smooth Dorey Flow equations: Comparison of model complexity and efficiency across PINN variants for both Dirichlet and Neumann boundary conditions for the largest-data regime. Reported are the total number of trainable parameters (NPpramms; the computational cost, expressed as the average number of training itcrations completed per second (Tirain) with standard deviation in parentheses; the mean relative le error with standard deviation in parentheses; and the corresponding efficiency metries 1pparam: Teomps and 7DD... Higher efficiency values indicate models that achieve lower relatives errors relative to their representational and computational costs.
relative & errors for both 2D Dirichlet and Neumann cases. The advantages of HyResPINNs become most evident in the large-data regime- specifically, as the dataset size increases, HyResPINNs continue to reduce error, whereas other models plateau or even degrade in accuracy. When accounting for both parameter count and computational cost. HyResPINNs outperform the other basclines, achieving the highest composite efficiency metric (7Do)) across all cases. In terms of computational efficiency (Nemmp), HyResPINNs achieve the highest values in all but the 3D Neumann case, where ExpertPINNs perform slightly better. Nevert
23

---

Published in Transactions on Machine Learning Research (12/2025)
HyResPINNS consistently exhibit the largest Mparams: indicating that they maintain balanced parameter efficiency across problem dimensions and boundary conditions.
Summary. HyResPINN demonstrates (i) strong function space expressivity for smooth multidimensionall PDEs, (ii) robust adaptivity to varying boundary conditions and domain complexity, and (iii) superior efficiency per degree
competitive accuracy and convergence across a range of training set sizes. These results further validate the hybrid residual framework an effective and scalable solver for elliptic PDEs in both 2D and 3D settings.
5.4 Darcy Flow with Rough Coefficients
1.00
1.00
0..50
7.50
0.25
0.5
4.20
2.55
0.00
0.90
0.0
0.2
0.4
0.6
0.2
Figure 14: (Teftt) True solution on domain. (Right) True transverse flux.
Accurately solving the Darey Flow equation with discontinuous coefficients is fundamental challenge for both numerical and machine learning-based PDE solvers. In heterogeneous materials, the solution and flux field must correctly capture interface behavior: while the normal component of the flux remains continuous, the tangential component may be discontinuous. We focus on the well-studied "five strip problem," well- documented manufactured solution test (Trakk et al.) 2017 Nakshatrala et al.) 2006 Masud & Hughos, 2002), which evaluates the ability of a method to preserve the continuity of normal flux across material interfaces. This problem divides the domain Ω [0, 1]2 into five horizontal strips, each with a distinct diffusion coefficient μ4: Specifically, the prescribed exact solution domain [0, 1J2 under Neumann boundary conditions is given by:
= x, and Iw =
(24)
such that Ω divided five equal strips,
Ω (x, 0.2(i - 1) < y 1 0.21; 0 1}. i= 1,..,,
(25)
with different on each Ω such that μ사 = 16, P2 = 6, #s = 1, μ4 = 10, 1s = 2. Figure 14 shows the true solution and corresponding transverse flux.
Expressivily: Capturing Discontinuous and Physically Consistent Flur. The rough-coefficient Darcy Flow problem requires a model capable of representing changes material propertics and the resulting discontin- uous flux behavior. HyResPINN's compositional block representation enables it accurately approximate both the solution and its flux, even in the presence of abrupt material transitions.
Adeptivity: Robustness to Material Interfaces. Robust adaptivity is crucial for climinating spurious oscil- lations and enforcing correct flux continuily al interfaces. Unlike classical collocated methods which fre- quently exhibit oscillatory flux or fail to preserve normal continuity at discontimuities HyResPINN adapts representation to maintain the correct physical behavior of both normal and tangential flux components. The method handles Neumann boundary conditions and variable μ4, confirming its architectural flexibility for rough, heterogeneous domains.
24

---

Published in Transactions on Machine Learning Research (12/2025)
Figure 15 illustrates the evolution of the learned gating parameters during training. The residual gating parameters, ((3), increase steadily during training, indicating progressive strengthening of residual in- formation flow across network blocks. The hybrid gating parameters, Φ(a), initially decrease reflecting an carly reliance on the RBFNN components- before stabilizing to block-dependent values: blocks and converge to moderate preferences for the RBFNN, whereas block stabilizes at a slightly higher ((a1), indicating mild preference for the DNN component.
0.0105
0.0100
0.500
0.495
0.490
0.485
0.0065
25000
75000
25000
50000
75000
100000
Iterations
Iterations
(Ba)
((B1)
(1
((a)
)(a,)
((az)
Figure 2D Rough Darcy- Flow equation: Evolution the learned gating parameters during training. (Teft.) Evolution of residual gating parameters 6(8.), which modulate the strength of residual information flow across blocks. (Right) Evolution of hybrid gating parameters p(a=), which balance the DNN and RBF components within each block.
Neumann Boundaries
10-2
Mean Relative f Error
10-3
10-4
10-3
10a
10a
104
Training Set Size
PINN
PiratePINN StackedPINN
ExpertPINN HyResPINN
ResPINN
Figure 16: 2D Rough Dorey Flow equation: Comparison of the mean relative & errors across various methods, plotted against the number of training collocation points. Solid lines show the solution error, while dashed lines show the x-directional flux errors.
Efficiency per Degree of Freedom: Converyence with Incrensing Dala. To evaluate data efficiency, we perform convergence studies by progressively increasing the number collocation points while maintaining compara- ble parameter counts across all models. Figure [6] HyResPINN achieves lower solution and flux errors data size increases, similar Section 5.3 While convergence on rough problems requires more data, HyResPINNs ult iltimately reach the lowest relative & error. Furthermore, Table 0 reports the efficiency metrics across all
25

---

Published in Transactions on Machine Learning Research (12/2025)
Model
params
T train
Rel Eg Error
Mparams
Moo
PINN
14,913
196 (2)
6.84e-05 (0.00001)
9.803e- 1
7.456e1
5.000e-3
EXpertPINN
15,298
114 (1)
1.11e-04 (0.00003)
5.910e-1
7.957e1
5,201e-3
PiratePINN
15,302
115 (1)
5.44c-05
(0.00001)
1.201c0
1.599c2
1.045o-2
ResPINN 12,737
196 (2)
2.69e-05
(0.00001)
2.919e0
1.896e2
1.489e-2
StackedPINN 13,642
122 (2)
1.50e-04 (0.00001)
4.887e-1
5.467e1
4.007e-3
HyResPINN 14,248
125 (1)
1.05e-05 (0.00001) 6.684e0
7.619e2
5.347e-2
Table 6: 2D Rough Darcy Flow equation: Comparison of model complexity and efficiency across PINN variants for the largest-data regime. Reported are the total number of trainable parameters (Nparams); the computational cost, expressed the average number of training iterations completed per second (Tuaai) with standard deviation in parentheses; the mean relative & error with standard deviation in parentheses; and the corresponding efficiency metries Mparams, Neomp TJDu Higher efficiency values indicate models that achieve lower relatives errors relative to their representational and computational costs.
baselines, showing that HyResPINNs achieve the highest overall efficiency (7/oF = 5.347e - 2), along with the largest computational efficiency (Ncomp = 7.619e2), and parameter efficiency (7paraas 6.684c0).
Summary. HyResPINN demonstrates (i) strong expressivity for non-smooth PDEs, (ii) robust adaptivity to
discontinuous material interfaces, and (iii) high efficiency per DoF achieving physically consistent solutions and fluxes rough-cocfficient Darcy Flow problems.
5.5 Kuramoto-Sivashing Equation
In this section, we use the Kuramoto Sivashinsky equation as benchmark problem under periodic boundary conditions. The Kuramoto Mtramoto--ivashhinkky equation is fourth-order nonlinear partial differential equation widely used to model spatiotemporal patterns in fluid dynamics, combustion, and other systems exhibiting chaotic behavior. We consider the 1D Kuramoto Sivashinsky equation of the form:
du du pu 8²uu Ox
8 Ōt Ox
au
axs
0.
(26)
subject to periodic boundary conditions and an initial condition:
u(0, x) =x) cos(+) sin(z)))
(27)
The parameters a, 33, and control the dynamical behavior of the equation. We use the configurations: a = 100/16, 3 = 100/162, and = 100/162 for chaotic behaviors. Solving this equation allows us to assess HyResPINN's ability to approximate highly complex and chaotic systems. Following (Wang et al. 2023a), we employ a time-marching training routine to facilitate stable optimization. HyResPINN and all baseline models are trained using the same stne-stepping scheme and comparable parameter budgets. Full experimental details provided in Appendix D
True Solution
Predicted Solution
Absolute Error
0.010
0.008
4-
4-
0.006
X
0.004
1
1-
1-
2
1-
0.002
0.000
0.1
0.2
0.3
0.4
0.0
0.1
0.2
0.3
0.4
0.0
0.1
0.2
0.4
Figure 17: Kuramoto
Kuramoto-Sivsshinst Sivashinsky equation: Comparison of the best prediction against the reference solution.
26

---

Published in Transactions on Machine Learning Research (12/2025)
Model
N params
T train
Rel & Error
marrams
7comp
iDoF
PINN
724.353
1961 (7)
1.42c-01 (0.00300)
9.717e-6
3.590e-3
4.956e-9
ExpertPINN
727,170
1553 (9)
1.57e-03 (0.00024)
8.756e-4
4.099e-1
5.637e-7
PiratePINN
727,171
1520 (5)
2.00e-03
(0.00020)
6.876e-4
3.289e-1
4.523e-7
ResPINN
790,145
1631 (8)
1.56c-02
(0.00090)
8.113e-5
3.929e-2
4.973e-8
StackedPINN
793,738
909 (1)
1.86e-01 (0.00210)
6.773e-6
5.914e-3
7.451e-9
HyResPINN
759,686
909 (1)
8.12c-04 (0.00072) 1.620c-3
1.354c0
1.782c-6
Table 7: Curamoto-Sivashiie equation: Comparison of model complexity and efficiency across PINN vari- ants. Reported are the total number of trainable parameters (Nparams); the computational ional cost, expressed as the average number training iterations completed per second (Tirain) with standard deviation in paren- theses; the mcan relative l error with standard deviation parentheses; and the corresponding efficiency metrics Mparams: Noomp. and IDD.. Higher efficiency values indicate models that achieve lower relatives errors relative their representational and computational costs.
0.5
0.4
0.50
0.49
(1))
0.48
0.2
0.47
0.1
0.45
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Iterations
166
Iterations
1ee
((Bo)
((B1)
((22)
((ag)
(((rs)
((az)
Figure 18: Kuramoto Sivashinsky equation: Evolution of the learned gating parameters during training. (Left) Evolution residual gating parameters ((3,), which modulate the strength of residual information flow across blocks. (Right) Evolution of hybrid gating parameters @(a)), which balance the DNN and RBF components within cach block.
27

---

Published in Transactions on Machine Learning Research (12/2025)
Erpressivity: Capturing Chaotic and Multi-scale Dynamics. The Suramoto-SSvashinseee equation presents chaotic, multi-scale temporal behaviour and sharp spatial gradients that challenge standard PINN archi- tectures. HyResPINN demonstrates strong expressivity by accurately recovering the chaotic solutions generated by the Kuramoto- Sirasshissy equation, closely matching the reference solution shown Fig- ure 17
Adapticity: Robustness to evolving spatiotemporal features. HyResPINNS adapt to evolving chaotic spatial features, maintaining a high accuracy relative to the baselines through the time- time-marching scheme. Figure illustrates the evolution of the learned gating parameters under the time-marching training routine. The residual gating parameters o(3) increase rapidly within each time segment and then reset at the start of the next window, reflecting the progressive strengthening and reinitialization of residual information flow across blocks. In contrast, the hybrid gating parameters o(a) initially decrease and then stabilize within each win- dow, indicating a dominance the RBFNN components before rebalancing toward a mixed DNN-RBFNT representation. These cyclic patterns in á(a) and ((3) highlight how HyResPINNs dynamically adapt their represental ional balance al each stage of temporal evolution, enabling stable and accurale long-term inte- gration of chaotic PDEs.
Efficiency per Degree of Freedom. In terms of efficiency per DoF, HyResPINNs achieve the lowest mean relative l2 error (8. (8.12e-4) among all tested methods, as summarized in Table 7 They also obtain the highest parameter efficiency (7params = 1.620e - 3), computational efficiency (7omm = 1.354e0), and composite efficiency (VDOF 1.782e--00 indicating strong accuracy relative to representational complexity and training cost. These results highlight that HyResPINN sustain high predictive accuracy even in chaotic regimes where other PINN architectures deteriorate.
Summary. HyResPINNS demonstrate (i) strong expressivity for chaotic, spatiotemporally complex PDEs, (ii) robust adaptivity to evolving multi-scale features, and (iii) high empirical efficiency per NoF-achhevingg accurate, physically consistent predictions for the Kuramoto-Sivashinsky equation.
5.6 Korteweg-De De Vries Equation
Next, explore the one-dimensional Korteweg De Vries (KdV) equation, fundamental model used to describe the dynamics of solitary waves, solitons. The KIV equation is expressed as follows:
"t + nuu, + j LE (0,1), x (-1,1,,
u(x, 0) cos (TII) u(t, 11 u(t, 1),
where 17 governs the strength the nonlinearity and μ controls the dispersion level. Under the KdV dynamics, this initial wave evolves into a series of solitary-type waves. We adopt the classical parameters the KdV equation, setting 1) = and μ 0.022 (Zabusky & Kruskal 1965). Full experimental details are provided in A Appendix D
True Solution
Predicted Solution
Absolute Error
1.0
2.5
1.0
2.5
1.0
0.0035
2,0
2,0
0.5
0.5
0.5
0.0025
0.0020
0.0
0.0
0.0
0.5
0.5
0.0015
0.0
0,0
0.0010
-1.0
0.0000
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Figure 19: Korteweg-De Vries equation: Comparison of the best prediction against the reference solution.
28

---

Published in Transactions on Machine Learning Research (12/2025)
Model
params
T train
Rel & Error
mpraams
7cmmp
DoF
PINN
724,353
1947 (6)
8.17e-01 (0.00300)
1.689e-6
6.285e-4
8.676e- 10
ExpertPINN
727,170
1553 (9)
6.87e-04 (0.00010)
2.003e-3
9.377e-1
1.289e-6
PiratePINN
727,171
1520 (5)
5.61e-04
(0.00004)
2.451e-3
1.173e0
1.612e-6
ResPINN 790,145
1631 (8)
5.73e-01
(0.00190)
2.210e-6
1.071e-3
1.355e-9
StackedPINN 793,738
909 (1)
2.98e-01 (0.00210)
4.228e-6
3.693e-3
4.652e-9
HyRosPINN 759,686
885 (1)
4.09c-04 (0.00007) 3.214o-3
2.759c0
3.632e-6
Table 8: Korteweg-Di De Vries equation: Comparison of model complexity and efficiency across PINN variants. Reported are the total number of trainable parameters (Nparams); the computational cost, expressed as the average number training iterations completed per second (Turii)) with standard deviation parentheses; the mean relative l error with standard deviation parentheses; and the corresponding efficiency metrics Tparama: Neomp, and IDFF Higher efficiency values indicate models that achieve lower relatives errors relative to their representational and computational costs.
0.10
0.52
(8)
0.06
0.48
0.46
0.04
0.42
80000
160000
240000
80000
160000
240000
Iterations
Iterations
((Bo)
((B1)
((B2)
((ao)
((a1)
((az)
Figure 20: Korteweg-De Vries equation: Evolution of the learned gating parameters during training. (Left)
Evolution of residual gating parameters ((5,), which modulate the strength residual information flow across blocks. (Right). Evolution of hybrid gating parameters d(a;), which balance the DNN and RBF components within cach block.
29

---

Published in Transactions on Machine Learning Research (12/2025)
Solution (u-field)
1.0
1.0
1.0
0.8
0.8
0.05
0.5
0.5
0.04
0.6
0.6
0.5
0.5
0.5
0.5
0.4
0.01
-1.0
0.3
1.0
0.3
1.0
1.0
True Solution (-field)
Predicted Solution
Absolute
0.35
0.30
0.25
0.25
0,0
0.20
0.15
0.5
0.10
-0.5
0.10
-0.5
0.01
-1.0
-1.0
-0.5
Figure 21: Grey-Scott equation: Comparisons between the solutions predicted by trained HyResPINN and the reference solutions.
Capturing Dispersive and Nonlinear Wave Dynamics. The Korteweg-Da De Vries features strongly nonlinear, dispersive behaviour that produces stable solitary waves. HyResPINNs accurately reconstruct these soliton dynamics, as illustrated in Figure 19
Expressivity:
Preservation of Localized Wave Features. The model adaptively resolves the highly localized soliton structures and evolving nonlinear features across the domain, demonstrating its ability to allocate expressive capacity where needed as the solution evolves. Figure 20 illustrates the evolution of the learned gating parameters. The residual gating parameters ((3) increase steadily during training, indicating the progressive strengthening of residual information flow across the blocks. The hybrid gating paramoters d(a) steadily decrease then stabilize reflecting stronger reliance on the RBENN components.
Adeptivity:
Efficiency per Degree of Freedom: achieve the lowest relative & error (4.09e 4) among all tested models. They also exhibit the highest empirical elficiency per degree of freedom, outperforming baselines with closely matched parameter budgets. These results confirm that HyResPINNs achieve high accuracy and efficient approximation of dispersive nonlinear PDEs, effectively balancing accuracy and computational cost.
Trade-offs. As summarized in Table HyResPINNs
Summary. HyResPINNs demonstrate strong expressivity for nonlinear, dispersive wave dynamics, robust adaptivity in preserving localized soliton structures, and high efficiency per DoF- DoF-achieving accurate and computationally efficient solutions for the Korteweg-De Vries equation.
5.7 Grey-Scott equation
In this example, we solve the 2D Grey-Scott equation, reaction-diffusion system that describes the inter- action of two chemical species. The form of this PDE is given follows
= f Au + 61 u) cuuv, t (0,2), (z.v) E (-1,1)2 Av 22 ozwo?, te (0,2), (z,y) (-1,
30

---

Published in Transactions on Machine Learning Research (12/2025)
Model
N params
T trrain
Rel & Error
aaramss
7comp
Doo
Grey
Scott: u-field solution
PINN
724,353
990 (17)
7.54c-01 (0.00300)
1.830c-6
1.339c-3
1.849--9
ExpertPINN
727,170
804 (23)
4.80e-03 (0.00030)
2.867e-4
2.593e-1
3.566e-7
PiralePINN
727,171
795 (15)
3.61e-03
(0.00040)
3.809e-4
3.484e-1
4.791e-7
ResPINN
790,145
824 (20)
9.80c-03 (0.00090)
1.291c-4
1,238c-1
1.566e-7
StackedPINN
793,994
455 (4)
1.21e-03 (0.00098)
1.04le-3
1.818e0
2.290e-6
HyRes PINN
759,814
455 (3)
6.43e-03 (0.00090) 2.047e-4
3.421e-1
4.503e-7
Grey-Scott: Scott: u-field solution
PINN
724.353
990 (17)
9.56o-01 (0.00300)
1.444-66
1.056o-3
1.458c-9
ExpertPINN
727,170
804 (23)
8.99c-03 (0.00030)
1.530c-4
1.384e-1
1.903e-7
PiratePINN
727,171
795 (15)
9.81e-03 (0.00040)
1.402e-4
1,282e-1
1.763e-7
ResPINN 790,145
824 (20)
1.10e-02 (0.00090)
1.151e-4
1.103e-1
1.396e-7
StackedPINN 793,994
455 (4)
6.90c-02 (0.00030)
1.825c-5
3.188e-2
4.015e-8
HyResPINN
759,814
455 (3)
1.31e-02 (0.00090) 1.002e-4
1.676e-1
2.205e-7
Table 9: Grey Scott equation: Comparison of model complexity and efficiency across PINN variants for the u- field (top) and v-field (bottom) solutions. Reported are the total number trainable parameters (N (Nparams); the computational cost, expressed the average number of training iterations completed per second (Tirain) with standard deviation in parentheses; the mean relative l error with standard deviation in parentheses; and the corresponding efficiency metrics 1params: 7copp: and MDF.. Higher efficiency values indicate models that achieve lower relatives errors relative to their representational and computational costs.
0.51
0.4
0.50
0.3
(
0.49
0.48
0.1
0.0
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
Iterations
l05
Iterations
105
((Bo)
((8,)
((82)
((ag)
((a,)
((az)
Figure 22: Grey-Scott equation: Evolution of the learned gating parameters during training. (Left) Evolution of residual gating parameters d(3+), which modulate the strength of residual information flow across blocks. (Right) Evolution of hybrid gating parameters Φ(a,), which balance the DNN and RBF components within each block.
31

---

Published in Transactions on Machine Learning Research (12/2025)
subject to the periodic boundary conditions and the initial conditions
"o(x,1) = 1 - exp(- + 0.05)² + (v + 0.02)))), m0(2r = 1 exp(-10((1 - 0.05)² + (y 0.02)2))
Here and represent the concentrations of the two species. The system can generate wide range of patterns, including spots, stripes, and more complex forms, depending on the parameters chosen. For this example, we set é1 = 0.2, 0.1, bj = 40, by = 100, c1 c 100. We employ standard time-marching strategy to train our PINN models, as outlined in various studies (Wight & Zhao 2020 Wang et al. 2023a) Full experimental details are provided in Appendix F
Expressivily: Capluring Mulli-comppnent Pallerns. For the Grey-Scout reaction diffusion system, HyResPINNS capture the essential spatiotemapora patterns. While not achieving the lowest overall er- rors among the testod models, the predicted u- and e-field solutions remain consistent with the reference patterns, as shown in Figure 21 demonstrating that the hybrid iormulation can represent both localized and diffuse structures.
Adeptivity: Evolving Spatiotemporal Features. The complex, timo-varying nature of the Grey-Scote system requires adaptive modeling of spatially localized features and their dynamic interactions. HyResPINNs dy- namically resolve these interactions by adaptively allocating expressive capacity where needed as the solution evolves in each time-marching window. Figure 22 illustrates the evolution of the learned gating parameters under the time-marching training routine. The residual gating parameters 6(3) increase rapidly within each time segment and then reset at the start of the next window, reflecting the progressive strengthening and reinitialization of residual information flow across blocks. In contrast, the hybrid gating parameters Φ(a) initially decrease and then stabilize within each window, indicating a dominance of the RBFNN components before rebalancing toward a mixed DNN RBENN representation.
Efficiency per Degree of Freedom: Accuracy- Accuraey-Efficieney Trade-offs. As summarized in Table 9 HyRcsPINNs achieves competitive error rates compared to the baseline PINN models, with StackedPINNs attaining the lowest u-field relative lg error (1.21e - 3) and ExpertPINN the lowest e-field relative &g error of (8.99e - 3). While these results indicate that HyResPINNS are not the most accurate computationally efficient model for this particular system, they remain comparable across all efficiency metrics. The reduced performance likely arises from the strong stiffness and multiscale coupling between the reaction and diffusion terms, which pose challenges for the hybrid RBENN-DNN representation. In particular, the current gating structures may be too restrictive, limiting the model's ability to simultancously capture diffusive behavior and rapidly varying reaction dynamics, making it difficult to maintain an optimal balance across space and time. Future work could address these limitations by introducing higher-dimensional adaptive skip connections, allowing the model to better accommodate the distinct scales present in stiff, multi-component systems.
Summary. HyResPINNS exhibit comparable accuracy for capturing the culti-component pattern formation present in the Grey-Scott problem. Although not the most accurate or computationally efficient model for this problem, the hybrid architecture remains competitive compared to the baseline methods. These results highlight the challenges posed by stiff, multiscale coupling in such systems and suggest that further adaptivity in the gating or skip-connection design could enhance performance in coupled problems.
6 Conclusion
In this work, we introduced HyResPINNs, a novel class of physics-informed neural networks that integrate adaptive hybrid residual blocks, combining the complementary strengths of DNNs and RBFNNs. The use of adaptive gating parameters allows the model to dynamically balance the contributions of DNN and RBFNN components throughout training, resulting in enhanced expressivity and adaptivity. Through our experiments on a diverse set of benchmark PDEs, we demonstrated that HyResPINNS generally outperforms taate-of-the-are neural PDE solvers in accuracy and greater empirical efficiency, as quantified by our proposed motrics that account for error, computational cost, and representational capacity. Our method excels particularly in problems featuring mixed smooth and non-smooth solution features, where existing approaches often strugll..
32

---

Published in Transactions on Machine Learning Research (12/2025)
Limitations and Future Work. While HyResPINNs demonstrate clear advantages in handling PDEs with sharp gradients and non-standard domains, several limitations remain. First, the inclusion of both RBF- based and DNN components increases the number of hyperparameters and architectural choices, introducing additional challenges in model selection and training. Hyperparameter tuning, in particular, can become computationally demanding for large-scale problems. Second, our current experiments are restricied moderately sized PDEs; scaling HyResPINNs to higher-dimensional systems will require further investigation into both efficiency and numerical stability. Finally, extending the HyResPINN framework toward more general learning paradigms suchh such operator teariing-representts represents promising direction for future work.
Broader Impacts This work contributes to the field of scientific machine learning by proposing a hybrid architecture for solving PDFs. The potential impact of HyResPINNs lies primarily in improving the accuracy of PDE solvers, which may accelerate progress in fields such as physics and engineering, As the method focuses on numerical modeling and does not directly process personal or sensitive data, we do not foresee any negative societal or ethical consequences.
Acknowledgments
This research was supported by the U.S. Army Research Office (ARO) Award No. received support from the National Science Foundation under Award No. NSF SHF-2403379. We additionally acknowledge the computational resources provided by the University of Utah Center for High Performance Computing (CHPC) and the use of large language models (LLMs) for manuscript editing.
VS
References
Ben Adcock and Nick Dexter. The gap between theory and practice in function approximation with deep neural networks, 2021. URL https::/arxii.. hwtp://wwww/aaab/1..0.020
Mark Ainsworth and J Tinsley Oden. A postcriori error cstimation in finite element analysis. Computer methods in applied mecharics and engineering, 142(1-2):1-88) 188, 1997.
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. Advances neural information processing systems, 32, 2019.
Ivo Babuska and Manil Suri. The p and hp versions of the finite element method, basic principles and properties. SIAM Review, 36(4):578-633 -638, 1994.
Andrew R Barron. Universal approximation bounds for superpositions of sigmoidal function. Transactions on Information theory, 39(3):930-99) 2002.
IEEE
Susanne C. Brenner and L. Ridgway Scott. The Mathematical Theory of Finite Element Methods. Springer, 2008.
Hans-Joachim Bungartz and Michael Griebel. Sparse grids. Acta numerica, 13:147-269, 2004.
Zhang Chen, Zhong Li, Liangchen Song, Lele Chen, Jingyi Yu, Junsong Yuan, and Yi Xu. Neurbf: A neural fields representation with adaptive radial basis functions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4182-4194 2023.
Madison Cooley, Varun Shankar, Mike Kirby, and Shandian Zhe. Fourier PINNs: From strong boundary conditions to adaptive fourier bases. Transactions on Machine Learning Research, 2025a. ISSN 2835-8856.
URL /ttpp:///penrrevie met/forum?idd-aaaaee
Madison Coolcy, Shandian Zhe, Robert M. Kirby, and Varun Shankar. Polynomial-augmenttd neural net- works (panns) with weak orthogonality constraints for enhanced function and pde approximation. arXiv preprinl 2406.02336, 2025b.
33

---

Published in Transactions on Machine Learning Research (12/2025)
Steven M Cox and Paul C Matthews. Exponential time differencing for stiff systems. Journal of Computa- tional Physics, 176(2):430-4553 2002.
George Cybenko. Approximation by superpositions of sigmoidal function. Mathemalics of control, signals and systems, 2(4):303 314, 1989.
Arka Daw, Jie Bu, Sifan Wang, Paris Perdikaris, and Amuj Karpatne. Rethinking the importance of sampling in physics-informed neural networks. arXiv preprint Xix:22207. 02338, 2022.
Leszek Demkowicz, John Kurtz, David Pardo, Maciej Paszynski, Waldemar Rachowicz, and Adam Zdunek. Fully automatic hp-adaptive 2d fem algorithms. Computer Methods in Applied Mechanics and Engineering, 191 2002.
Ronald DeVore, Boris Hanin, and Guergana Petrova. Neural nelwork approximation, 2020. URL https:
//arxiv.org/abs/20012 14501
Suchuan Dong and Naxian Ni. A method for representing periodic functions and enforcing exactly periodic boundary conditions with deep neural networks. Journal of Computational Physics, 435:110242, 2021.
Tobin A Driscoll, Nicholas Hale, and Lloyd N Trefethen. Chebfun guide, 2014.
Lawrence C. Evans. Partial Differential Equations. American Mathematical Society, 2nd edition, 2010.
Gregory F. Fasshauer. Meshfree approzimation methods with Mallab ( (With Cd-rom), volume 6. World
Scientific Publishing Company, 2007.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249 256. JMLR Workshop and Conference Proceedings, 2010.
Somdatta Goswami, Aniruddha Bora, Yue Yu, and George Em Karniadakis. Physics-informed deep neural operator networks. In Machine learning in modeling and simulation: methods and applications, pp. 219- 254. Springer, 2023.
Michael Hauser. On residual networks learning a perturbation from identity, 2019. URL https: pettss//arce //axxvi.
org/abs//902.04100 04106
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Newral Networks, 2(5):359-366) 1989. doi:
Amanda A Howard, Sarah H Murphy, Shady E Ahmed, and Panos Stinis. Stacked networks improve Allyics-inffrmer training: Applications to neural networks and deep operator networks. Foundations of Data Science, 7(1):134 162, 2025.
Ameya D Jagtap and George Em Karniadakis. Extended physics-informed neural networks (xpinns): A gen- eralized space-time domain decomposition based deep learning framework for nonlinear partial differential equations. Communicaatins in Compulational Physics, 28(5):2002-20) -2441, 2020.
Ameya D. Jagtap, Ehsan Kharazmi, and George Em Karniadakis. Conservative physics-informent neural net- works on discrete domains for conservation laws: Applications to forward and inverse problems. Computer Methods in Applied Mechanics and Engineering, 365, 2020. doi: (0.1016/j.cma.20000 113028.
George Em Karniadakis, Ioannis G Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang. Physics- informed machine learning. Nature Reviews Physics, 3(6):422 -440, 2021.
E. Kharazmi, Zhang, and G. E. Karniadakis. Variational physics-infone informed neural networks for solving partial differential equations. Xiv preprint 191.00300 2019.
34

---

Published in Transactions on Machine Learning Research (12/2025)
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2013. http:://axxi...rg/////////////// attgg//b//1/...000000
Randall J LoVeque. Finite difference methods for ordinary and partial differential equations: steady-state and time-dependent problems. SIAM, 2007.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for paramelric partial differential equations. arXiv preprint XrXiv:2010. 08895, 2020a.
Zongyi Li, Nikola Kovachki, Kamyar Azizzaenesheeli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Graph kernel network for partial differential equations. arXiv preprint Xrx::2003. 08485, 2020b.
Zongyi Li, Daniel Zhengyu Huang, Burigede Liu, and Anima Anandkumar. Fourier neural operator with learned deformations for pdes on general geometries. Journal of Machine Learning Research, 24(388): 1 26, 2023.
Lu Lu, Xuhui Meng, Zhiping Mao, and George Em Karniadakis. Deepxde: A deep learning library for solving differential equations. SIAM review, 63(1):208 228, 2021.
Dawei Luo, Soo-Ho Jo, and Taejin Kim. Progressive domain decomposition for efficient training of physics- informed neural network. Mathemalics, 13()), 2025. doi: 1.3390/matth1300
Tanya Marwah, Zachary C. Lipton, and Andrej Risteski. Parametric complexity bounds for approximating pdes with neural networks, 2021. URL
Arif Masud and Thomas JR Hughes. A stabilized mixed finite element method for darcy flow. Computer methods in applicd mechanics and enginecring, 191(339400)4341144370 2002.
Levi McClenny and Ulisses Braga-Neto. Self-adaptive physics-informed informed neural networks using a soft attention mechanism. arXiv preprint arXiv:2009.c 04544, 2020.
Jens Markus Melenk. On the robust exponential convergence of hp finite element methods for problems with boundary layers. IMA Journal of Numerical Analysis, 1774)5577-601 1997.
Xuhui Meng, Zhen Li, Dongkun Zhang, and George Em Karniadakis. Ppinn: Parareal physics-informed neural network for time-dependent pdes. Computer Methods in Applied Mechanics and Engincering, 370: 113250. 2020. ISSN 0045-7825. doi: http:://dbi.org//000110.... http:/aaeetttccoecccc/ceeaaaaaaa
20t..2220.11111..1 113250. URL https::/www.
KB Nakshatrala, DZ Turner, KD Hjelmstad, and Arif Masud. A stabilized mixed finite element method for darcy flow based on a multiscale decomposition of the solution. Computer Methods in Applied Mechanics and Engineering, (3//3-33))40336 4049, 2006.
A. Noorizadegan, R. Cavoretto, D. L. Young, and C. S. Chen. Stable weight updating: A key to reliable pde solutions using deep learning, 2024a. URL http::://arriv.oorg////////00000
A Noorizadogan, R Cavoretto, Der-Liang Young, and CHUIN-SHAN Chen. Stable weight updating: A key reliable pde solutions using deep learning. Engineering Analysis with Boundary Elements, 168:105933, 2024b.
Guofei Pang, Lu Lu, and George Em Karniadakis. fpinns: Fractional physics-informed neural networks, 2018.
Guofei Pang, Marta D D'Elia, Michael Parks, and George E. Karniadakis. npinns: nonlocal physics-informed neural networks for a parametrized nonlocal universal laplacian operator. algorithms and applications, 2020.
35

---

Published in Transactions on Machine Learning Research (12/2025)
Nicolas Papernot, Abhradeep Thakurta, Shuang Song, Steve Chien, and Ülfar Erlingsson. Tempered sigmoid activations for deep learning with differential privacy, 2020. URL httts:://axxvv.org/ wwtp:/or/r.bbb//////////// 14191
M. Raissi, P. Perdikaris, and G.E. Karmiadakis. Physicc-inffrmed neural networks: A deep learning frame- work for solving forward and inverse problems involving nonlincar partial differential equations. Journal of Compulational Physics, 378:686 707, 2019a. ISSN 0021-9991. doi: http:://dbi.org/10000116/. 10.045.
Maziar Raissi, Paris Perdikaris, and George E. Karniadakis. Physics-informed neural networks: A deep learn- ing framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Compulational physics, 378:686 707, 2019b.
Robert Schaback and Holger Wendland. Adaptive greedy techniques for approximate solution of large rbf systems. Numerical Algorithms, 24(3):239-22 254, 2000.
Christoph Schwab. p- and hp- DinFinete Element Methods: Theory and Applications in Solid and Fluid Mechan- ics. Oxford University Press, 1998.
Ayush Sekhari, Karthik Sridharan, and Satyen Kale. Syd: The role of implicit regularization, batch-size and multiple-epochs. Advances In Neural Information Processing Systems,
2021.
Varun Shankar. The overlapped radial basis function-finite difference (RBF-FD) method: A generalization of RBF-FD. J. Comput. Phys., 342:211 228, 2017.
Ramansh Sharma and Varun Shankar. Accelerated training of physics-informed neural networks (PINNs) using meshless discrelizations. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL htttpss/oppareeriev.nt/toooo 1d=NYDU9BRODOS
Jie Shen, Tao Tang, and Li-Lian Wang. Spectral methods: algorithms, analysis and applications, volume 41. Springer Science & Business Media, 2011.
Khemraj Shukla, Ameya D Jaglap, and George Em Karniadakis. Parallel physics-informed neural networks via domain decomposition. Journal of Computational Physics, 447:110683, 2021.
Gilbert Strang, George J Fix, and DS Griffin, An analysis of the finite-element method. Journal of Applied Mechanécs, (1()):62, 1974.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems, 33:7537-7547, 2020.
Nathaniel Trask, Mauro Perego, and Pavel Bochev. A high-order staggered meshless method for elliptic problems. STAM Journal on Scientific Computing, (9(2):A479-A5000 2017. doi: 10.1137/16M1055000.
Sifan Wang, Yujun Teng, and Paris Perdikaris. Understanding and mitigating gradient flow pathologies in physics-informed neural networks. SIAM Journal Scientific Computing, 43(5):A3055-A3081, 2021a.
Sifan Wang, Hanwen Wang, and Paris Perdikaris. On the eigenvector bias of fourier feature networks:
From regression to solving multi-scale pdes with physics-informed neural networks. Compuler Melhods in Applied Mechanics and Engineering, 384:113938, 2021b.
Sifan Wang, Shyam Sankaran, and Paris Pordikaris. Respecting causality is all you need for training physics- informed neural networks. arx preprint or Xixx22203.07404, 2022a.
Sifan Wang, Xinling Yu, and Paris Perdikaris. When and why pinns fail train: neural tangent kernel perspective. Journal of Compulational Physics, 449:110768, 2022b.
36

---

Published in Transactions on Machine Learning Research (12/2025)
Sifan Wang, Shyam Sankaran, Hanwen Wang, and Paris Perdikaris. An expert's guide to training physics- informed neural networks. arXiv preprint wrXi:2308.08468, 2023a.
Sifan Wang, Bowen Li, Yuhan Chen, and Paris Perdikaris. Piratenets: Physics-informed deep learning with residual adaptive networks. arXiv preprint or Xx2::2402.00326, 2024.
Zhiwen Wang, Minxin Chen, and Jingrun Chen. Solving multiscale elliptic problems by sparse radial basis function neural networks. Journal of Computational Physics, 492:112452, 2023b.
Bernard Widrow and Michael A Lehr. 30 years of adaptive neural networks: perceptron, madaline, and backpropagation. Proceedings of the IEEE, 78(9):1415 1442, 2002.
Colby Wight and Jia Zhao. Solving allen-cahn and cahn-hilliard equations using the adaptive physics informed neural networks. arXiv preprint arXiv:2007. 04542, 2020.
Yue Wu, Hui Wang, Biaobiao Zhang, and K-L Du. Using radial basis function networks for function approximation and classification. International Scholorly Research Nolices, 2012(1):324194, 2012.
Fabian Wurzberger and Friedhelm Schwenker. Learning in deep radial basis function networks. Entropy, 26 (5):368, 2024.
X. I. A. Yang, S. Zafar, J.-X. Wang, and H. Xiao. Predictive large-eddy-simmlattin wall modeling via physics- informed neural networks. Phys. Rev. Fluids, 4:034602, Mar 2019. doi: 10. F.00:/PyysReer/lldd....3. URL https::/linkk aps. org/ htt///oo/11..0 htt./10..110////yyee/luuuud. 0.46622
Jeremy Yu, Lu Lu, Xuhui Meng, and George Em Karniadakis. Gradient-enhance physics-informed neural networks for forward and inverse pde problems. Computer Methods in Applied Mechanics and Engineering. 393:114823, 2022.
Norman Zabusky and Martin Kruskal. Interaction of "solitons' collisionless plasma and the recurrence of initial states. Physical review letters, 15(6):240, 1965.
Shaojie Zeng, Zong Zhang, and Qingsong Zou. Adaptive deep neural networks methods for high-dimensional partial differential equations. Journal of Computational Physics, 463:111253 2022.
Dongkun Zhang, Lu Lu, Ling Guo, and George Em Karniadakis. Quantifying total uncertainty in physics- informed neural networks for solving forward and inverse stochastic problems. Journal of Computational Physics, 397:108850, 2019. ISSN 0021-9991. doi: http://///.oooooo0000000///....p
URL https:
37

---

Published in Transactions on Machine Learning Research (12/2025)
A Allen-Cahn with Dirichlet Boundary Conditions
A.1 Reference Solution Generation
We solve the equation numerically using a semi-implicit finite difference method. The spatial domain is discretized uniformly with 1042 grid points, and the temporal domain discretized into 600 steps. The nonlinear reaction term is treated explicitly, while the linear diffusion term is handled implicitly, forming IMEX (implicit-explicit) Euler scheme. Homogeneous Dirichlet boundary conditions imposed at each time step by modifying the linear system accordingly. Solutions are recorded by saving the solution u(x, t) at all time steps along with the initial condition and -ential- grids.
A.2 Ablation Study: Impact of RBF Kernels and Center Density
To evaluate the impact of different radial basis function (RBF) kernels on solution quality, conduct ablation study comparing several anisotropic RBF formulations within the hybrid PINN framework. The goal is to determine how the kernel's support, smoothness, and decay behavior affect the model's ability to capture localized features, represent sharp interfaces, and maintain global solution coherence. The architecture, initialization, and training procedure are held constant across trials to isolate the influence of the kernel's functional form. Table summarizes the five kernels studied.
Wendland C2
Inverse MQ
Gaussian
10a
Nb-1
Nb-1
Mean Relative L? Error
Nb-3
10
Mutiquadratic
Matern-5/2
10a
Mean Relative L? Error
Nb-2
Nb-3
10-1
1
Nb-2
Nb.3
Iterations
Iterations
Figure 23: Allen-Cahn equation: Prediction errors for various RBF kernels within the HyResPINN framework using 1, 2, and 3 blocks.
Results demonstrate that kernel choice substantially influences the model's capacity to resolve steep gradients and preserve global coherence. The best overall performance is achieved by the Wendland C kernel, which we adopt for all subsequent experiments, as it offers a slightly better trade-off between accuracy, computations efficiency, compact support, and C2 smoothness which make it well-suited for the localized dynamics. addition to kernel type, we also investigate how the number of RBF centers affects predictive performance, using the Wendland C kernel. Results are shown in Table 11 which show that balance between resolution and overfitting emerges, with best performance at 64 centers.
38

---

Published in Transactions on Machine Learning Research (12/2025)
Name
Computational Cost
Gaussian (Nb-1)
0.020064 (0.00067)
Gaussian (Nb-2)
0.040506 (0.00044)
Gaussian (Nb-3)
0.062309 (0.00045)
Inverse MQ(Nb-1)
0.020070 (0.00057)
Inverse MQ(Nb-2)
0.040792 (0.00024)
Inverse MQ(Nb-3)
0.062468 (0.00075)
Mat6m--5/(NNb-))
0.020033 (0.00076)
Matémn-5/2(Nb-2)
0.040725 (0.00087)
Matérn-5/2(ND--)
0.064149 (0.00043)
MultiquadriicNN--I)
0.019847 (0.00080)
Multiquadricc(ND22)
0.040620 (0.00077)
Mulliquadric(NN-33
0.062131 (0.00044)
Wendland (NNb-))
0.021837 (0.00071)
Wendland C (Nb-2)
0.043131 (0.00043)
Wendland C (Nb-3) 0.065628 (0.00097)
Table 10: Computational cost measured as the average time in seconds to run 100 iterations for each RBF using 1,2, and blocks. Standard deviations are in parenthesis.
Number of Centers
Error (Std)
Computational Cost
8
0.028290 (0.042173)
0.017529
(2.634898)
16
0.007372 (0.002045)
0.017578
(3.782532)
32
0.003666 (0.001104)
0.017849
(6.860452)
64
0.002462 (0.000063)
0.018479
(2.414555)
128
0.003265 (0.001240)
0.019347
(7.184156)
256
0.002938 (0.000411)
0.021444 (6.183969)
Table 11: Effect of increasing the number of RBF centers on HyResPINN mean relative l2 error and com-
putational cost measured the average wall-clock time (in seconds) required perform 100 iterations. Standard deviations in parentheses.
39

---

Published in Transactions on Machine Learning Research (12/2025)
A.3 Ablation Study: Impact of Gating Parameter Initializations
To better understand the role of the learnable gating parameters in modulating the hybrid outputs and the residual skip connections, conducted an ablation study varying the initial values of both gales (Ainit and 9mi.). We tested the benchmark due to its mixed smooth and sharp features. All other hyperparamcters were kept fixed across runs, including the number of blocks (N, 3), hidden dimension (dn 64), and batch size (1024). Unless otherwise noted, we the regularization strengths A >3 10 and training was performed for 200,000 epochs over five random seeds.
equation with Dirichlet boundary conditions from Section 5.1 as the
Joint a-ß Initialization Sweep
0.0060
0.0
0.00239
0.0024
0.00239
0.0055
Mean Relative f2 Error
0.0050
(Binn)
0.0045
0.5
0.00234
0.00236
0.00286
0.0040
0.0035
1.0
0.00242
0.00328
0.00614
0.0030
0.0025
0.0
0.5
1.0
((ait))
Figure 24: AllenCCahn equation: Mean relative & error for different combinations of ((ama)) and @(pma))
Figure 24 shows the joint effect of the gating initializations by reporting the mean relative l errors all (Ø( Osmit) ) o(Buit)) pairs in {0, 0.5, 1}. We observe clcar coupling between the two gating mechanisms: performance deteriorates when both gales are initialized near unily, suggesting excessive weighting of the DNN and transform pathways hinder learning. In contrast, intermediate or asymmetrie initialization such such as ((amm)) R 0.5 and ((Buit)) z 0- l--yicld the lowest test errors, suggesting that balanced initialization between block components (RBFNN and DNN, controlled by ((a) combined with an initially identity-favoring ((3), enables the network to form a more effective mixture between the block components and progressively learn deeper transforms through x(3) during training.
To further isolate the individual effects of each gating mechanism, visualize, in Figures 25 and 26 the final learned values of σ(a)) and x(ß) across block depth (0-2) for varying initializations, sigmoid temperatures (t), and regularization strengths (Ax. As). The left-most subplots correspond to unconstrained cases where no sigmoid constraint (o(-)) applied meaning that the gating parameters are not constrained to lie in [0, 1), while the remaining columns apply progressively higher temperatures controlling the sharpness of the sigmoid function (soe Scction 4.1 for further details). the top rows, line colors represent the initial parameter values; in the bottom rows, the same experiments are color-mapped by the resulting mean relative l error.
Higher initialization or lower temperature leads to sharper, DNN-dominant gating (á(a) - 1), emphasizing DNN components, while lower initialization or stronger regularization biases the network (oward RBFNN- dominant gating (ø(a) + 0). Variations d(a) across depth demonstrate the adaptive blending of local and global representations. In contrast, the 3-gates primarily modulate residual flow: smaller x(3) values promote identity mapping (residual-dominant) and larger values encourage deeper, transform-dominant be- havior. Together, these results confirm that the gating parameters not only balance the RBFNN and DNN
40

---

Published in Transactions on Machine Learning Research (12/2025)
Block-wise Geting Profiles (Effect of Temperature, Initialization, and.A.)
gate
gate = (la t= 0.5)
gate ((a, 1.01
gate ((a, 2.0)
1.00
6lOve
0.75
0.0
0.50
Aa
0.25
0.00
2
TERY 0
Block Depth
Block Dopth
Block Dapth
Block Dopth
Block-wise Gating Profiles (Effect of 9. Temperature, Initialization,, and As)
gate
gate ((, 0.5)
gate a t= 1.0)
gate d(a, 2.0)
Mean relative f; Error (log scele)
1.00
Mean gate
0.50
0.0 1.0e044
0.25
Tring
0.00
Block Depth
Block Depth
Block Depth
Blook Depth
Figure 25: Allen-Cale Cahn equation: Depli-wise a gating parameters for varying initialization, temperature, and regu- larization. o-gates balance RBFNN and DNN representations, where larger gate values bias the output towards the DNN and smaller values to the RBENN. The top figure shows the
Block-wise ((8) Gating Profiles (Effect a Temperature, Initialization, and Aş)
gate
gate (B, t 0.5) gate @B, 1.0)
gate $(B, 2.0)
1.5
(18me)
Mean gate
1.0
0.0
0.5
1.0
An
Rodbas
0.0
1.0e-04
0
2
Block Depth
Block Depth
Block Depth
Block Depth
Block-wise ((B) Gating Profiles (Effect é. Temperature, Initialization, and Ap)
gate =
gate (B, t 0.5)
gate $ t 1.0)
gate (B8, t 2.0)
Mean relative & Error (log scale)
1.5
Transom
Trandom
Mean gate
1.0
x
Ag
0.0
0.5
1.0e-04
10
0.0
2
2
Block Depth
Block Depth
Block Depth
Block Depth
Figure 26: Allen- Cahn equation: Depth-wise 8 gating parameters for varying initialization, temperature, and reg- ularization. The 8-gates modulate residual information flow across depth. Smaller gate values bias toward identity mapping (residualldominantt), and larger gate values increase transform-dominnnt behavior.
41

---

Published in Transactions on Machine Learning Research (12/2025)
contributions, but also dynamically regulate the effective network depth, enabling HyResPINNS self-adjust their representational complexity according to the PDE structure and training regime.
B Allen-Cahn with Periodic Boundary Conditions
B.1 Reference Solution Generation
Following the approach in (Wang et al, 2024]. we employ standard spectral methods to solve the Allen Cahn equation under periodic boundary conditions. Starting from the initial state uo(x) x² cos (TT), inte- grate the equation up to final time of T = 1. For generating synthetic validation data, Chebfun (Driscoll ct al., 2014) package to perform Fourier spectral discretization with 512 modes. Time integration is carried out. using the fourth-order exponent time-differencing Runge- Kutta method (ET- DRK4) [Cox & Matthews 2002) with a time step of 10-5, Solutions are sampled every At = 0.005, resulting in a high-resolution validation dataset of size 200 x 512.
the
C Darcy Flow with Smooth Coefficients
C.1 Learned Gating Parameter Evolution
0.52
0.012
0.011
0.51
0.010
0.50
0.009
0.49
0.008
0.48
0.007
25000
50000
75000
100000
25000
50000
75000
100000
Iterations
Iterations
(Ba)
((11)
((Bz)
((ag)
((az)
((az)
Figure 27: 2D Smooth Darcy-Flow equalion (Neumann BCa): Evolution of the learned gating parameters
during training. (Left) Evolution of residual gating parameters o(3),, which modulate the strength of residual information flow across blocks. (Right) Evolution of hybrid gating parameters Φ(a;), which balance the DNN and RBF components within each block.
D Kuramoto-Sivashins Equation
D.1 Reference Solution Generation
We employ standard spectral methods to solve the Kuramoto-Sivashinse equation under periodic boundary conditions. For generating synthetic validation data, we use the Chebfun (Driscoll el al. 2014) package to perform a Fourier spectral discretization with 512 modes. Time integration carried out using the fourth- order exponential time-differencing Runge-Kui Kutta method (ETDRK4) (Cox & Matthews 2002) with a time
step of 10-5, Solutions are sampled every At 0.004, resulting in high-resolution validation dataset of size 250 512.
42

---

Published in Transactions on Machine Learning Research (12/2025)
0.500
0.011
0.495
((
0.010
@
0.490
0.009
0.480
0.007
0.475
25000
50000
75000
100000
25000
50000
75000
100000
Iterations
Iterations
((Bo) ((L))
(B2)
Maa)
((a1)
(((z))
Figure 28: 3D Smooth Darcy-Flow equation (Dirichlet BCs): Evolution the learned gating parameters
during training. (Left) Evolution of residual gating parameters o(3),, which modulate the strength of residual information flow across blocks. (Right) Evolution of hybrid gating parameters Φ(a;), which balance the DNN and RBF components within each block.
0.51
0.011
(())
(())
0.49
0.009
0.008
0.48
0.007
0.47
25000
50000
75000
100000
25000
50000
75000
100000
Iterations
Iterations
(Ba))
((A))
((B=)
((a.)
((a;)
((az)
Figure 29: 3D Smooth Darcy-Flow equation (Neumann BCs): Evolution of the learned gating parameters
during training. (Left) Evolution of residual gating parameters o(3,), which modulate the strength of residual information flow across blocks. (Right) Evolution of hybrid gating parameters Φ(a;), which balance the DNN and RBF components within each block.
43

---

Published in Transactions on Machine Learning Research (12/2025)
E Korteweg- De Vries Equation
E.1 Reference Solution Generation
Similar to [aag et al., 2024 we generate synthetic validation data for the Korteweg-DD De Vries equa- tion, we use standard spectral methods under periodic boundary conditions. The simulation begins with the initial condition set to uo( cos(TT2 and is integrated forward time up to T = 1. The Cheb- fun package (Driscoll et al. 2014) is utilized for Fourier spectral discrctization employing 512 modes. Time integration performed with the lourth-order exponential Lime-differencing Runge Kutta (ETDRK4) method (Cox & Matthews/ 2002;; using step size of 10-5. The solution is sampled every At 0.005, resulting in validation dataset of size 200 x 512 covering the temporal and spatial domains.
F
Grey-Scott Equation
F.1 Reference Solution Generation
For data generation, solve the Grey-Scott equation using standard spectral techniques under periodic boundary conditions following the details provided in (Wang et al. 2024 The system is initialized with the prescribed initial condition and integrated up to a final time of T 2. Synthetic validation data are produced with the Chebfun package [Driscoll et al. 2014), employing a 2D Fourier spectral discretization with 200 200 modes. Temporal integration performed using the fourth-order exponential time-differencing Runge Kutta (ETDRK4) method (Cox & Matthews 2002) with a time step of 10-3, Solutions are saved every At 0.02, resulting in validation dataset of dimensions 100 x 200 200 spanning both the temporal and spatial domains.
44


Reviews:
[{"review_id": "MdHyzQuHpS", "reviewer": "Reviewer_vFVz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper aims to enhance the function space richness, adaptivity and efficiency per degree of freedom of physics-informed neural networks (PINNs) for solving PDEs. Specifically, it introduces a hybrid residual PINN architecture that combines deep neural networks with radial basis function neural networks, using a learnable gating parameter to adaptively balance the contributions of these two modules. Besides, a second learnable gating parameter is used to modulate the residual connection, enabling dynamic adaptation of network depth during training.\n\nThe proposed architecture is evaluated on a variety of PDEs with different input dimensions and boundary conditions, including cases with sharp solution features. It demonstrates higher accuracy than baseline methods, indicating the effectiveness of the hybrid design.", "claims_and_evidence_yn": "No", "claims_explanation": "- The definition of efficiency per degree of freedom (DoF) is unclear, and no quantitative metric is provided to evaluate this property. In the paper, efficiency per DoF is illustrated through plots of mean relative L2 error with respect to training iterations or training set size. While these results indicate that the proposed HyResPINN architecture converges faster to a lower error compared to other baseline methods, this does not constitute a quantitative measure of efficiency per DoF. Moreover, as shown in Tables 3 and 4, HyResPINN does not achieve the lowest computational cost. Therefore, the claim of 'superior efficiency per DoF' is not fully supported. Instead, the results suggest that HyResPINN provides a reasonable trade-off between computational cost and accuracy.\n\n- In Section 5.3, “Efficiency per Degree of Freedom”, the authors claim that 'HyResPINN consistently achieves lower solution and flux errors than competing models, especially in the low-data regime.' However, in the first row of Figure 10, HyResPINN does not achieve lower solution and flux errors than the competing models; in fact, ResPINN attains a lower solution error than HyResPINN at the lowest training set size for Dirichlet boundaries. I suggest the authors revise this statement to ensure greater accuracy and rigor in their scientific expressions.", "audience_yn": "Yes", "audience_explanation": "This work addresses the challenge of solving PDEs with deep neural networks and aims to improve the accuracy and expressiveness of PINN-based approaches, which should be a topic of broad interest.", "requested_changes": "- It would be good if the authors could further analyze the values of the two learnable gating parameters (one controlling the weighting between the DNN and RBFNN, and the other controlling the weighting of the residual module) for different PDEs, to investigate if they have some physical interpretations or insights.\n\n- In table 2, for DarcyFlow with 3D Dirichlet BCs boundary conditions, ResPINN achieves the same L2 test error as HyResPINN and should also be highlighted in bold. Similarly, ExpertPINN should be highlighted for DarcyFlow 3D with Neumann boundary conditions.", "broader_impact_concerns": "There are no broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "g5aeUtozh6", "reviewer": "Reviewer_yiAS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces hybrid residual physics-informed neural networks, HyResPINNs, a new architecture to solve PDEs in a mesh-free data driven manner. The authors motivate this new architecture by decomposing expressivity into three parameters: function space richness, adaptivity, and efficiency per DoF, over which they situate their method and baselines to analyze contributions and limitations. By combining the efficiency and adaptivity of classical methods through the use of local RBF neural networks and skip connections with the richness of neural networks through deep neural networks within each block and at the end of the architecture, HyResPINNs aim at addressing the three axes at once. The authors motivate each architectural and training choice with a wide range of literature references, and demonstrate the performance of various PDE experiments.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper is well written and is pleasant to read. There are very few typos, it is well-structured and well-divided. I enjoyed the explanation of the three axes of comparison for efficiency, and how the overall discussion follows them. \n\nThe method itself is well-supported by the literature. Each design choice is properly motivated with references. They are logically connected to each goal of the paper and the evidence provided by the experiments, notably the ablation studies, further supports the overall architecture.\n\nThe experiments are diverse, encompass 1D to 3D for the spatial domain. I think the choice of the baselines is appropriate, but could be slightly enhanced to further support the claims of the paper against classical and general ML methods.\n\n### Questions\n- When describing your method, you support the gating mechanism for its ability to selectively adapt the required depth of the network depending on the problem, in contrary to other overparameterized neural networks. In practice, your formulation of the model uses a gated external skip connection between each subsequent blocks. Did you observe, in your experiments, models which properly used only a subset of the block, e.g.,  some phi(betas) being null for the last m blocks? Did you experiment with skip connections without gating? I find the identity initialization interesting, maybe I missed some details but were some other initializations experimented with? It would be interesting to compare between gated and ungated skip connections as well as different initializations to support their impact beyond gradient stability. \n- Although I understand the focus on PINNs, the claims of the paper address more general comparisons with classical methods and especially different ML classes of architectures. Have you had the opportunity to train other classes of models, such as FNOs, GNOT, GeoFNO, or GINO? It would be interesting, even just for indicative purposes. I understand this can be too much work, so I am not really asking for new experiments but rather for possibly already conducted ones, expectations, or reasons not to have done them. \n- Although some of the PINN baselines are motivated because they were described in the related work section, I think a deeper baselines paragraph which further explains the reasons for each of them, especially regarding the three axes of expressivity being tackled, could help reading and interpreting Section 5.\n- Table 4 & Fig 8: HyResPINN has similar mean relative L2 error over training compared to Expert & Pirate, yet is respectively 37 and 19% slower. I am not sure these times should be considered as really comparable in the text, in contrary to the errors in this case.", "audience_yn": "Yes", "audience_explanation": "This method will definitely be of interest to the community. PINNs are an active field of research with wide applications, and these results against recent state-of-the-art methods will definitely contribute to the field. Future directions, including experiments on higher-dimensional problems, deeper comparisons over different classes of methods, definitely deserve attention.", "requested_changes": "First, please address my questions in the first section. The next points are minor writing-related remarks.\n\n- I'm not really considering it strictly required, but why do only some subsections in Section 5 have a \"summary\"? This is only a question regarding writing style, I was expecting sort of the same structure for each experiment. This is minor.\n\n### Typos\n\nThese are only some minor typos I found when reading.\n\n- The citation of \"The p and h-p Versions of the Finite Element Method, Basic Principles and Properties\" has a bold v in Babuška, which appears in the text. \n- Penultimate paragraph of page 4 (starting with \"For instance\"), end of line 3: double \"potentially\".\n- Section 3.2, end of first line, missing \"in\" between \"resulting\" and \"training\".\n- Typo in Jax version? 0.7.2 is the latest as far as I know and you wrote 1.11.", "broader_impact_concerns": "None.", "additional_comments": "### Recommendation\n\nOnce my remarks have been addressed, I will be leaning toward accepting this paper, considering its sound results and its potential for the PINN community.", "extra_scores": {}, "extra_text": {}}, {"review_id": "ov8N84ZsSz", "reviewer": "Reviewer_CAVd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces HyResPINNs, an architecture that combines PINNs with radial basis functions (RBFs) to address the tendency of PINNs to produce oversmoothed solutions. The method demonstrates promising performance across diverse benchmark problems, with rigorous comparisons conducted under a fixed parameter budget. An ablation study further shows that the RBF and PINN components complement each other positively.\n\nA key strength of the work is the thorough evaluation, including benchmarks and ablations, which provide evidence for the benefits of the proposed hybrid design. A weakness is the increased architectural complexity: the additional RBF network introduces more hyperparameters and may complicate training, though the reported training curves appear robust. A further limitation is that the approach may be less beneficial for PDEs with inherently smooth solutions.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The claims in the submission are generally well supported by accurate and convincing evidence. The authors present results on seven subproblems, showing that HyResPINNs outperform meaningful benchmarks and providing substantial experimental evidence for the benefits of the architecture. The experiments are run over multiple random seeds, which strengthens the statistical significance of the findings. \n\nWhile it is possible that there are some cases where the architecture does not outperform, particularly for problems with inherently smooth solution functions, the presented evidence is strong and demonstrates clear improvements for problems with sharp gradients in the solution function.", "audience_yn": "Yes", "audience_explanation": "Yes, the findings of this paper would be of interest to the TMLR audience. The work represents a meaningful contribution at the intersection of machine learning and computational science, particularly for PDE problems with sharp gradients where standard PINNs often struggle. More generally, the proposed approach provides a helpful inductive bias in the parametrization of neural networks for addressing such problems.", "requested_changes": "In the Related Works section, the differences relative to the most similar methods are left largely implicit. The paper would benefit from a concrete contrasts that states how HyResPINN differs from each closest prior work (e.g., brief one-sentence “X does …; HyResPINN differs by …” statements) either at the end of Related Work or in the “Positioning” paragraph in the Introduction. This will also place the chosen baselines in clearer context and help readers interpret the experimental comparisons and the scope of the claimed contributions.\n\nTo strengthen the experimental support, it would be helpful to report standard deviations across seeds in Table 2 and to mark the second-best results (e.g., with underlining). \n\nIt would also be valuable to visualize how the gating mechanism behaves at different network depths, as this could inspire architectural variants that fix these values from the start and would further demonstrate that both components are actively used. Hereby, I would encourage the authors to create a plot with the network depth as the x-axis and the value of the learned gating weight on the y-axis, over multiple seeds, to see where the network uses which part of the Hybrid Residual Block.\n\nFinally, adding a clear limitations section would improve the paper: for example, discussing when this architecture might struggle or cases where a regular PINN might be expected to outperform.", "broader_impact_concerns": "The paper does not include a broader impact statement. I do not see any pressing ethical or societal concerns arising from this work. The contribution is primarily methodological within the domain of scientific machine learning, and I see little potential for negative impact.", "additional_comments": "There is a typo in Figure 3(a): It should be \"left\" instead of \"lef\".", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)