Holistic Physics Solver: Learning PDEs in a Unified Spectral-Physical Space
Authors: Xihang Yue
Yi Yang
Linchao Zhu
Venue: ICML 2025 poster (2025)
Keywords: PDE
Neural Operator
Attention Mechanism
Primary area: applications->chemistry_physics_and_earth_sciences
TL;DR: We introduce a novel neural operator that combines attention's flexibility with spectral methods' continuity constraints through learning PDEs in a unified spectral-physical space, achieving superior performance in PDE solving.

Abstract: Recent advances in operator learning have produced two distinct approaches for solving partial differential equations (PDEs): attention-based methods offering point-level adaptability but lacking spectral constraints, and spectral-based methods providing domain-level continuity priors but limited in local flexibility. This dichotomy has hindered the development of PDE solvers with both strong flexibility and generalization capability. This work introduces Holistic Physics Mixer (HPM), a novel framework that bridges this gap by integrating spectral and physical information in a unified space. HPM unifies both approaches as special cases while enabling more powerful spectral-physical interactions beyond either method alone. This enables HPM to inherit both the strong generalization of spectral methods and the flexibility of attention mechanisms while avoiding their respective limitations. Through extensive experiments across diverse PDE problems, we demonstrate that HPM consistently outperforms state-of-the-art methods in both accuracy and computational efficiency, while maintaining strong generalization capabilities with limited training data and excellent zero-shot performance on unseen resolutions.

Holistic Physics Solver: Learning PDEs in a Unified Spectral-Physical Space
Xihang Yue 12 Yi Yang" 12 Linchao Zhu 12
Abstract
Different Neural Modules for I Learning Operator Mappings
Spectral Mixer eg. PNO, FAANO, U-O
Attention Mixer
Holistic Physies Mixer
Recent advances in operator learning have pro- duced two distinct approaches for solving par- tial differential equations (PDEs): attention- based methods offering point-level adaptability but lack- ing spectral constraints, and spectral-base based meth- ods providing domain-level continuity priors but limited local flexibility. This dichotomy has hindered the development of PDE solvers with both strong flexibility and generalization capa- bility. This work introduces Holistic Physies Mixer (HPM), simple framework that bridges this gap by integrating spectral and physical in- formation in a unified space. HPM unifies both approaches as special cases while enabling more powerful spectral-physical interactions beyond ei- ther method alone. This enables HPM to inherit both the strong generalization of spectral methods and the flexibility of attention mechanisms while avoiding their respective limitations. Through extensive experiments across diverse PDE prob- lems, we demonstrate that HPM consistently out- performs state-of-the-an methods in both accu- racy and computational efficiency. while maintain- ing strong generalization capabilities with limited training data and excellent zero-shot performance on unseen resolutions.
eg. Gnteenaa-din GNDT..
The intodaced new enterr
Input Function
Input Function
Imput Funcrion
Spectral Space
Sputial Space
Holistie Spectrul Space
attention among spatial points
spectral
Holistic spectral hasts
Ouiput Function
Ouiput Fanction
Output Function
Domain-lovel
Domain Prior
Domain level
Point-level Flexibiliyy
Paint-level Flexibility
Point-level Flecibility
Figure 1. HPM combines domain-level spectral structure and point- wise physical states in unified holistic spectral space, enabling more powerful feature learning beyond either approach alone.
emerged as a data-driven approach that learns continuous mappings between function spaces for solving parametric PDEs.
Recent works have developed two main categories of neural operators. Spectral-based methods (Li et al., 2020; Ko- vachki et al., 2023; Tran et al., 2021) can efficiently learn operator mappings with limited training data by approximat- ing physical functions in truncated spectral spaces. However, their fixed spectral processing mechanism lacks point-level flexibility (ability to process individual spatial points differ- ently based on local features), making them less effective capturing
1. Introduction
details and fine-scale variations in physical systems (George et al., 2024; Qin et al., 2024). In contrast, iitentiion-asse methods (Hao et al., 2023; Xiao et al., 2023; Wu et al., 2024) can flexibly learn various physical dynamics through point-wise learning in physical domains, achieving superior performance with sufficient training data. However, their pure data-driven framework without spectral priors tends to overfit with scarce training data and has limited generalization capability (Xiao et al., 2023), leading significant performance degradation.
Solving partial differential equations (PDEs) efficiently re- mains a major challenge in scientific computing and en- gineering applications. Traditional numerical solvers rely on high-precision meshes and substantial computational re- sources, making them impractical for many real-world sce- narios. To address this challenge, neural operators (Lu et al., 2019; Li et al., 2020; Tripura & Chakraborty, 2022) have
'The State Key Lab of Brain-Machine Intelligence, Zhejiang University, Hangzhou, China "College of Computer Science and Technology, Zhejiang University, Hangzhou, China. Correspon- dence to: Linchao Zhu <zhulinchao www.aaaoozzuu.u.c.... zju.edu.en>.
This work introduces Holistic Physics Mixer (HPM) ', a framework that unifies spectral and physical information in a single holistic space. The key contribution lies in our holistic spectral space design that simultaneously encodes
Proceedings of the 4 International Conference on Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).
'Code: https: tgtbttttthuggggggg www./uwwi/iicccooooooo//e/.....a hang/HPM

---

Learning PDEs in Unified Epectral-Physien Space
both domain-level spectral structure and point-wise physi- cal states. This strategy naturally balances global continu- ity constraints with local adaptivity. Through learnable coupling mechanism, HPM allows each spatial location to adaptively modulate different spectral components while maintaining the benefits of spectral learning. This unified treatment enables HPM to inherit both the strong general- ization of spectral methods and the flexibility of attention mechanisms while avoiding their respective limitations.
The operator learning problem aims to learn a parameterized surrogate model 9/ for approximating the operator mapping Gl. As established in previous works (Li et al., 2020; Wu et al., 2024), the neural operator is commonly implemented as a stack of network layers:
G = V o o..... o M2 o M1 o P,
(3)
where P and V are element-wise projecting layers that map between the input/output functions and latent functions. M; takes v,- E Rxx as input and produces v, E R xd, as output, mixing features along both token and channel dimensions. In previous works (Tran et al., 2021; Wu et al., 2024), M, is formulated as:
Through extensive experiments across diverse PDE prob- lems, we demonstrate that HPM: (a) consistently outper- forms both spectral-based and attention- methods, (b) maintains strong generalization with limited training data and excellent resolution generalization capability like spec- tral methods, (c) efficiently utilizes increased data like at- tention methods. (d) Our visualization analysis shows that HPM learns meaningful spectral modulation patterns that could guide the design of fixed spectral neural operators.
= J_mieer (LayerNorm(()-))) + 1;-1, (4)
0: = FreedForward(LayyNooon mid + mid
(5)
where F represents operations like spectral processing or self-attention for mixing spatial information. Various neural mixers (Li et al., 2020; Wu et al., 2024; Raonic et al., 2024) have been explored for neural operator learning.
Our core contributions are summarized as follows:
We present Holistic Physies Mixer (HPM), a unified framework that processes features in holistic spectral space, integrating both domain-level structure and point- wise physical states.
2.2. Spectral-based Neural Operators
Unlike the classical neural modules such as CNN (LeCun et al., 1995), RNN (Chung et al., 2014), and Self-Attention mechanism (Vaswani, 2017) that directly mix features of spatial tokens the spatial domain, a significant line of works (Li et al., 2020; Lee- Lee-Thorp al., 2021: Guibas et al., 2021; Yue al., 2024; Liu et al., 2024b; Gao et al., 2025) ex- plores learning operator mappings in spectrum space, which reduces learning difficulty through efficient function approx- imation with spectral basis functions. The spectral-base token mixer can be formulated as:
We present an effective implementation of HPM that (a) maintains strong performance with limited training data and excellent resolution generalization through spectral priors, and (b) efficiently handles increased training data and fine-scale status variations in physical space via inte- grating point-wise adaptivity.
We validate HPM's superiority in various scenarios through comprehensive experiments, and find the spectral processing patterns learned by HPM can help design fixed spectral neural operators.
F (x) = - o Project o Touuiie(x))
(6)
2. Background and Related Work
where (.)urier()) represents the spectral transform opera- tor such as Fourier Transform, yielding spectral feature a E Rexd, & represents the number of retained frequen- cies in the spectral domain. Toorier (-) denotes the inverse spectral transform operator. Project represents operations in the spectral domain, commonly including simple fully connected layers and normalization operations.
2.1. Neural Operator Learning
In neural operator learning (Lu et al., 2021; Li et al., 2020; Kovachki et al., 2023), the solution of parametric partial differential equations is formulated as the operator mapping between two
function spaces:
gt : A + U,
(L) Different spectral transforms have been explored: FNO (Li et al., 2020) learns operators in Fourier spectral space, LNO (Cao et al., 2024) learns in Laplacian spectral space, and WMT (Gupta et al., 2021) learns in wavelet spec- tral space. Additional works investigate complex physical domain processing (Li et al., 2023a; Bonev et al., 2023; Lingsch et al., 2023; Liu et al., 2024a;c; 2023a). computa- tional efficiency enhancement (Poli et al., 2022; Tran et al., 2021; Wang & Wang. 2024), and multi-scale feature pro- cessing (Rahman et al., 2022; Zhang et al., 2024).
A= = {ala: : Ω RR },u1 = {ulu: : Ω + )
(2)
where 9t denotes the physical domain, da and dy represent the channel number of input functions and output functions respectively. For example, in the steady-state problem Darcy Flow, a denotes the diffusion coefficient and u represents the solution function. In the time-series problem Navier- Stokes, a is the vorticity states in previous time steps and u the vorticity states of following time steps.
2

---

Learning PDEs in Unified Spectral-Physice Space
However, spectral-based methods employ static spectral eigenfunctions, which restricts their point-level adaptability and makes them struggle to process high-frequency details and fcne-scale status variations in physical systems. As training data increases, the fixed frequency design limits significant improvement in prediction accuracy.
forms (e.g., FNO) that fail capture the complex interplay between local physical states and global domain structure.
Holistic Spectral Space. The key insight is that effec- tive feature processing should simultaneously consider both point-wise physical states and domain-level spectral prop- erties. We achieve this through a new coupling function H : RNxd VV - RNN (where & is the number of spectral basis functions) that integrates physical states x with spectral basis functions + E N (e.g., Fourier basis) to generate adaptive spectral eigenfunctions. This coupling enables unified representation space where fea- tures dynamically respond to local physical variations while preserving global spectral properties.
2.3. Attention-based Neural Operators
Recent works extensively explore learning operator map- pings based on attention mechanisms (Vaswani, 2017), en- abling flexible handling of diverse physical domains. Given the input feature æ E RNx'",, the vanilla attention mecha- nism could be simply formulated as follows:
Holistic Physics Mixer. Based on holistic spectral space, we propose the Holistic Physics Mixer Fuuiser that adaptively processes features through spectral transformation:
attn = (Qrm(Q(x)) (2))
(7)
where Q(-). K(-) and V(-) are all fully connected layers that map the feature æ into x E xd x E RNxxd
Holi
(a))
-Turt Project TTpr((r),
(8)
and xx E xx,"" respectively. Norm(.) represents the normalization operation such as Softmax.
THPT(x)) = H(x,, 4)"a.
(9)
( = H(x, ))
(10)
To address the quadratic complexity of attention, previ- ous works (Li et al., 2023b; Cao, 2021; Hao et al., 2023; Calvello et al., 2024) employ efficient attention variants. Factformer (Li et al., 2023c) enhances efficiency with multi- dimensional factorized attention. Additionally, HT-Net (Liu et al., 2023b) introduces hierarchical transformers for bet- ter multi-scale features, and ONO (Xiao et al., 2023) im- proves generalization with orthogonal attention. And some works (McCabe et al., 2023; Ye et al., 2024; Hao et al., 2024; Chen et al., 2024; Li et al., 2024; Alkin et al., 2024; Shen et al.) have explored large scale pretraining for neural operators with attention-bant based methods.
Here, THPT and TIPT denote our proposed Holistic Physies Transform and its inverse operation. The transform maps input features to transformed representation è E Rixd in the holistic spectral space. This dual encoding in a capturing both intrinsic domain structure and point-specific variations establishes a new paradigm for operator learning that naturally balances global continuity constraints with local adaptivity. Additionally, Tang et al. (2024) shows that integrating spectral and spatial features potentially improves the model's 's robustness.
Degraded Cases. Both spectral and attention based meth- ods can be viewed as simplified cases of Holistic Physics Mixer Fmixer by degraded instantiations of H:
While attention based methods achieve impressive perfor- mance on various PDEs (Wu et al., 2024), their lack of spectral constraints results in subpar performance under limited training data and unseen resolution samples com- pared to spectral-based methods. This motivates our work to combine the advantages of both approaches.
Fixed Spectral Neural Operators. When H(x; b) Φ. Jumixer reduces to fixed spectral neural operators like FNOs (Li et al., 2020; Kovachki et al., 2023). This classi- cal formulation enforces strong domain-level structure but capability for captuing fine-scale variations in physical systems and utilizing enough training samples.
3. Methodology
We introduce Holistic Physics Mixer, a unified form of spectral-based and attention- neural operators, for in- tegrating complementary advantages of both approaches.
Linear Attention Mechanism. When H(x, 4) (MLP(())) where t/ is a normalization function, Finixer becomes equivalent to linear attention mecha- Holi nism (Katharopoulos et al., 2020; Cao, 2021). While this enables flexible point-wise processing, fails to leverage domain-level continuity constraints, potentially compro- mising stability and generalization.
=
3.1. Holistic Physies Mixer: A Unified Formulation
Physical systems often involve complex field quantities represented as features x E xxd defined on a discrete sampling of the physical domain Ω with N points, where each point is characterized by a d-dimensional state vec- tor encoding physical properties. Traditional approaches to processing such features often rely on fixed spectral trans-
These special cases reveal how existing approaches subopti- mally prioritize either domain structure & or point states æ. To overcome these limitations, we introduce a few instanti- ations of coupling functions H that effectively harmonize
3

---

Learning PDEs in Unified Spectral-Physical Space
Output Function
Degraded HPM: Alication (implilmpl
Fleshsing
Hi
+
Holistic Spectral Eigenfunctiins
MLP
MatMul Ge. KFF)
Pruject log. XEP)
MacMul Ge. 0X70)
Mapping Layer
x
HC
PMMPP
x
w/O
Chaunel Mixer
Holistic Physics
Jia PM
MLP for llesiay)
MatMul lie Furz
Project (ogg XE
MacMul lie.Fibel
In Holistic Spectral Eigenfenction some local regions of spectrum bash are Bexildly modwlated for predkcting more accursie target function
LayerNurm
Mi
Bathetic Spactral Riganfanctiies
gemixer
H(
*MMLP(
x
The Reprososaniie insouttative
Integriing pein-llnnd plysins staiis (frmm and docaln lond spectal strecuues (froes 9%.
A
-Fised Spectral Eigenfunctions
LayerNorm
Degraded HPM: Fived Specteal Neural Operator
Deesses Price
MI.P tce idenlly)
MatMul Re. Re:J
Project Got NE
MatMul (ie."riet)
Projecting Layer
Spectral Elarafenction
Fixed eigenfenctions encode spectrom strecture of physical demaiin, but
Input Function
7%
overlook point-level plysical status.
Model Architecture
(b) Holistic Physics Mixer
(c) Holistic Spectral Eigenfunctions
Figure 2. (a) Overall architecture of Holistic Physics Solver. (b) Structure of Holistic Physics Mixer. (c) Visualization of Fixed Spectral
Eigenfunctions and Holistic Spectral Eigenfunctions.
spectral structure with point-wise adaptivity.
Form Point-wise Softmax Coupling. A straightforward approach using point-wise coupling with Softmax normal- ization:
Difference with Spectral Modes Selection. While some previous works (Guibas et al., 2021; George al., 2024) also incorporate data-dependent modulation spectral fea- tures, HPM differs with them in several aspects: (a) Dif- ferent Objective. Guibas et al. (2021); George et al. (2024) aim to improve efficiency and robustmess of spectral fea- tures by sparsifying the frequency modes. contrast, HPM focuses on enhancing the flexibility of preset spectral fea- tures via point-wise modulation. (b) Different Methodology. Guibas et al. (2021); George et al. (2024) select modes in the spectral domain post-rransformation, while HPM introduces spatial domain modulation pre- te-transformation enabling point-wise flexibility while preserving spectral structure. (c) Different Application. AFNO (Guibas et al., 2021) targets computer vision tasks and iFNO (George et al., 2024) focuses large-scale meshes, whereas HPM addresses the PDE learning challenge requiring both global spectral coherence and local adaptivity.
H α. ① Softmax(MMP((1)) O +
(11)
where @ represents element-wise multiplication. The Soft- max ensures balanced feature scales and lets each point adaptively focus on different spectral components. This de- sign provides an effective balance of stability and flexibility, making it our preferred choice.
Form Point-wise Sigmoid Coupling, variation using Sigmoid instead of Softmax:
H x . Sigmoid(MLP((:)) O 4
(12)
This removes the sum-to-1 constraint of Softmax, allowing multiple frequency components to be active simultancously.
Form 3: Global-pooled Coupling, An approach that first
aggregates spatial information:
These spectral processing techniques (Guibas et al., 2021; George et af., 2024) could complement HPM, potentially combining both benefits for diverse applications.
( 4)) = Softmax(Avgoool e Φ (13)
While computationally efficient, this global pooling sac- rifices the fine-grained adaptivity that makes wisnt-wise coupling effective.
3.2. Instantiations
The coupling function (æ,, 4 in Holistic Physics Mixer can be implemented in various ways to achieve different balances between spectral structure and point-wise flexibil- ity. In this section, explored several designs to find the effective way integrate spectral structure with wisn adaptivity under minimal overhead:
Form 4: Additive Coupling, A simple additive approach:
H a.. tb Softmax(MLP(z)) + +
(14)
This directly adds transformed point-wise features to the spectral basis.

---

Learning PDEs in Unified Spectral-Physicee Space
Form 5: Concatenation Coupling. A direct concatenation approach:
for token mixing:
V 1
mid
J-mixer
Holi
(LayerNorm(ei--))) + U;-1, (17)
(a, 4) = Softmax(LLP(Conaatte + +)));
(15)
8, FecdForwaaddLLyyeeNomm11 + aiii (18)
This processes the physical states and spectral basis jointly by concatenating them before applying the transformation.
We instantiate Project in J LayerNorm(-) followed by an MLP layer. Following Wu et al. (2024), we adopt multi-head design where features are processed by multi- ple parallel Holistic Physics Mixers and then concatenated. This enables the model to capture different aspects of the operator mapping simultaneously. The detailed multi-head implementation is described in Section A.2.1.
Discussion. Through empirical evaluation (Table 5), we found that the first form Point-wise Softmax Coupling- achieves superior performance. Although simple, this de- sign effectively balances several key requirements: (a) The point-wise coupling enables adaptive spectral processing at each spatial location. (b) The Softmax normalization ensures stable training and consistent feature scales. (c) The direct multiplication with basis functions maintains spectral structure while allowing flexibility. Moreover, the simple point-wise multiplication design maintains interpretability we can directly analyze how each spatial location modu- lates different frequency components, enabling insights into learned spectral processing patterns (see Section 4.6).
Calculation of Φ. Following Chen et al. (2023), we employ the cigenfunction of Laplace-Beltrami operator as & for its flexibility in handling both regular and irregular domains. Given the discrete representation of physical domain, we use the robust-laplacian library² to compute these eigenfunc- tions (Sharp & Crane, 2020). This allows processing both structured grids and unstructured meshes.
While more sophisticated coupling designs are possible, we found this straightforward integration provides effective point-wise adaptivity and stability, outperforming previous methods. We believe developing more advanced instantia- tions could potentially lead better performance.
Discussion. Holistic Physics Solver presents following ad- vantages: (a) It unifies spectral- based and attention-bast based approaches in a single representation space, enabling simul- taneous modeling of domain- level structure and point-wise adaptivity for complex PDEs. (b) Benefiting from the in- tegration of domain-level spectral priors and point-level flexibility, it not only preserves strong generalization on samples with unseen resolutions and in limited training data scenarios but also presents flexible adaptivity for handling fine-scale local details and utilizing increased training sam- ples. (c) It maintains computational efficiency through a simple coupling mechanism, avoiding attention's 's quadratic complexity (Wu et al., 2024) while adding minimal over- head to fixed spectral methods (Chen al., 2023).
3.3. Universal Approximation Property
Theorem 3.1 (Universal Approximation Property). Holistic Physics Mixer is learnable integral neural operator (Ko- vachki et al., 2023), indicating possesses universal ap- proximation capability for continuous operator mappings between function spaces. The proof is provided in Sec- tion 1.1.2 of the Appendix.
The universal approximation capability provides the theoret- ical guarantee that Holistic Physics Solver can approximate any continuous operator mapping with arbitrary precision, making it suitable for learning solutions of various PDEs.
4. Experiment
4.1. Overview
Baselines. We compare HPM against a lot of existing neural operators, including both spectral-based and attention-based approaches. For attention methods, we use Transolver (Wu et al., 2024) as the primary baseline due its leading perfor- mance across diverse PDE problems. For spectral methods, we construct SpecSolver our main spectral baseline by adapting fixed spectral operators into the same modern ar- chitecture as Transolver (Wu et al., 2024), ensuring fair comparison by maintaining architecture consistency while preserving the core benefits of spectral approaches.
3.4. Implementation of Holistic Physies Solver
We implement Holistic Physics Mixer based PDE Solver (de- noted as HPM) by inserting Holistic Physies Mixer Fmixer into existing model architectures (Wu et al., 2024). We used the toolkit of PaddlePaddle to develop this new model and solved the problem.
Network Architecture. Following previous works (Wu et al., 2024; Tran et al., 2021),, we implement the Holistic Physics Solver as stack of Holistic Physics Blocks, along with projecting layers P and V the beginning and end:
Main Results. Our results lead following findings:
HPM consistently outperforms both spectral-based and
Holli = Va o Mfoli e
MfHlli MHoli P, (16)
"Rouustlapplacian library link: https: http://gttttbbooo///
where each block MoHl employs Holistic Physics Mixer
nmwsharp/nonman foid-
5

---

Learning PDEs in Unified Apectral-Physitee Space
Table I. Performance comparison on structured mesh prohlems.
Darcy Flow
Airfoil (Irregular, Steady)
Navier-Stokes (Regular, Time)
Plasticity (Irregular, Time)
Model
(Regular, Steady)
FNO (Li et al., 2020)
1.08e-2
1.56e-1
WMT (Gupla et al., 2021)
8.20c-3
7.50c-3
1.54c-1
7.60e-3
U-FNO (Wen al., 2022)
1.83e-2
2.69c-2
2.23e-1
3.90e-3
Geo-FNO (Li et al., 2023a)
1.08e-2
1.38e-2
1.56e-1
7.40e-3
Spectral Methods
U-NO (Rahman et al., 2022)
1.13e-2
7.80e-3
1.71e-1
3,40e-3
F-FNO (Tran al., 2021)
7.70e-3
7.80e-3
2.32e-1
4.70e-3
LSM (Wu et al., 2023)
6.50e-3
5,90e-3
1.54e-1
2.50e-3
NORM (Chen al., 2023)
9.71e-3
5,44e-3
1.15e-1
4.39c-3
SpecSolver
5.41e-3+1.1ed
3.13e-3+1778
9.46e-2 212244
1.21e-3-a61e
Galerkin (Cao, 2021)
8,40c-3
1.18e-2
1.40e-1
1.20e-2
HT-Net (Liu al., 2023b)
7.90e-3
6.50e-3
1.85e-1
3.33e-2
OFORMER (Li et al., 2023b)
1.24e-2
1.83e-2
1.71e-1
1.70e-3
Attention Methods
GNOT (Hao ct al., 2023)
1.05c-2
7.60c-3
1.38c-1
3.36c-2
FactFormer (Li et al., 2023c)
1.09e-2
7.10e-3
1.21e-1
3.12e-2
ONO (Xiao et al., 2023)
7.60e-3
6.10e-3
1.20e-1
4.80e-3
Transolver (Wu al., 2024)
5,70c-3
5,30c-3#1.00-4
9.00c-2+13003
1.23c-3=1.0044
HPM
4.59e-3 11.944
4.72e-3
7.34e-2.1.22ee
8.00e-4 -.558e5
attention-baat based approaches across various PDE problems (Section 4.2 and 4.3).
the evaluation metric, lower value indicating higher PDE solving accuracy. (c) Baselines. We compare HPM with a lot of neural operators, covering both spectral- based meth- ods and attention- based methods. Section A.3 presents more experimental setup detail.
HPM exhibits strong generalization and adaptation ca- pabilities, maintaining robust performance with limited training data while efficiently utilizing additional data when available. It also demonstrates excellent zero-shot generalization to unseen resolutions (Section 4.4).
Quantitative Comparison. Table 1 presents the quantita- tive results. HPM significantly improves the performance over past spectral based methods LSM (Wu et al., 2023) and NORM (Chen al., 2023), and outperforms the most per- formed ittention-based method Transolver (Wu al., 2024). This concludes that holistic spectral processing effectively integrates domain-level spectral prior with point-level adap- tivity, leading to better feature learning for operator learning on various problems.
Analysis of learned holistic spectral processing patterns reveals meaningful physical insights that could guide fixed spectral operator design (Section 4.6).
Additional results show that HPM achieves faster con- vergence during training (Figure 5), requires minimal computational overhead compared to fixed spectral meth- ods (Table 6), and maintains superior performance even with reduced frequency numbers l (Table 9).
Qualitative Comparison. In Figure 3, we visualize the prediction error of HPM and Transolver on different prob- lems. The prediction error of HPM is evidently reduced, especially physical boundaries and some regions with sharp status changes. This further demonstrates the superior operator learning capability of HPM.
4.2. Structured Mesh Problems
This section compares HPM with previous neural operators on structured mesh problems, where the physical domains are represented with meshes aligned with standard rectangle grids. For these problems, we implement HPM with LBO eigenfunctions calculated on standard rectangle grids.
4.3. Unstructured Mesh Problems
This section compares HPM with previous works on un- structured mesh problems, where the physical domains are represented with irregular triangle meshes. For handling these problems, we independently calculate LBO eigenfune- tions for each problem based on their triangle meshes.
Setup. (a) Problems. The experimental problems include two regular domain problems Darcy Flow and lavier-Stokes from Li et al. (2020), and two irregular domain problems Airfoil and Plasticity from Li et al. (2023a). Darey Flow and Airfoil are steady-state solving problems, while Navier- Stokes and Plasticity are time-series solving problems. (b) Metric. Same as previous works (Li et al., 2020; Wu et al., 2024; Gao & Wang, 2023), we use Relative L2 between the predicted results and ground truth (the simulated results) as
Setup. (a) Problems. The evaluated problems include Ir- regular Darcy. Pipe Turbulence, Heat Transfer, Composite, and Blood Flow from Chen et al. (2023). All problems
6

---

Learning PDEs in Unified Jpectral-Physicee Space
Zorget
Erew
Enor
Error
Arw
Enwer
221 S1
111 26
Ground Truth
HPM Foow
Ground Truth
HPM
Ground Truth
HPM
Firer
Fror
Firter
Firur
45
SpecSolver Transolver
SpecSolver
Transolver
SpecSolver
Transolver
HPM
SpecSolver
Transolver
(n) Darcy Flow
(b) Navier-Stokes
(c) Plasticity
(d) Different Resolution on Airfol
Figure 3.
Prediction error visualization on different problems.
Table 2. Performance comparison on unstructured mesh problems.
Irregular Darey
Pipe Turbulence
Heat Transfer
Composite (8232 Nodes)
Blood Flow
Model
(2290 Nodes)
(2673 Nodes)
(7199 Nodcs)
(1656 Nodes)
GraphSAGE (Hamilton et al., 2017)
6.73e-2-5530e
2.36e-1
2.09e-1 =5.00e
DecpOnet (Lu ct al., 2019)
1.36c-2:1.30e-
9,36c-2:1.0ne
7.20c-4=2.00s
1.88c- 2:3,40--
8.93c-1 #2.37622
POD- ODD-DeepOnet (Lu et al., 2022)
1.30c-2=2.30e4
2.59c-2a2.55e
5.70c-4:1.00ss
1.44c-2:6.00e
2:6000e4
3.74c-1=1.19-3
FNO (Li al., 2020)
3.83e-2=77700c
3.80e-2::20065
NORM (Chen et al., 2023)
1.05e-2 2.000e4
1.01e-2 +22022222..000 +2.00e
2.70e-4:200e-5
9.99e-3#2.70e- #2.70e44
4.82c-2 :6. 10-
SpecSolver
7.96e-3=7.tn
1.1 +1 1.00633
1.1
1.00e-2 2
3.73e-2
HPM
7.38c-3 66200-5
8.26e-3 67600e44
1.84e-4 +22276-5
9.34c-3 +2771e4
2.89c-2 33.25633
come from realistic industry scenarios and include both steady-state problems and time-series problems. (b) Met- ric. Same as Section 4.2, Relative L2 between the pre- dicted results and ground truth (the simulated results) is used as the evaluation metric, lower value indicating better performance. (c) Baselines. The compared methods include GraphSAGE (Hamilton et al., 2017). DeepOnet (Lu et al., 2019), POD-DDeepOnet (Lu et al., 2022), FNO (Li et al., 2020) and NORM (Chen et al., 2023). Section A.3 presents more experimental setup detail.
Table 3. Zero-shot resolution generalization on Airfoil.
Resolution
Transolver
SpecSolver
HPM
Training Resolution
221 51
5.24e-3
5.33e-3
4.38e-3
Consistent Ratio
26
7.68e-2
1.90e-2
1.74e-2
9.73e-2
7.30e-2
5.34--2
Inconsistent Ratio
221 x 26
7.85e-2
1.91e-2
1.69e-2
111 x 51
1.26e-2
5.80e-3
5.37e-3
lower value indicating preferred performance.
Results. The results are shown in Table 2. Compared to previous methods, HPM obtains consistent enhanced performance across all problems. This validates the benefits of our holistic spectral processing approach that effectively integrates domain-level structure with point-wise adaptivity on complex physical domains and operator mappings.
Table presents the quantitative comparison results, where significant performance gaps between Transolver and HPM are observed. Additionally, we visualize the prediction error of different resolutions in Figure (d) (more in Figure 8), In contrast to Transolver, HPM apparently diminishes pre- diction error, particularly lower-resolution samples.
4.4. Generalization Capability Comparison
This indicates that HPM retains remarkable resolution gen- eralization ability like spectral- methods while main- taining the flexibility of attention mechanisms, thus outper- forming both Transolver and SpecSolver.
This section compares the generalization performance of HPM with the attention-based neural operator Tran- solver (Wu al., 2024) and the spectral-basse basel SpecSolver.
Limited Training Numbers. We additionally evaluate the generalization ability of HPM, Transolver (Wu et al., 2024). and SpecSolver with limited training data amount. Specif- ically. for Darcy Flow and Navier-Stokes, we train neural operators with 200, 400, 600, 800, and 1000 trajectories re- spectively, and then test on additional 200 trajectories. Same as other experiments, we use Relative L2 as the performance measure, with lower value meaning better performance.
Zero-shot Resolution Generalization. We evaluate the zero-shot capabilities of HPM, Transolver, and SpecSolver on samples with unseen resolutions on Airfoil. The model trained on the 211 x 51 resolution and then tested on lower resolutions including 111 26 and 45 11, as well varied ratio resolutions including 221 x 26 and 111 51. We utilize Relative L2 the performance metric, with

---

Learning PDEs in Unified Spectral-Physical Space
Table Comparison on different training numbers.
Table Comparison of different forms of H(,, +).
Training Number
Implementation of (xx, +)
Relative Error
Problem
Transolver
SpecSolver
HPM
Forml: Softmax (a]) 0
4.38e-3
200
1.75c-2
1.10e-2
1.06e-2
Form2: Sigmoid(MLPP (z)) @
4,86e-3
400
1.04c-2
7.32c-3
6.66e-3
Form3: Softmax Avg @ +
5.47e-3
Darcy Flow
600
6,87e-3
6.20e-3
6.03e-3
Form4: Softmax (MLP(()))) +
4.69c-3
800
6,33e-3
5.64c-3
4.98e-3
Form5: Dottmax(MLP((Conaall +)))
5,24c-3
1000
5.24e-3
5.33e-3
4.38e-3
Table 6. Comparison of inference time.
200
3.76e-1
1.93e-1
1,85e-1
400
3.14c-1
1.48c-1
1.26e-1
Model
Parameter Count
Inference Time
Navier-Stokes
600
2.87e-1
1.21e-1
1.17e-1
SpecSolver
601,537
14,8 ms
800
2.49c-1
1.04e-1
8.25c-2
Transolver
2,819,521
30.9
1000
9.60e-2
9.34e-2
7.44e-2
HPM
751,041
17.4 ms
Table shows the results. HPM outperforms both base- lines across all training data quantities. With limited data, HPM matches SpecSolver while significantly outperform- ing Transolver, showing the benefits of spectral continuity prior. As the data amount increases, HPM leverages its point-level flexibility to fully exploit additional training data and surpass both baselines.
physical integration strategies.
Modulation Pattern Definition. We use H(x;, +) Softmax (ML(ma)) @ o coupling function and study how each point modulates different spectral basis functions by visualizing M(x) - Softmax(MLP( (MLP(x)). ). To show this clearly, for each point, we calculate the difference between how much it uses high frequencies (last half of basis func- tions) versus low frequencies (first half). This gives us a score from -1 to 1, where positive means stronger high frequencies and negative means stronger low frequencies.
This illustrates that HPM effectively combines domain-level spectral structure with point-wise adaptivity for strong per- formance across different data regimes.
Pattern Analysis Results. In Figure.4 4 (more in Figure.6. 7), we show how a 4-layer HPM for Darcy Flow and an 8-layer HPM for Airfoil problem. We found three main patterns: (a) Spatially Adaptive Modulation. Different parts of the space use different patterns some areas focus on high-order functions, others on low-order ones. This shows that HPM adjusts its processing based local features. (b) Layer-wise Evolution. The patterns change in clear way through the layers. Early layers generally use uniform patterns across space, middle layers vary more and use more high-order functions where needed, and later layers return to uniform patterns across space. (c) Physical Feature Enhancement. At boundaries and areas where solutions change rapidly, the model uses more high-order functions, showing these areas need finer detail for prediction.
4.5. Additional Comparison
Different Holistic Instantiations. We compare several instantiations to find the effective way to fuse spectral struc- ture with point- adaptivity under minimal overhead.
The results in Table 5 highlight two crucial coupling insights: First, point-wise processing is essential As Form 3 shows, local adaptivity cannot be achieved through global opera- tions. Second, enforeing competition between frequency components via Softmax proves more effective, likely due to explicit frequency trade-offss each point. We believe developing more sophisticated coupling mechanisms that better balance local flexibility with global spectral structure could potentially achieve better performance.
Inference Time Comparison. We compare the inference time of different methods on the Airfoil problem, using sin- gle RTX 3090 with batch size 1. As shown in Table 6, HPM adds minimal overhead over SpecSolver while maintaining fewer parameters and faster inference than attention-based method Transolver (Wu et al., 2024).
Guiding Fixed Basis Design. HPM's learned patterns can help improve fixed spectral methods. We test if adding more high-frequency components in the middle layers, based on HPM's layer evolution patterns, could help. As Table 7 shows, manually adding high-frequency components in mid- dle layers works better than adding in carly late layers. This shows that HPM's patterns reveal useful information about how to process spectral features.
4.6. Analysis of Learned Modulation Patterns
This section studies how HPM learns to modulate spectral patterns. Benefiting from the straightforward point-wise multiplication design of H(.,)), J we can directly visualize how the model adjusts different spectral components across network layers, providing insights into its learned spectral-
In summary, HPM learns patterns that change across both space and network layers, balancing local flexibility with domain spectral structure. These patterns help us understand how HPM works and design better fixed spectral methods.
8

---

Learning PDEs in Unified Spectral- Physsall Space
Inpue
Layer
Layer
3ayer
Layer
High
Input
Layer
Layer
Layer
Layer
Layer
Layer
Layer?
Layer
High
Hlad Hend
Hrod Head
Freguemey
Ereqummry
Head
Head Head
Onrgut
Ourput
Hend
Low
Low
(a) Layer HPM for Darcy Flow
(b) 8 Layer HPM for Airfoil
Figure Visualization learned holistic spectral modulation patterns on Darey Flow and Airfoil.
Table Fixed basis design guided by HPM's modulation patterns.
Acknowledgments
Basis Enhancement Strategy
Relative Error
This work supported by National Science and Technology Major Project (2022ZD01 17802). This work was supported in part by General Program of National Natural Science Foundation of China (62372403) and the Earth System Big Data Platform of the School of Earth Sciences, Zhejiang University.
Enhanced High-Freq in Early Layers
5.12e-3
Enhanced High-Freq in Middle Layers
4.84e-3
Enhanced High-Freq in Late Layers
4.97e-3
SpecSolver (No Enhancement)
5.33e-3
HPM (Adaptive Modulation via H(:.))
4.38e-3
5. Limitation and Future Work
Impact Statement
Despite obtaining superior performance a lot of scenarios, the introduced Holistic Physies Mixer inevitably suffers certain limitations that do affect the core conclusion of this work, and they worth further exploration in future works. (a) Firstly, the current coupling function H(x, + remains a fundamental design, without considering demands in particular circumstances. Therefore, is meaningful to develop more sophisticated coupling mechanisms for specific requirements. (b) Additionally, although this work has explored a broad PDE solving problems, numerous real- world physical systems still warrant further investigation. It is significant to investigate the application of Holistic Physics Transform in general deep learning tasks such as time-series signal prediction and computer vision learning.
This paper presents work whose goal is to advance the field of Machine Learning based PDE solving. There are many potential societal consequences of work, of which we feel must be specifically highlighted here.
References
Alkin, B., Fürst, A., Schmid, S. L., Gruber, L., Holzleitner, M., and Brandstetter, J. Universal physics transformers: A framework for efficiently scaling neural operators. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Bonev, B., Kurth, T., Hundt, C., Pathak, J., Baust, M., Kashinath, K., and Anandkumar, A. Spherical fourier neu- ral operators: Learning stable dynamics on the sphere. In International conference on machine learning, pp. 2806 2823. PMLR, 2023.
6. Conclusion
This work presents Holistic Physics Mixer based Neural Operators (HPM), which integrates domain-level spectral structure and point-level physical states into a unified repre- sentation space for operator learning. The proposed Holistic Physics Transform holds significant potential applications numerous physics-informed deep learning tasks, as it en- ables adaptive processing that considers both local physical variations and global spectral properties. Comprehensive experiments validate the superior performance of HPM in various PDE solving scenarios, benefiting from its ability to simultancously capture domain-level continuity constraints and point-level physical dynamics. We hope HPM's s unified treatment of physical systems can inspire future explorations neural operator learning.
Calvello, E., Kovachki, B., Levine, M. E., and Stuart, M. Continuum attention for neural operators. arXiv preprint arXiv:2406.06488 2024.
Cao, Q. Goswami, S., and Karniadakis, G. E. Laplace neural operator for solving differential equations. Nature Machine Intelligence, 6(6):631 2024.
Cao, Choose transformer: Fourier or galerkin. Ad- vances in neural information processing systems, 34: 24924 24940, 2021.
Chen, G., Liu, X., Meng, Q., Chen, L.., Liu, C., and Li,

---

Learning PDEs in Unified
Space
Y. Learning neural operators riemannian manifolds. arXiv preprint ar arXiv:2302.088666 2023.
Kovachki, N., Li, Z., Liu, B., Azizzadenesheli, K., Bhat- tacharya, K., Stuart, A., and Anandkumar, A. Neural operator: Learning maps between function spaces with applications to pdes. Journal of Machine Learning Re- search, 24(89):1 2023.
Chen, T., Zhou, H., Li, Y., Wang, H., Gao, C., Shi, R., Zhang, S., and Li, J. Building flexible machine learning models for scientific computing at scale. arXiv preprinl arXiv:2402.16011 2024.
LeCun, Y., Bengio, Y., et al. Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks, 3361(10):1995, 1995.
Chung. J., Gulcchre, C., Cho, K., and Bengio, Y. Empirical evaluation of gated recurrent neural networks sequence modeling. arXiv preprint arXiv:1412.3555, 2014.
Lee- Lee--horp. J., Ainslie, J., Eckstein, I,, and Ontanon, S. Fnet: Mixing tokens with fourier transforms. arXiv preprint arXivv2105.038824 2021.
Gao, W. and Wang, C. Active learning based sam- pling for high-dimensional nonlinear partial differen- tial equations. J. Comput. Phys., 478:1, February 2023. URL https:
Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Bhat- tacharya, K., Stuart, A., and Anandkumar, A. Fourier neural operator for parametric partial differential equa- tions. arXiv preprint arXiv:2010.08895 2020.
2022. 111848.
Gao, W. Xu, R., Deng, Y., and Liu, Y. Discretization- invariance? on the discretization mismatch errors in neu- ral operators. The Thirteenth International Conference on Learning Representations, 2025.
Li, Z.,, Huang, D. Z., Liu, B., and Anandkumar, A. Fourier neural operator with learned deformations for pdes general geometries. Journal of Machine Learning Re- search, 24(388): 2023a.
George, R. J., Zhao, J., Kossaifi, J., Li, Z., and Anandkumar, A. Incremental spatial and spectral learning of neural operators for solving large-scale pdes. Transactions on Machine Learning Research, 2024.
Li, Z., Meidani, K., and Farimani, A. B. Transformer for partial differential equations' operator learning. Transac- tions on Machine Learning Research, 2023b. ISSN 2835- 8856. URL https: http://pppeeeeeeeeeeeeeeeeeeeeee net forum? i 1d=EPPQt
Guibas, J., Mardani, M., Li, Z., Tao, A., Anandkumar, A., and Catanzaro, B. Adaptive fourier neural operators: Efficient token mixers for transformers. arXiv preprint arXiv:2 13587, 2021.
Li, Z., Shu, D., and Farimani, A. B. Scalable transformer for pde surrogate modeling, 2023c.
Gupta, G., Xiao, X., and Bogdan, P. Multiwavelet-based operator learning for differential equations. Advances in neural information processing systems, 34:24048-24062. 2021.
Li, Z., Shu, D., and Barati Farimani, A. Scalable trans- former for pde surrogate modeling. Advances in Neural Information Processing Systems, 36, 2024.
Lingsch, L., Michelis, M. Y., de Bezenac, E., Perera, S. M., Katzschmann, K., and Mishra, S. Beyond regular grids: Fourier-based neural operators arbitrary do- mains. arXiv preprint 2xXvv:2305.1966 2023.
Hamilton, W., Ying, Z., and Leskovec, J. Inductive repre- sentation learning on large graphs. Advances in neural information processing systems, 30. 2017.
Hao, Z.,, Wang, Z., Su, H., Ying, C., Dong, Y., Liu, S., Cheng, Z., Song. J., and Zhu, J. Gnot: A general neu- ral operator transformer for operator learning. In Inter- national Conference on Machine Learning, pp. 12556- 12569. PMLR, 2023.
Liu, N., Jafarzadeh, S., and Yu, Y. Domain agnostic fourier neural operators. Advances in Neural Information Pro- cessing Systems, 36, 2024a.
Liu, P., Hao, Z.,, Ren, X., Yuan, H., Ren, J., and Ni, Papm: A physics-aware proxy model for process systems. arXiv preprint arXiv:2407.05232. 2024b.
Hao, Z., Su, C., Liu, S., Berner, J., Ying, C., Su, H., Anand- kumar, A., Song, J., and Zhu, J. Dpot: Auto-regressive denoising operator transformer for large-scale pde pre- training, arXiv preprint arXiv: arXiv:2403.03542 2024.
Liu, R., Wang, X., Wang, W., and Yang, Y. Bird' hird's-eye-view scene graph for vision-language navigation. In ICCV, 2023a.
Katharopoulos, A.,
Vyas, A., Pappas. N., and Fleuret, F. Transformers are mns:: Fast autoregressive transformers with linear attention. In International conference on ma- chine learning, pp. 5156-5165, PMLR, 2020.
Liu, R., Wang, W., and Yang, Y. Volumetric environment representation for vision- lisgun-language navigation. In CVPR, 2024c.
10

---

Learning PDEs in Unified
Space
Liu, X., Xu, B., and Zhang, L. HT-net: Hierarchical transformer based operator learning model for multiscale
Tang, L., Yin, Z., Su, H., Lyu, W., and Luo, B. Wiss: weighted fusion of spectral transformer and spatial self- attention for robust hyperspectral image classification against adversarial attacks. Visual Intelligence, 2(1):5, 2024.
PDEs, 2023b. URL https: http:///wppeeeeeeeeeeeeeeeeeeeee / Corum?2a--YY22000kkee
Lu, I.., Jin, P., and Karniadakis, G. E. Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of opera- tors. arxiv preprint arXiv:1910.03199 2019.
Tran, A., Mathews, A., Xie, L., and Ong, C. S. Factorized fourier neural operators. arXiv preprint arXi::2111.33800 2021.
Tripura, T. and Chakraborty, S. Wavelet neural opera- neural operator for parametric partial differential equations, 2022. URL https: 2205 02191
Lu, L., Jin, P., Pang, G., Zhang, Z.., and Karniadakis, G. E. Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nature machine intelligence, 3(3):218-23) 229,, 2021.
abs/
Vaswani, A. Attention all you need. Advances Neural Information Processing Systems, 2017.
Lu, L., Meng, X., Cai, S., Mao, Z., Goswami, S., Zhang, Z., and Karniadakis, G. E. A comprehensive and fair comparison of two neural operators (with practical exten- sions) based on fair data. Computer Methods in Applied Mechanics and Engineering, 393:1 14778, 2022.
Wang, T. and Wang, C. Latent neural operator for solv- ing forward and inverse pde problems. arXiv preprint prXiv:2406.039255 2024.
McCabe, M., Blancard, B. R.-S., Parker, L. H., Ohana, R., Cranmer, M., Bietti, A., Fickenberg, M., Golkar, S., Krawezik, G., Lanusse, F., et al. Multiple physics pre- training for physical surrogate models. arXiv preprint SXiv:2310.02994,, 2023.
Wen, G., Li, Z., Azizzadenesheli, K., Anandkumar, A., and Benson, S. M. U-fno-an an enhanced fourier neural operator-based deep-learning model for multiphase flow. Advances in Water Resources, 163:104180, 2022.
Wu, H., Hu, T., Luo, H., Wang. J., and Long, M. Solving high-dimensional pdes with latent spectral models. arXiv preprint arXiv:2301. 12664, 2023.
Poli, M., Massaroli, S., Berto, F., Park, J., Dao, T., Ré, C., and Ermon, S. Transform once: Efficient operator learn- ing in frequency domain. Advances in Neural Information Processing Systems, 35:7947-7959, 2022.
Wu, H., Luo, H., Wang. H., Wang. J., and Long, M. Tran- solver: A fast transformer solver for pdes on general geometries, arXiv preprint arXiv:2402.02336. 2024.
Qin, S., Lyu, F., Peng, W., Geng, D.. Wang, J., Gao, N., Liu, X., and Wang, L. L. Toward better understanding of fourier neural operators: Analysis and improvement from spectral perspective. arXiv preprint arXiv:2404.07200. 2024.
Xiao, Z., Hao, Z., Lin, B., Deng, Z., and Su, H. Improved operator learning by orthogonal attention. arXiv preprint arXiv:2310.12453 2023.
Ye, Z., Huang, X., Chen. L., Liu, Z. Wu, B., Liu, H., Wang. Z., and Dong, B. Pdeformer-1: A foundation model for one-dimensional partial differential equations. arXiv preprint arXiv:2407.06661 2024.
Rahman, M. A., Ross, Z. E., and Azizzadenesheli, K. U-no: U-shaped neural operators. arXiv preprint 1rXiv:2204.11127, 2022.
Raonic, B., Molinaro, R., De Ryck, T., Rohner, T., Bar- tolucci, F., Alaifari, R., Mishra, S., and de Bézenac, E. Convolutional neural operators for robust and accurate learning of pdes. Advances in Neural Information Pro- cessing Systems, 36, 2024.
Yue, X., Zhu, L., and Yang, Y. Deltaphi: Learning phys- ical trajectory residual for pde solving. arXiv preprint arXiv:2406.0979) 2024.
Zhang, X., Helwig, J., Lin, Y., Xie, Y., Fu, C., Wojtowytsch, S., and Ji, S. Sinenet: Learning temporal dynamics in time-dependent partial differential equations. In The Twelfth International Conference on Learning Represen- tations, 2024. URL forum wit.midd--SYY2hLKCC
Sharp. N. and Crane, K. A laplacian for nonmanifold trian- gle meshes. In Computer Graphics Forum, volume 39, pp. 69-80. Wiley Online Library, 2020.
Shen, J., Marwah, T., and Talwalkar, A. Ups: Efficiently building foundation models for pde solving via cross- modal adaptation. Transactions on Machine Learning Research.

---

Learning PDEs in Unified Spectral- Physscal Space
A. Appendix
A.I. Theoretical Foundations
We first introduce the preliminary lemmas about neural operator learning in Section A.I.I. Next, we provide the theoretical demonstration (Theory A.4) that Holistic Physies Mixer the learnable integral neural operator in Section A.1.2.
A.I.I. PRELIMINARY THEOREM: INTEGRAL NEURAL OPERATOR LEARNING
The following theorems summarized from previous works (Li et al., 2020; Kovachki al., 2023; Wu et al., 2024), which provide the theoretical basis of the proposed Holistic Physics Mixer. Theorem A.1. PDEs could be solved by learning integral neural operators.
Kovachki al. (2023) formulate the common architecture of neural operators for PDE solving as a stack of network layers.
G. = Q o KG) ..... o (W; + K.) ....0 o o + Ki) P.
(19)
where P and Q are both linear point-wish wise projectors as shown in Equation W; is the point-wise fully connected layer and K. is the non-local integral operator.
In each network layer, the key is to learn the non-local integral operator K2; defined as follows:
(x))(()== = (z E, u(x), (()))
(20)
where u is the input function and the physical domain. As presented in (Kovachki al., 2023), the learnable integral kernel operator enables the mapping between continuous functions, similar to the weight matrix operation that enables the mapping between discrete vectors. It could be demonstrated that various neural operators (Li al., 2020; Cao, 2021; Chen et al., 2023; Wu al., 2024) are learning different kernel functions of the stacked integral neural operators shown in Equation 20.
Lemma A.2. ENO (Li et al., 2020) learns integral neural operators.
This is demonstrated in (Li al., 2020) and (Kovachki et al., 2023). By setting the kernel function as k<xx, &, u(z), u(E)) <(x E), it could be demonstrated that the kernel integral operator could be implemented with Fourier Transform. For more details you refer to (Li et al., 2020).
Lemma A.3. The standard Transformer (Vaswani, 2017) learns integral neural operators.
(Kovachki et al., 2023) demonstrates that the canonical attention mechanism (Vaswani, 2017) a special case integral neural operators. This could be demonstrated by setting the kernel function as follows:
6(x, (z)) u(()) = .// exp(W, (
(21)
where W, E Rdxd W Rdd and R Rddx are all the training parameter of the neural network. For simplification, eliminate the division operation with Vd. With this formulation, we can derive the attention mechanism based on the kernel integral operator shown in Equation 20 and Monte-Carlo approximation. The proof can be found in (Kovachki et al., 2023). Therefore, the attention mechanism could be employed for PDE solving.
A.1.2. HOLISTIC PHYSICS MIXER AS INTEGRAL NEURAL OPERATORS
Theorem A.4. Holistic Physics Mixer is a learnable integral neural operator.
Proof. The Holistic Physics Mixer is represented in the following form:
Finiter (x) = THPT Project THPT(x).
(22)
where x E R Nxd x is the input feature, THPT and HPT represent the Holistic Physics Transform and its inverse transform:
(H) = H 4)TT a.,
(23)
( H(x,, .)
(24)
12

---

Learning PDEs in Unified Spectral-Physicee Space
To prove this is integral neural operator, construct the kernel function of the integral operator follows:
x(x, 5, (()), u(E)) H (z. 4 ((x (E)
(25)
where (æ, (2) represents the æ-th row of matrix ( + E IR N** and R E Rdxd is a learnable parameter matrix.
Based on this kernel function, we can derive the Holistic Physics Mixer from the integral neural operator formulation:
[ n(x,5, u(x), (()))
(x)
H(æ, (x) (5) Ru(E)dg
(Equation 25)
HH( (x R
H(x 4 ()))
Q
(26)
Ha (T) R Σ
H( +) u(E) (Monte-Carlo approximation)
(ax, 4) (z) RTur((æ)
-TiTT (z)
-THrl Project (z),, (Matrix multiplication as Project),
where Q2 is the set of sampled points from domain Ω. The Monte-Carlo approximation requires: (a) The sampling points in Y' are sufficiently dense. (b) The coupling function is continuous and bounded. Project (including LayerNorm and FC layer) can represented as matrix multiplication that independent of spatial locations.
The final form in Equation 26 is exactly same as Holistic Physics Mixer defined in Equation 22. This concludes that Holistic Physics Mixer is equivalent to integral neural operator with kernel function defined in Equation 25.
A.2. Methodology Extension
A.2.1. MULTI-HEAD HOLISTIC PHYSICS MIXER
Following the multi- head attention mechanism (Vaswani, 2017; Wu et al., 2024), we enhance the Holistic Physics Mixer by introducing a multi-head architecture that processes features in parallel holistic spectral spaces. Specifically, first split the latent features æ R-N into h vectors hen--1 heead-222 ... x. along the channel dimension, where (i x (i-1):das xi] and h denotes the number of heads. dhead = d is the dimension of features in single head.
x head.
Next, every vector x heud-i e R mdd is independently processed by Fmixer The Holistic Physics Mixer in each head formulated as:
Tmixer (x) Tupp FC LayerNorm o THPr(x))
(27)
where LayerNorm(.) is introduced to normalize the holistic spectral features for more efficient optimization and enhanced generalization. Additionally, we share the learnable weights of FC for all holistic spectral components.
Finally, all output vectors are concatenated as the final output:
mmulti head mixer (x) Concat (F (z head-i))) Holi
(28)
The multi-head design brings several benefits:
Each head can learn different coupling patterns between spectral and physical information, allowing the model to capture various aspects of the operator mapping simultancously.
The parallel computation of multiple heads enables efficient implementation on modern hardware.
The concatenation of multiple heads provides richer feature representations that combine different aspects of the holistic spectral space.
In practice, we find that using 4-8 heads typically provides good performance across different PDE problems. The specific head numbers used for each problem are reported in Table
13

---

Learning PDEs in Unified Spectral- Physsccl Space
A.2.2. SPARSE-FREQUENCE FREQUENCY FIXED SPECTRAL TRANSFORM
To enhance the learning capability of fixed spectral methods, we attempt to manually add high-frequency spectral features in several network layers based on insights gained from HPM''s learned coupling patterns. Specifically, instead of using the lowest & frequencies, uniformly take & frequencies from the lowest & x frequencies, where r is the sparsity ratio. Higher r indicates using more high frequencies and we set = 2 and r = 4 for different layers.
We find that using Oparse-Frequenge Spectral Transform in partial network layers effectively improves the performance of fixed spectral methods. However, such manual frequency design relies on prior knowledge and repeated experiments to select appropriate layers and sparsity ratios, and additional computational cost is required for calculating LBO eigenfunctions with k x r frequencies. To address this issue, we experiment with the fixed spectral design guided by learned coupling patterns of HPM, as shown in Table 7.
A.3. Experiment Setups
A.3.1. IMPLEMENTATION DETAIL
We implement HPM with comparable parameter count to the compared baselines (Hao al., 2023; Wu et al., 2024; Chen al., 2023). The same optimizer setup Transolver (Wu al., 2024) is employed. All experiments (including all baselines, ablations and our method) could be conducted with single A100 device. The implementation detail for each problem is presented in Table 8.
A.3.2. METRIC
Same as previous works (Li et al., 2020; Wu et al., 2024), the assessed metric in this work is the Relative L2 Error, formulated as follows:
[
uslle
L2
(29)
N
[1
where Nust is the number of evaluated samples, û, represents the predicted trajectory, and denotes the ground-truth trajectory.
A.3.3. EVALUATED PDE PROBLEMS
Darcy Flow. Darcy Flow is a steady-state solving problem from Li et al. (2020). We experiment with the identical setup as previous works (Li al., 2020; Tran et al., 2021; Wu et al., 2024). The resolution of input and output functions 85 85 and there are 1000 trajectories for training and additional 200 data for testing.
Navier-Stokes. Navier-Stoken is the PDE solving problem introduced in FNO (Li et al., 2020). We experiment with the most challenging split where the viscosity coefficient is le-5. The input is the vorticity field of the first 10 time steps and the target is to predict the status of the following 10 steps. The training and test amounts are 1000 and 200 respectively.
Airfoil. Airfoil is an irregular domain problem from Geo-FNO (Li et al., 2023a). In this experiment, the neural operators take the airfoil shape as input and predict the Mach number the domain. The irregular domain is represented as structured meshes aligned with standard rectangles. All airfoil shapes come from the NACA-0012 case by the National Advisory Committee for Aeronautics. 1000 samples are used for training and additional 200 samples are used for evaluation.
Plasticity. This task requires neural operators to predict the deformation state of plasticity material and the impact from the upper boundary by an tregular-shaped rigid die. The input the shape of the die and the output is the deformation each physical point in four directions in future 20 time steps. There are 900 data for training and an additional 80 data for testing.
Irregular Darcy, This problem involves solving the Darcy Flow equation within an irregular domain. The function input a(x), representing the diffusion coefficient field, and the output u(x) represents the pressure field. The domain is represented by triangular mesh with 2290 nodes. The neural operators are trained on 1000 trajectories and tested on extra 200 trajectories.
Pipe Turbulence. Pipe Turbulence system is modeled by the Navier-Stokee equation, with an irregular pipe-shaped computational domain represented as 2673 triangular mesh nodes. This task requires the neural operator to predict the next frame's velocity field based on the previous one. Same as Chen et (2023), we utilize 300 trajectories for training and
14

---

Learning PDEs in Unified Spectral-Physse Phyyscll Space
Table 8. Implementation detail for each PDE problem.
Model Configurations
Training Configurations
Problems
Depth
Width
Head Number
Optimizer
Scheduler
Initial Lr
Weight Decay
Epochs
Batch Size
Darcy Flow
128
8
128
A
OneCycleLR
le-3
lc-5
500
Airfoil
128
8
128
Adam
OneCycleLR
le-3
lc-5
500
4
Navier-Stokes
256
AdamW
OneCycleLR
le-3
le.5
500
Plasticity
128
8
128
A
OneCycleLR
le-3
lc-5
500
Irregular Darcy
4
64
4
64
Adam W
OneCycleLR
le-3
Ic-5
2000
16
Pipe Turbulence
AdamW
OneCycleLR
le-3
le.5
2000
Heat Transfer
64
4
64
A
OneCycleLR
le-3
lc-5
2000
16
Composite
4
64
4
64
Adam
OneCycleLR
le-3
lc-5
2000
16
Blood Flow
AdamW
OneCycleLR
le-3
le.5
2000
0,040
HPM
HPM
0.355
SpecSolver
0.4
SpecSolver
0,030
Rolative L.2
Transolver
Transolver
0.025
Rolative
0.020
0.015
0.2
0.005
Epochs
Epochs
(a) Darcy Flow
(b) Navier-Stokes Equation
Figure 5, Comparison of validation loss curve during training.
then test the models on 100 samples.
Heat Transfer. This problem about heat transfer events triggered by temperature variances the boundary. Guided by the Heat equation, the system evolves time. The neural operator strives to predict 3-dimensional temperature fields after 3 seconds given the initial boundary temperature status. The output domain is represented by triangulated meshes of 7199 nodes. The neural operators are trained on 100 data sets and evaluated on another 100 data.
Composite. This problem involves predicting deformation fields under high-temperature stimulation, a crucial factor in composite manufacturing. The trained operator anticipated to forecast the deformation field based on the input temperature field. The structure studied this paper is an air-intake component of jet composed of 8232 nodes, as referenced in (Chen al., 2023). The training involved 400 data, and the test examined 100 data.
Blood Flow. The objective is to foresee blood flow within the aorta, including inlet and 5 outlets. The flow of blood is deemed a homogeneous Newtonian fluid. The computational domain, entirely irregular, is visualized by 1656 triangle mesh nodes. Over simulated 1.21 duration, with 0.01-second temporal steps, the neural operator predicts different times' velocity fields given velocity boundaries at the inlet and pressure boundaries at the outlet. Same as (Chen et al., 2023), our experiment involves training on 400 data sets and testing on 100 data.
A.4. Additional Experimental Results
This section ablates the core modules of HPM to reveal the main factors affecting its performance.
Optimization Efficiency Comparison. In addition, we compare the validation loss curves of HPM and Transolver during training, as portrayed Figure 5. We notice that HPM reaches the same prediction accuracy as Transolver earlier, often dozens even hundreds of epochs ahead, especially in the initial and middle stages of training (the first 300 epochs). This confirms the excellent operator fitting ability of HPM benefiting from the suitable combination of spectral priors (offering fundamental function approximation basis) and point-wise adaptivity (providing efficient local feature processing).
Frequency Number. We compare the performance of HPM and SpecSolver with different frequency numbers k, as shown in Table 9. HPM consistently performs better than SpecSolver across all frequency settings. The performance gap is particularly significant under lower frequency numbers (16, 32, and 64). This leads to the conclusion that holistic spectral
15

---

Learning PDEs in Unified Spectral-Physices Space
Table 9, Performance of diffeernt frequency numbers on Darcy Flow.
Problem
Frequency Number
SpecSolver
HPM
16
1.04e-2
5.10e-3
32
8.08e-3
5.02e-3
Darcy Flow
64
6.15e-3
4.85e-3
128
5.31e-3
4.38e-3
16
1.18e-1
9.51e-2
32
1.05e-1
8.77e-2
Navier-Stokes
64
9,47e-2
7.44e-2
128
8.37e-2
7.28e-2
processing eliminates the dependeney large number of frequencies, benefiting from the point-wise adaptive mechanism. Therefore, HPM potentially performs better in some practical industry scenarios where computing high-frequency basis functions is computationally expensive.
Head Number Analysis. We evaluate the impact of varying head numbers in HPM on the Darcy Flow problem. As shown in Table 10, using multiple heads (4, 8, or 16) improves performance compared to a single head. However, HPM remains relatively insensitive the specific number of heads chosen, with all multi-heat variants achieving similar performance levels.
Table 10. Impact of head numbers.
Head Number
Relative Error
4.77e-3
2
4.79e-3
4
4.56e-3
8
4.38e-3
16
4.54e-3
Impact of Parameters. To validate that HPM's performance gains come from its architecture rather than increased parameters, we evaluate HPM with reduced depth on the Darcy Flow problem.
Table 11. Performance comparison with different parameter amounts.
Model
Layers
Parameter Amount
L2 Error
Transolver
2,835,649
5.24e-3
SpecSolver
617,665
5.31e-3
HPM
8
767,169
4.38e-3
HPM
4
408,737
4.71e-3
As shown in Table 11, HPM with only 4 layers still outperforms both Transolver (Wu et al., 2024) and SpecSolver, despite having significantly fewer parameters. This demonstrates that the performance gains come from the holistic spectral processing architecture rather than increased model capacity.
Additional Resolution Generalization Experiment. To further evaluate resolution generalization capabilities, we train models on middle resolution (111 26) and test both higher (221 x 51) and lower (45 11) resolutions the Airfoil problem.
As shown in Table 12, HPM maintains strong performance across both upsampling and downsampling scenarios. While attention-base methods like Transolver excel at training resolution, they struggle to generalize to unseen resolutions. In contrast, HPM combines high accuracy at the training resolution with robust generalization capabilities inherited from spectral methods.
16

---

Learning PDEs in Unified Spectral-Physiee Space
Table 12. Resolution generalization performance on Airfoil.
Model
Train Resolution
Higher Resolution
Lower Resolution
Transolver
4.50e-3
6.95e-2
1.22e-1
SpecSolver
5.37e-3
1.95e-2
5.84e-2
HPM
4.17e-3
1.57e-2
3.85e-2
A.5. Additional Visualization of Point-wise Frequency Preference
Visualization of Modulation Patterns for Samples with Different Resolutions. Figure 6 presents additional visualizations of holistic spectral modulation patterns on the Airfoil problem for samples with varied resolutions. We observe that the learned modulation patterns remain consistent as the domain resolution changes. This demonstrates that HPM's coupling mechanism maintains stable frequency preferences across different resolutions, contributing to its strong resolution generalization capability.
Visualization of Modulation Patterns for Different Samples. Figure shows the visualization of learned modulation patterns for different samples on Darey Flow. Despite handling different input functions, each head maintains consistent modulation strategies across samples at specific layers. For example, Head-1 in Layer-1 consistently enhances high- frequency components near boundaries, while Head-1 in Layer-2 emphasizes high frequencies in regions with sharp state changes. This indicates that the multi-head architecture HPM learns specialized modulation strategies, with different heads focusing on distinct aspects of spectral-physical coupling. These systematic patterns reveal how HPM combines domain-level spectral structure with point-wise adaptivity achieve effective operator learning.
17

---

Learning PDEs in Unified Opectral-Physitee Space
Layer
Layer
Layer
Layer
Layer
Layer
Layer
Layer
Head
Head
Head
Head
(a) Layer HPM for Airfoil (Resolation 111 x 26)
Layer
Layer
Layer
Layer
Layer
Layer
Layer
Layer
Hend
Head
Head
Head
(b) Layer HPM for Airfoil (Resolution 45 11)
Layer
Layer
Layer
Layer 4
Layer
Layer
Layer
Layer 8
Head 1
Head 2
Head's 5
Head 4
Layer
(Resolution
High Frequency
Low Frequency
Figure 6, Visualization of learned holistic spectral modulation patterns Airfoil for samples with different resolutions.
18

---

Learning PDEs in Unified Apectral-Physitee Space
Layer
Layer
Layer 3
Layer
Layer
Tayer
Layer?
Layer
Read
Eaput
Ilizk
Hoal
Tropersyy
Houl
Hoan 4
Ouiput
8 Layer IIMM for Darcy Flow (Case
Layer 1
Layer
Layer
Layer
Layer
Layer
Layer
Layer 8
Head
Lapul
Hoal
Propurrr
Hoad
Head +
Output
(b) 8 Laver HPM for Darcy Flow (Case 2)
Layer
Layer
Layer
Layer
Layer
Layer
Layer?
Layer
Hoad
Input
Head 2
Heal
Low
Head+
Onlput
Layer Darcy 3)
Figure 7, Visualization of Iearned holistic spectral modulation patterns on Darcy Flow for different samples.
19

---

Learning PDEs in Unified Epectral-Physieee Space
SpeeSolver
Transober
SpecSolver
Transolver
SpecSoler
Leeur
Resolution
Figure More visualization prediction error on diffcrent test resolutions.
20


Reviews:
[{"review_id": "2y54ib5bY6", "reviewer": "Reviewer_13wp", "summary": "The paper introduces Holistic Physics Mixer (HPM), a framework that integrates spectral transformation and data-dependent modulation (i.e. attention). HPM employs a learnable coupling mechanism that enables adaptive modulation of spectral components while preserving the advantages of spectral transformation. The numerical benchmark demonstrates the proposed model outperforms other baselines on both uniform grid and unstructured grid.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and insightful questions. They have significantly improved this work. We respond to them below.\n\n**#1 Computational cost of LBO**\n> What is the computational cost of precomputing the LBO? \n\nThe LBO eigenfunctions is computed efficiently using the robust-laplacian library, with `near-linear time scaling relative to mesh nodes` for both structured and unstructured meshes. Our empirical evaluation on a single Intel Xeon CPU demonstrates the favorable scaling:\n|Nodes Number|Computation Time (s)|\n|---|---|\n|2.5K|$0.008\\pm 6.0\\times10^{-6}$|\n|10K|$0.037\\pm 1.3\\times10^{-4}$|\n|40K|$0.177\\pm 4.3\\times10^{-4}$|\n|160K|$0.894\\pm 4.4\\times10^{-3}$|\n|640K|$4.581\\pm 9.4\\times10^{-2}$|\n|2.56M|$19.86\\pm 9.0\\times10^{-2}$|\n\nImportantly, this computation is a one-time cost per physical domain, with eigenfunctions reusable across simulations in that domain. It is negligible cost for applications with fixed geometry but varying ICs.\n\nFuture work could explore more methods (e.g., learning-based approaches, and FFT-like approaches for uniform grids) to further enhance the computational efficiency.\n\n**#2 Chosen of frequency number**\n> And how is the number of bases being chosen?\n\nThe number of frequency bases $k$ in HPM is chosen based on several considerations to balance representational capacity and fair comparison.\n\nFor most structured mesh problems, we use $k=128$, while unstructured mesh problems use $k=64$. This aligns with: (a) We align $k$ with the hidden dimension to ensure fair comparison with linear attentions, where $k$ corresponds to $Q$, $K$ dimensions. (b) We maintain consistency with spectral methods [1,2], which typically use comparable total frequency numbers. (c) Following [2,3], we make problem-specific adjustments - set $k=128$ for Navier-Stokes despite its larger hidden dimension (256) to control parameter count, and set $k=32$ for Blood Flow due to limited mesh nodes.\n\nTable 9 confirms HPM performs well across different $k$ values, demonstrating the effectiveness of HPM regardless of basis count. While more sophisticated basis selection strategies (such as AFNO [4]) could be explored in future work, current results effectively demonstrates the benefits of spectral-physical integration.\n\n**#3 Computational complexity comparison with linear attention**\n> As HPM can be viewed as an extension of linear attention, its computational complexity should also be similar to linear attention ...\n\nBelow we provide a concise analysis comparing HPM with linear attentions.\n\n(a) HPM and linear attention share similar computational complexity. Linear attention operates at $O(Nd^2)$ for features $\\mathbf{x} \\in \\mathbb{R}^{N \\times d}$, while HPM requires $O(Ndk)$ operations, where $k$ represents number of frequency basis. The main computations include computing coupling function at $O(Nk)$ cost, and forward/inverse transforms at $O(Ndk)$.\n\nSince $k$ is typically much smaller than $N$ and comparable to $d$ (e.g., $k=d=128$, $N=11271$ for Airfoil), HPM maintains efficiency like linear attentions.\n\n(b) Our measurements below confirm this analysis. On Airfoil, Galerkin attention requires 16.5ms for inference compared to HPM's 17.4ms. This minor overhead enables HPM to capture both domain-level structure and point-wise adaptivity, delivering significant advantages.\n|Model|Inference Time|Training Time / Epoch|\n|---|---|---|\n|Galerkin Attention|16.5 ms|45.25 s|\n|HPM|17.4 ms|46.11 s|\n\n**#4 In-depth discussion about AFNO**\n> Despite the author cited AFNO in the paper, I think it is worth more ...\n\nThanks for this insightful suggestion. We'll include a comprehensive discussion about AFNO in the methodology section of final version.\n\nWhile AFNO [4] also incorporates data-dependent modulation, HPM differs in several aspects: (a) Objective: AFNO aims to improve \"efficiency and robustness\" of spectral features by sparsifying the frequency modes. In contrast, HPM focuses on \"enhancing the flexibility of preset spectral features\" via point-wise modulation. (b) Methodology: AFNO selects and modifies modes in the spectral domain post-transformation, while HPM introduces spatial domain modulation pre-transformation, enabling point-wise flexibility while preserving spectral structure. (c) Application: AFNO targets computer vision tasks, whereas HPM addresses PDE challenges requiring both global spectral coherence and local adaptivity.\n\nThe spectral processing technique from AFNO could complement HPM in future work, potentially combining both benefits (AFNO's efficiency, HPM's flexibility) for diverse applications.\n\n**Reference**\n- [1] Fourier Neural Operator for Parametric Partial Differential Equations\n- [2] Learning Neural Operators on Riemannian Manifolds\n- [3] Transolver: A Fast Transformer Solver for PDEs on General Geometries\n- [4] Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers", "claims_and_evidence": "The majority of the claims are numerically tested and validated.", "methods_and_evaluation_criteria": "The paper benchmarked on benchmark problems that are widely adopted by the community, the selected baseline models are also comprehensive.", "theoretical_claims": "The theoretical result is based on the universal approximation theorem of neural operator in Kovachki et al., 2023.", "experimental_designs_or_analyses": "While the experiments provided cover quite a lot baseline and ablation studies are pretty comprehensive, the following aspects are not entirely clear to me.\n\n1.\tWhat is the computational cost of precomputing the LBO? And how is the number of bases being chosen?\n2.\tAs HPM can be viewed as an extension of linear attention, its computational complexity should also be similar to linear attention based model like Galerkin transformer/OFormer/GNot. Yet, it is shown in the paper that it is faster than Transolver (which computes attention on only a few slices) and only a small overhead over SpecSolver which does not have any linear attention part in it.", "relation_to_broader_scientific_literature": "The key contribution can be extended to other applications that involves non-Euclidean geometries like sphere, or tasks beyond PDE modelling such as image modelling.", "essential_references_not_discussed": "Despite the author cited AFNO in the paper, I think it is worth more in-depth discussion and if possible, adding a empirical comparison, since AFNO is also a model combing spectral transformation and data-dependent modulation.", "other_strengths_and_weaknesses": "I think the overall idea is straightforward but interesting and definitely a meaningful addition to the community. The strong performance of SpecSolver itself also hints the potential of this direction.", "other_comments_or_suggestions": "See Experimental Designs Or Analyses.", "questions_for_authors": "See Experimental Designs Or Analyses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qEuEq5Ipqh", "reviewer": "Reviewer_bPLw", "summary": "The paper introduces the Holistic Physics Mixer (HPM), a unified framework that leverages a holistic spectral feature space to integrate domain-level structures with point-wise physical states. The author claims that HPM achieves strong performance in scarce-data scenarios, offers resolution generalizability, and maintains computational efficiency. To validate these claims, the author conducts experiments across multiple PDE problems, comparing HPM with existing spectral and attention-based neural operators.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and insightful questions. They have significantly improved this work. We respond to them below.\n\n**#1 Discussion about other generalization scenarios**\n> Question 1: Beyond resolution generalization, would the design of HPM still be beneficial in other generalization scenarios, such as PDEs with different physical parameter settings?\n\nThank you for this insightful question. Below we provide further discussion about the generalization capability of HPM.\n\n(a) First, we wish to clarify that the strong resolution generalization and limited-data performance naturally stem from the preservation of spectral bias. As derived in Section 3.1, HPM inherits the established inductive bias of fixed neural operators (like FNO) for learning continuous operator mappings. Beyond this, the coupling mechanism allows adaptive spectral modulation based on local physical features, improving the flexibility while maintaining such inductive bias. While not specifically optimized for other generalization scenarios, this unified approach creates fundamental advantages applicable to various generalization tasks.\n\n(b) For specific generalization scenarios, as a versatile neural module, HPM could potentially be combined with dedicated techniques (such as hypernetworks or meta-learning approaches [1,2]) to enhance generalization across different physical parameters. This integration would be a promising direction for future, leveraging HPM's unified representation alongside specialized parameter generalization methods.\n\n**#2 More real-world scenarios beyond standard benchmarks**\n> Experimental design has thorough considerations that include structured and unstructured mesh PDEs, generalizability assessments in zero-shot resolution, and the impact of training data. In addition, the author considers ablation studies in spectral coupling functions. One improvement could be the inclusion of some real-world data, such as climate data or smoke plume data, to further demonstrate the robustness of HPM.\n\nThanks for your recognition for our comprehensive experiments.\n\n(a) First, We'd like to highlight that the standard benchmarks encompass diverse physical scenarios and have been widely adopted by the community [4,5]. They can effectively validate HPM's capabilities across various physical domains, from fluid dynamics to elasticity problems, demonstrating its general applicability and superior performance compared to existing methods.\n\n(b) Additionally, our current experiments already include problems derived from real-world industry scenarios. For example: (1) The Composite problem simulates deformation fields of Carbon Fiber Reinforced Polymer under high-temperature conditions, directly relevant to aerospace manufacturing of jet air intake components. (2) The Blood Flow problem models hemodynamics in the human thoracic aorta, which has significant clinical applications. These real-world examples demonstrate HPM's practical utility beyond synthetic benchmarks. \n\nWe agree that expanding to additional domains like climate data or smoke plume simulations would further validate HPM's robustness. We plan to explore more real-world applications in future work to further demonstrate the practical benefits of our unified spectral-physical approach.\n\n**Reference**\n- [1] Meta-Auto-Decoder for Solving Parametric Partial Differential Equations\n- [2] HyperDeepONet: learning operator with complex target function space using the limited resources via hypernetwork\n- [3] Learning Neural Operators on Riemannian Manifolds\n- [4] Fourier Neural Operator for Parametric Partial Differential Equations\n- [5] Transolver: A Fast Transformer Solver for PDEs on General Geometries", "claims_and_evidence": "The author makes three primary claims: (1) Strong performance and generalizability in data-scarce scenarios, attributed to the spectral priors; (2) Improved training efficiency and the ability to capture fine-scale variations due to point-wise adaptivity; (3) HPM’s learned spectral processing patterns provide insights for designing fixed spectral neural operators. Table 1 and Table 3 compare performance across structured and unstructured meshes, while data scarcity is partially explored in the Darcy Flow and Navier-Stokes experiments. Resolution generalizability is evaluated in zero-shot airfoil cases, though further discussion on generalization under diverse PDE conditions would further strengthen the claim.", "methods_and_evaluation_criteria": "The benchmark datasets include nine PDE problems, covering both structured and unstructured meshes. The author ensures a comprehensive comparison by including a wide range of existing spectral and attention-based methods. The evaluation metric used is relative L2 error, which is a standard and appropriate measurement in PDE learning, allowing for fair comparisons across different resolutions and problem settings.", "theoretical_claims": "The theoretical claims in the paper are concisely formulated, mainly presenting short representations of key properties rather than extensive derivations. Specifically, the theoretical claims related to the coupling functions are presented in Equations 11-15, defining how spectral features interact with point-wise physical states.", "experimental_designs_or_analyses": "Experimental design has thorough considerations that include structured and unstructured mesh PDEs, generalizability assessments in zero-shot resolution, and the impact of training data. In addition, the author considers ablation studies in spectral coupling functions. One improvement could be the inclusion of some real-world data, such as climate data or smoke plume data, to further demonstrate the robustness of HPM.", "relation_to_broader_scientific_literature": "The broader scientific literature relevant to HPM includes spectral-based and attention-based PDE learning methods. HPM contributes a new perspective by bridging these two categories, integrating spectral priors with point-wise adaptivity to enhance performance.", "essential_references_not_discussed": "The author discusses the majority of the essential references.", "other_strengths_and_weaknesses": "Strengths: The limitations and future work are well-discussed, and the paper flows smoothly, making readers easy to follow.", "other_comments_or_suggestions": "Comments and suggestions are addressed in the earlier sections.", "questions_for_authors": "Question 1: Beyond resolution generalization, would the design of HPM still be beneficial in other generalization scenarios, such as PDEs with different physical parameter settings?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zxEDU0vwUy", "reviewer": "Reviewer_7jxj", "summary": "Holistic Physics Mixer (HPM) unifies attention-based and spectral methods for PDE solving, combining point-level adaptability with spectral continuity constraints. This integration enables strong generalization and flexibility, surpassing existing methods in accuracy, efficiency, and zero-shot performance across diverse PDEs.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and insightful questions. They have significantly improved this work. We respond to them below.\n\n**#1 Handling triangular mesh data**\n> I do have one question to ask, is the method applicable to triangular mesh data? Since FNO is only applicable to structured grid, I am not sure if the method is the same.\n\nYes, HPM is fully applicable to triangular mesh data. Unlike FNO which is limited to structured grids, our method works effectively on both structured and unstructured meshes. The following is detailed explanation:\n\n(a) We use Laplace-Beltrami Operator (LBO) eigenfunctions [1] as our spectral basis, which can be computed on arbitrary mesh structures, including triangular meshes. This is a key advantage over traditional Fourier-based methods that require structured grids.\n\n(b) Our experiments explicitly demonstrate this capability in Section 4.3, where we evaluate HPM on five unstructured mesh problems: Irregular Darcy, Pipe Turbulence, Heat Transfer, Composite, and Blood Flow. These problems use triangular meshes with node counts ranging from 1,656 to 8,232.\n\n**Reference**\n- [1] A Laplacian for Nonmanifold Triangle Meshes", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "No such", "experimental_designs_or_analyses": "Yes, all of them.", "relation_to_broader_scientific_literature": "I think it brings a new view of blend of grid and spectral method for AI model.", "essential_references_not_discussed": "All good.", "other_strengths_and_weaknesses": "Overall, I like this paper for its bringing the new way of representative method of PDE-based solution. I do have one question to ask, is the method applicable to triangular mesh data? Since FNO is only applicable to structured grid, I am not sure if the method is the same.", "other_comments_or_suggestions": "No", "questions_for_authors": "No such", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jvUmNW4ruy", "reviewer": "Reviewer_t7vf", "summary": "The paper introduces Holistic Physics Mixer (HPM), a neural operator framework that integrates spectral-based and attention-based PDE solvers in a unified space. The authors claim that HPM inherits the generalization ability of spectral methods while maintaining the local adaptability of attention mechanisms, overcoming their respective limitations. The key contribution is a holistic spectral space that simultaneously encodes spectral domain structure and point-wise physical states, allowing adaptive modulation of spectral components. They provide a universal approximation proof, extensive empirical evaluation across structured and unstructured mesh problems, and claim superior performance over existing baselines such as FNO, Transolver, and SpecSolver. The authors also present a study on learned spectral modulation patterns, suggesting that insights from HPM can guide the design of fixed spectral methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and insightful questions. They have significantly improved this work. We respond to them below.\n`\"#1-#4\"` provide complements for several important concerns. `\"#5-#12\"` provide \"point-by-point responses\" exactly aligned with your review.\n\n**#1 Statistical Significance Analysis**\n\nTo confirm statistical significance, we perform paired t-tests across multiple runs, confirming confidence levels above 95% for all improvements.\n|Problem|Statistical Confidence (%)|\n|---|---|\n|Darcy|98.37|\n|Airfoil|96.96|\n|Navier-Stokes|99.96|\n|Plasticity|99.39|\n|Irregular Darcy|98.95|\n|Pipe Turbulence|96.95|\n|Heat Transfer|96.50|\n|Composite|95.39|\n|Blood Flow|96.35|\n\n**#2 Evaluation Metrics**\n\nWe test more metrics on the Navier-Stokes in the table below, demonstrating HPM's improvement across all evaluation criteria.\n|Model|Max Error|Boundary Error|LowFreq Error|MiddleFreq Error|HighFreq Error|\n|---|---|---|---|---|---|\n|Transolver|1.02e+0|1.17e-1|1.42e-2|1.47e-2|9.14e-3|\n|SpecSolver|1.01e+0|1.15e-1|1.12e-2|1.32e-2|8.65e-3|\n|HPM|8.24e-1|8.99e-2|9.20e-3|1.04e-2|7.12e-3|\n\n**#3 Computational Efficiency**\n\nWe provide more computational measurements, concluding that the HPM don't increase computational burden.\n|Model|Memory|Training Time / Epoch|\n|---|---|---|\n|Transolver|2.12GB|50.49s|\n|SpecSolver|1.83GB|36.48s|\n|HPM|2.13GB|46.11s|\n\n**#4 Design of $H(x,\\Phi)$**\n\nSoftmax-based modulation is a favorable approach with several practical advantages. It ensures balanced scales and lets each point adaptively focus on proper spectral components, providing an effective balance of stability and flexibility.\n\nIt is typically challenging to establish theoretical optimality for neural architecture design. Nevertheless, this doesn't diminish our core contribution, a framework that bridges spectral prior with point states. We have provided two principles for $H(x,\\Phi)$ in Section 4.5: point-wise processing for local adaptivity, and normalization for balanced information across physical points. They will provide guidance for future explorations.\n\n**#5 Claims And Evidence**\n\n(1) To statistically validate the improvements, we report stds across multiple runs per dataset, and add statistical significance in \"#1\". This fully demonstrates HPM's consistent superiority and notable gains in some problems (e.g., Navier-Stokes: 18.4%, Plasticity: 35.0%).\n\n(2) In Table 6, HPM with 4 layers outperforms both baselines, despite having fewer parameters. This confirms our gains come from architectural design, not parameter count.\n\n(3) Please see \"#4\".\n\n(4) We agree including complex PDEs is important for comprehensive evaluation. We clarify that our benchmark have included significantly nonlinear PDE, the Navier-Stokes with very low viscosity (1e-5), exhibiting turbulence features.\n\n(5) This work goes further by explicitly formulating physical-spectral structure. This theoretical foundation validates that HPM properly extends previous operator frameworks, complementing the empirical results. \n\n**#6 Methods And Evaluation**\n\n(1) Benchmarks: Please see our response \"#2\" to `Reviewer bPLw`, which addresses the same concern.\n\n(2) Baselines: To avoid misleading comparison, we have compared baselines following exact protocol as previous works.\n\n(3) Metrics: Please see \"#2\".\n\n**#7 Theoretical Claims**\n\n(1) Please see \"(5)\" in \"#5\".\n\n(2) Please see \"#4\".\n\n(3) Time-series results on Navier-Stokes confirm HPM's spectral stability with reduced error. Section 4.4 demonstrates consistent spectral patterns across resolutions, validating stability beyond training resolutions.\n\n**#8 Experimental Designs Or Analyses**\n\n(1) We have included state-of-the-art comparisons with advanced Transformer-based solvers (Transolver, GNOT, etc.).\n\n(2) The datasets are standard benchmarks used across the field, ensuring no bias favoring HPM over baselines.\n\n(3) Please see \"#3\".\n\n**#9 Broader Scientific Literature**\n\n(1) We will add more related works.\n\n**#10 Other Strengths And Weaknesses**\n\n(1) Please see \"#4\".\n\n(2) Please see \"(1)\" in \"#5\".\n\n(3) This may be due to the small fonts in Figure 2 and a lack of details. We will fix it.\n\n**#11 Other Comments Or Suggestions**\n\n(1) Please see \"#1\".\n\n(2) Please see \"#3\".\n\n(3) Yes, to isolate component contributions, we have included ablations on $k$ value (Table 9), head count (Table 10) and model depth (Table 11).\n\n**#12 Questions For Authors**\n\nQ1: HPM and PINNs solve different problems - HPM learns operator mappings across multiple PDE solutions, while PINNs encode PDEs in losses for single-instance solutions.\n\nQ2: Please see our response \"#1\" to `Reviewer bPLw`, which addresses the same concern.\n\nQ3: Please see our response \"#2\" to `Reviewer bPLw`.\n\nQ4: HPM is promising for higher-dimensional PDEs due to its linear scaling with physical points and high accuracy with fewer frequencies.\n\nQ5: The basis functions are derived directly from geometry and don't undergo training, thus initializations don't affects.", "claims_and_evidence": "**Claim**: HPM consistently outperforms state-of-the-art neural operators.\n\n**Issue**: The performance gains, while present, are marginal in several cases, and no statistical significance analysis is provided. Given the noise in PDE datasets, these improvements may not be meaningful.\nIssue: The comparison against spectral-based methods is unfair because HPM explicitly introduces additional learnable parameters and adaptive mechanisms, while the baselines operate with fixed frequency bases.\n\n**Claim**: HPM balances spectral priors with local adaptability better than previous methods.\n\n**Issue**: The effectiveness of HPM’s coupling function H(x,Φ) is only demonstrated through limited ablation studies. There is no evidence that this particular formulation is optimal beyond empirical validation. A more rigorous mathematical analysis of the inductive biases introduced by HPM is missing.\n\n**Claim**: HPM generalizes better under limited training data.\n\n**Issue**: The claim is supported by results on only a few PDE benchmarks, all of which follow the same general format. The paper does not test on significantly different physical domains (e.g., highly nonlinear PDEs, turbulent flow beyond the studied Pipe problem). Without such cases, the claim of broad generalization is overstated.\n\n**Claim**: The universal approximation theorem guarantees that HPM can learn any PDE operator.\n\n**Issue**: The theoretical proof is standard and does not differentiate HPM from prior neural operators, all of which can be formulated as integral neural operators. This claim does not justify practical superiority.", "methods_and_evaluation_criteria": "**Benchmark Selection**: The paper primarily evaluates standard PDE problems (Darcy Flow, Navier-Stokes, Airfoil, etc.), which many previous neural operators have already tackled. The results do not convincingly demonstrate that HPM generalizes beyond these well-studied cases.\n\n**Baselines**: The selection of baselines is reasonable, but some comparisons are misleading. For instance:\n\n(1) Fixed spectral neural operators are inherently at a disadvantage since they do not have additional learnable modulation mechanisms like HPM.\n\n(2) Transolver is an attention-based baseline but is not optimized for spectral constraints, making the comparison against HPM’s hybrid design somewhat artificial.\n\n**Evaluation Metrics**: The paper solely relies on Relative L2 Error, which does not fully capture solution stability, robustness, or interpretability. A more comprehensive evaluation should include energy conservation properties, long-term stability, and sensitivity to initial conditions.", "theoretical_claims": "The universal approximation theorem (Theorem A.4) is a minor extension of existing results. The proof follows directly from prior work on integral neural operators and does not provide deeper insights into the spectral-physical coupling mechanism.\n\nLack of theoretical justification for H(x,Φ): While the authors empirically compare different formulations of the coupling function H(x,Φ), there is no theoretical analysis of why Softmax-based modulation is optimal.\n\nStability of learned spectral representations is not analyzed. Given that the spectral basis dynamically adapts, it is unclear whether this process induces instability in long-term PDE integration.", "experimental_designs_or_analyses": "The comparisons with baselines appear selective. For example, some state-of-the-art methods (e.g., advanced Transformer-based PDE solvers) are omitted from the comparison tables.\n\nThe zero-shot generalization experiments are promising but fail to account for potential biases in the training data that may favor HPM’s architecture.\n\nThe computational efficiency claims are weakly supported. While inference times are reported, there is no detailed analysis of memory consumption or training efficiency.", "relation_to_broader_scientific_literature": "The paper adequately discusses related work in neural operators but overlooks some recent advances in hybrid spectral-attention architectures.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "**Strength**: (1) The idea of unifying spectral and physical spaces is novel and aligns with recent trends in physics-informed machine learning, (2) The proposed method is conceptually simple and easy to integrate into existing architectures, (3) The experiments cover a diverse range of PDE problems, demonstrating the method’s versatility.\n\n**Weakness**:  The lack of rigorous theoretical justification weakens the impact of the proposed approach, (2) The empirical results, while promising, do not convincingly demonstrate superiority over all existing methods, (3) The clarity of the paper is suboptimal, with several technical details buried in the appendix and figures that are difficult to interpret without extensive cross-referencing.", "other_comments_or_suggestions": "The authors should provide statistical significance testing for their experimental results.\n\nA more detailed discussion of computational complexity would improve the credibility of the efficiency claims.\n\nAdditional ablation studies are needed to isolate the contributions of different architectural components.", "questions_for_authors": "(1) How does HPM compare to PINNs and other physics-informed architectures for PDE solving?\n\n(2) Can you provide a formal theoretical justification for why the proposed coupling function improves generalization?\n\n(3) Have you tested HPM on real-world PDE applications beyond the benchmark datasets?\n\n(4) How does HPM scale with increasing problem complexity (e.g., higher-dimensional PDEs)?\n\n(5) Could you clarify how the spectral basis functions are initialized and whether different choices impact performance?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces a framework to integrate spectral and local information within a unified space for neural operators (in emulating systems such as PDEs). Their framework enforces domain-level priors (through spectral transforms) and point-wise flexibility that neither spectral-only models nor attention-only models offer.

Strengths identified:
* Simple idea that shows good performance advantages on many PDE benchmarks (uniform and non-uniform grids) and can be extended to tasks beyond PDE modeling (images) and problems with non-euclidean geometries
* Well written paper with comprehensive evaluations/validations on benchmark PDEs

Weaknesses:
* The main weakness was limited evaluations on real-world scenarios. 
* While the idea is simple/novel (which is a key strength), some performance advantages might be minimal owing to simpler benchmarks (2D PDEs etc). However, the diversity of PDE tasks looked at was identified as a strength. Other non-PDE domains can also benefit and the paper could have benefitted from benchmarks in the imaging community. These limitations are acknowledged well in the paper.

Overall, the reviewers found sufficient merit to converge to acceptance of the paper. Some concerns/clarifications on computational cost were raised and answered during the rebuttal.
Decision: Accept (poster)