F-Adapter: Frequency-Adaptive Parameter-Efficient Fine-Tuning in Scientific Machine Learning
Authors: Hangwei Zhang
KangChun
Yan Wang
Difan Zou
Venue: NeurIPS 2025 poster ()
Keywords: Scientifc Machine Learning
Primary area: machine_learning_for_sciences
TL;DR: This work is the first to explore PEFT for scientific machine-learning and establishes F-Adapter as an effective paradigm for this domain.

Abstract: Parameter-efficient fine-tuning (PEFT) powerful pre-trained models for complex downstream tasks has proven effective in vision and language processing, yet this paradigm remains unexplored in scientific machine learning, where the objective is to model complex physical systems. We conduct the first systematic study of PEFT for pre-trained Large Operator Models (LOMs) obtained by scaling variants of Fourier Neural Operator. We observe that the widely used Low-Rank Adaptation (LoRA) yields markedly poorer performance on LOMs than Adapter tuning.  We further theoretically establish that stacked LoRA incurs a depth-amplified lower bound on approximation error within Fourier layers, whereas adapters retain universal approximation capacity and, by concentrating parameters on energy-dominant low-frequency modes, attain exponentially decaying error with bottleneck width in the Fourier domain. Motivated by the robust empirical gains of adapters and by our theoretical characterization of PDE solutions as spectrally sparse, we introduce Frequency-Adaptive Adapter (F-Adapter). F-Adapter allocates adapter capacity based on spectral complexity, assigning higher-dimension modules to low-frequency components and lower-dimension modules to high-frequency components. Our F-Adapters establish state-of-the-art results on multiple challenging 3D Navier–Stokes benchmarks, markedly enhancing both generalization and spectral fidelity over LoRA and other PEFT techniques commonly used in LLMs. To the best of our knowledge, this work is the first to explore PEFT for scientific machine-learning and establishes F-Adapter as an effective paradigm for this domain. The code will be made publicly available upon acceptance.

F-Adapter: Frequency-Adaptive Parameter-Efficiet Fine-Tuning in Scientific Machine Learning
Hangwei Zhangl223
Chun Kang Yan Wang Difan Zou
School of Computing and Data Science, The University of Hong Kong
Institute for Al Industry Research, Tsinghua University
Beihang University
(hangweizhang, kangchuu))ebaaa edu. bangyan@air.tsinnhuun tsinghua. edu, en, dzouêcs hku hk
Abstract
Parameter-efficient fine-tuning (PEFT) of powerful pre-trained models for complex downstream tasks has proven effective vision and language processing, yet this paradigm remains unexplored in scientific machine learning, where the objective is to model complex physical systems. We conduct the first systematic study PEFT for pre-trained Large Operator Models (LOMs) obtained by scaling variants of Fourier Neural Operator. First, we observe that the widely used Low-Rank Adaptation (LoRA) yields markedly poorer performance on LOMs than Adapter tuning. Then, we further theoretically establish that stacked LoRA incurs depth- amplified lower bound on approximation error within Fourier layers, whereas adapters retain universal approximation capacity and, by concentrating parameters on energy-domingg low-frequency modes, attain exponentially decaying error with bottleneck width in the Fourier domain. Motivated by the robust empirical gains of adapters and by our theoretical characterization of PDE solutions as spectrally sparse, we introduce Frequency-Adaptive Adapter (F-Adapter). F- Adapter allocates adapter capacity based on spectral complexity, assigning higher-dimension modules to low-frequency components and lower-dimension modules to high-frequenee components. Our F-Adapters establish state-of-the-ar (SOTA) results multiple challenging 3D enhancing both generalization and spectral fidelity over LoRA and other PEFT techniques commonly used in LLMs. To the best of our knowledge, this work is the first to explore PEFT for scientific machine-learning and establishes F-Adapter effective paradigm for this domain. The code is publicly available at here
benchmarks, markedly
Introduction
Learning solution operators for partial differential equations (PDEs) is a fundamental challenge in scientific machine learning (SciML). Among the most promising approaches operator- Dpernnnn architectures, particularly the Fourier Neural Operator (FNO) and its variants 223 52
49 28
12 (44) These models leverage mesh-independent spectral convolutions efficiently capture fine-
scale dynamics in the frequency domain [I] 38 55) enabling orders-of-magnituee faster inference
compared traditional numerical solvers 37
Recently, the field has seen the rise of Large Operator Models (LOMs) [61]. which scale these architectures and employ large-scale pre-training on diverse datasets, unlocking remarkable generalization capabilities for complex downstream tasks.
When adapting large pretrained models to downstream tasks, parameter-efficient fine-tuning (PEFT) has emerged powerful strategy, offering minimal computational and storage overhead [27 56) 29 21 13] Unlike full-model fine-tuning, PEFT techniques fine-tune only a small subset of trainable parameters. This approach preserves the benefits of pretraining while enabling rapid deployment
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

across tasks and domains. This paradigm has proven highly effective in natural language processing
(NLP) [59 26 36] and computer vision (CV) 32 5 19,,
where large foundation models dominate
and efficient task adaptation is critical for scalability.
However, despite its effectiveness in NL.P and CV, its potential within SciML remains unexplored. Although a few recent studies employ LoRA-style Physics-Informed Neural Networks (PINNs) [40] to build surrogates for parameterized PDEs 77 51). they train modest networks from scratch on small and single-type equation datasets. A systematic PEFT study for pre-trained LOMs therefore remains to be established. Physical systems governed by PDEs pose qualitatively different challenges: their solution manifolds exhibit broadband. muscade-coupler spectra and reside in high dimensional continuous domains [35] (4 [2]]. These distinctions prompt our central question: Can PEFT be adapted to LOMs in SciML so that it explicitly respects the frequency -adattive structure and physics- based priors inherent to PDE solution spaces?
In this work, we present the first systematic study of PEFT for pretrained LOMs. Through a combination of empirical analysis and theoretical investigation, we identify a fundamental limitation in the widely used LoRA approach [18]: its rank-constrained linear updates create a depth-amplified spectral error floor when applied to Fourier-based operator architectures. On the other hand, we show that replacing these linear updates with lightweight non-linear adapters, implemented as residual two-layer MLP bottlenecks, can lead to surprisingly effective fine-tuning for LOMs. We further demonstrate that this approach can maintain universal approximation capabilities while strategically concentrating model capacity on the energy poeggy-dmminatt spectral subspace, thus enabling parameter- efficient adaptation without sacrificing spectral fidelity 17).
Building on these insights, we propose Trequency-Adaptiin Adapters (F-Adapters), novel PEFT architecture for LOMs that allocates adapter capacity according to spectral complexity. Concretely, the Fourier Layer in LOMs bins its Fourier coefficients into different spaced radial shells, creating disjoint frequency bands for capacity-aware F-Adapter assignment. Specifically, F-Adapters assign larger bottleneck dimensions to oowffreueency bands which typically contain most of the signal energy and govern long-rang, physical interactions, and smaller dimensions to figh-frequency bands that often sparse and susceptible to numerical noise. We summarize our main contributions follows:
We empirically and theoretically establish that residual two-layer MLP Adapters significantly outperform LoRA for fine-tuning in scientific machine learning. Next, we rigorously analyze the energy distribution of PDE solutions in the Fourier domain. All the resulting theory guides the design of our architectural innovations.
We devise a Prequency-Adaptiin Adapter (F-Adapter) that allocates parameters in proportion to the spectral energy profile of PDE operator solutions, which in turn couples model capacity to tesk-relevant frequencies.
We achieve the SOTA performance on multiple challenging 3D Navier-Stokes forecasting benchmarks, which surpasses LoRA and prior PEFT baselines in I.2RE accuracy with only less than 2% of backbone parameters tuned. Comprehensive ablation studies and direct comparisons with other Fourier domain adapter designs confirm the superior effectiveness of F-Adapters.
2 Related Works
Parameter-Efficient Fine-Tuning, PEFT adapts frozen backbones through minimal trainable components. Prompt Tuning learns compact "soft" prompt that is prepended the input while keeping all model weights fixed (20). Adapter tuning inserts narrow bottleneck MLPs between Transformer sub-layers so that only these adapters are updated [17]. FiLM Adapter extends adapter tuning by treating channel-wise FiLM layers as adapters and updating only their (7, 3) parameters [3). LoRA injects pair of trainable low-rank matrices whose product is added to each frozen weight tensor [18]. AdaLoRA allocates the low-rank budget across layers dynamically according to data-drived importance scores [58]. HydraLoRA shares one down- lown-prooection across multiple LoRA heads, enlarging expressiveness without extra memory [48.. RandLoRA couples fixed random bases with trainable diagonal scalings to approximate full-rank updates at constant parameter cost [0] SVFT updates each tensor via sparse mixture of its own singular-vector outer products, training only the corresponding coefficients 25). Concurrently, Loeschcke et al. [3I] uses Tucker-factoriza low-rank updates, preserving cross-mode structure and outperforming unfolding-

---

based LoRA. Together, these methods illustrate how structural priors can drastically reduce trainable parameters while retaining tine-tuning flexibility.
Pretrained Large Operator Models for PDE Solving. A rapidly growing body of work now treats LOMs foundation models. These models first pretrained heterogeneous collections of partial differential equations and are later adapted to new physical regimes. The pioneering study of Subramanian et al. [45] shows that FNO trained on eight disparate PDES scales predictably and slashes downstream data needs by orders of magnitude. Building on this, MPP adds an autoregressive transformer over ten systems for strong zero-shot transfer [3], Poseidon cuts cost via multiscale conditioning [16]. UPS employs cross-modal adaptation for data- efficient generalisation 42). and
CoDA-NO introduces codomain-awes -wwrr attention for few-shot multiphysies tasks [39]. PreLowD [5] and OmniArch [6] demonstrate the powerful generalisation capabilities of large-scale operator models achieved by moving spatial field values into the frequency domain. DPOT further scales to 1B parameters with Fourier-denoising pretraining. achieving SOTA on 10+ datasets [14]. Collectively, these LOMs demonstrate that heterogeneous PDE pretraining provides a powerful tool for scientific machine learning.
3 Behavior of Fine- Tane-TTnnngga Methods for Large Operator Models
In this section, we examine the performance of fine-tuning techniques applied to pretrained Large Operator Models (LOMs). First, we conduct an empirical comparison of some typical fine-tuning methods, followed by a theoretical interpretation of the results. Next, we delve into a deeper theoretical analysis to illustrate how the information within the solutions is distributed across Fourier spaces. This analysis offers valuable insights that can inform the development of more effective PEFT methods for LOMs.
3.1 Empirical Comparions between Different Fine- Tuning Methods for LOMs
Task and experiment setup. To evaluate fine-tuning methods for LOMs, we focus on the three- dimensional forecasting problem, challenging scientific machine learning task characterized by highly nonlinear nigh-dimensional solution manifold and unstable truncation errors. Specifically, we use DPOT-H as the pretrained model, a IB parameter backbone that is currently the largest publicly available LOM 14. We fine-tune this model on two 3D Navier-Stokes datasets from PDEBench [46] with standard parameter-efficient methods, including LoRA and bottleneck adapters (implementation details appear in Appendix(N) (A.. These datasets are configured with random initial conditions at M = 1.0 and M 0.1.
Acapter
shoterded oil
We primarily integrate the PEFT modules into the Fourier-Attention layers, which form the computational core of the model, contain the majority of its parameters, and dominate the overall computational cost. During fine-tuning, we use the AdamW optimizer and train the model for 500 epochs with different efficiency levels (e.g., ranks for LoRA and bottleneck dimensions for Adapter). The performance is evaluated on the test set using the L2 relative error (L2RE), a standard metric in operator learning
somtc An
2:14
Opooch
Epech
Figure 1: Convergence comparison of LoRA and bottleneck Adapter. Adapter not only starts with lower loss but also reaches lower steady-state value, indicating faster and more stable convergence.
223. All experiments are conducted on single NVIDIA A800 80 GB GPU. Complete experimental details are provided in Appendix C
Experimental results. Building on the experimental setup outlined earlier, we conducted a comprehensive empirical comparison of the performance of the LoRA and Adapter methods across varying ranks and bottleneck dimensions. The results are summarized in Figure [ and Table [ where several key insights can be revealed. First, despite the widespread adoption of the LoRA method in many large language model (L..M) tasks, it demonstrates significantly poorer performance in fine- tuning LOMs (low-rank models) and does not benefit from increasing the ranks. Second, the Adapter method proves highly effective in fine-tuning LOMs (original models). The performance continues to
3

---

improve as the bottleneck width increases, although larger width introduces some overfitting. These findings highlight the distinct suitability of each method depending on the specific model architecture and fine-tuning objectives. This suggests that the Adapter method may be more appropriate choice for fine-tuning LOMs modeling physical systems.
Scheme
% Params
Mem (GB)
L2RE (M=1.0)
L2RE (M=0.1)
LoRA (r=4)
0.17%
12.58
0.6413
0.6218
LoRA (r=8)
0.34%
12.65
0.6345
0.6129
LoRA (r= 16)
0.69%
12.78
0.6427
0.6147
LoRA (r=32)
1.37%
15.85
0.6395
0.6211
Adapter (d-4)
0.59%
15.82
0.6169
0.5063
Adapter (d=8)
1.16%
15.85
0.5496
0.4893
Adapter (d=16)
2.30%
15.89
0.5227
0.4539
Adapter (d=32)
4.59%
15.98
0.5134
0.4570
Table 1: Comparison on LoRAs with different rank and Adapters with different bottleneck dimension.
3.2 Theoretical and Empirical Explanations on the Benefit of Adapter Methods
In this part, further provide explanations about why adapter lead to substantially better performance than LoRA in fine-tuning LOMs.
We mainly consider the comparison between the Block-wise LoRA and two-layer MLP Adapter within the Fourier layers of LOMs. In particular, block-wise LoRA applies the multiple low-rank adaption for different blocks of the target parameters separately. For adapter model, we model the adapter in Fourier blocks as a two-layer MLP g : CN +C g U a(V 8 + + with weights VE CCC.N,
U E C. xm, biases be Cm", cECN, and non-linearity (-)).
Then, we first deliver the following proposition that characterizes the approximation error for the block-wise LoRA method with rank r. Proposition 3.1 Block- wise LoRA lower bound ) Let AW, g blockdiag AW(),... ...... AW(K) be the block-wise model parameter updates and BA = blockdiag B(1) (1)) ,(),..,, B(K), ((K)) be the
block-wise low-rank approximation, where Blk) E dxxr ((k) E Crxd, Then, for any input x, the approximation error for block-wise LoRA satisfies
Th (xT, 1/2
[Aw,, BAAx||
ΣΣ R=li
)
(1)
In particular, the worst-case operator-norm error obeys
sup | - BA)alla Z (Ar,)(WW,)
||
Interpretation. Even if each block is well approximated in isolation, the worst-case LoRA error across the entire stack is still lower-bounded by the (Kr+1)-th singular value of the global matrix, revealing an intrinsic additive bottleneck as depth K grows.
Adapters, on the other hand, introduce non non-linear, width-controlled bottlenecks after the Fourier transform. Rather than adjusting the pre-trained model's parameters, carries out fine-tuning in separate representation space. Because the underlying two-layer MLP satisfies the universal- approximation theorem, is, in principle, capable of representing any measurable function. In the Fourier layers of large operator models (L.OMs), however, the practical rate at which this universal approximation is achieved is dictated by the spectral frequency content of the target update. We first introduce the following notation for adapters: Let x gd Xdy and define the unitary discrete Fourier transform and its inverse F IRd xd2 Xda + N F-1, CN + IRd xds xds, N = d d2d3. For multi-index & E Z3 we abbreviate (k) = / + | 7. An Adapter's Fourier coefficients are defined by gk (2mjd Sru g(x) torus.
iik dz, where d1d2d3, E 2d and Td is the d-dimensional
Proposition 3.2 (Frequency-seleectve approximation of adapters). Let [9x| S (kk) with o > For any 0 there exist frequency truncation radius K and adapter bottleneck width E N such that the Fourier-domate adapter ğ obeys
||l||
| (F'(())
FF -F <E
||ell (k²-o) +(K2 -
(3)

---

Implications. All proofs are given in Appendix B aequunccy-agoosee linear wlock-wise LoRA is bottle-necked by the depth-dependent lower bound in Proposition B.J] whereas Adapters in Fourier blocks concentrate parameters on the energy-dominn derggydomiiniiing low-frequency subspace and exploit the exponential accuracy of Proposition 3.2 This frequency-selective compression accounts for the superior balance between predictive accuracy and parameter count observed in all the experiments on DPOT [4], a kind of FNO-based LOM.
Experimental Verification. We explore whether the performance degradation caused by truncating full-rank updates in the parameter space can be effectively mitigated by employing lightweight, low-rank, and non-lineat adapter that operates directly in the representation space. Specifically, we aim determine if this adapter can accurately recover the functional shift that occurs due to such truncation. For more detailed experimental information, please refer to the Appendix C..
For the first Fourier- Attention block in DPOT we harvest the real Fourier activations H RNxd and their target outputs Y = H AWW", where AW denotes the exact full-rank weight update. After a 90/10 % train-validation split we benchmark two surrogates: (i) two- layer MLP adapter fMLP: : H + Y whose hidden width is m, and (ii) low-
RMSE Parameter Value
RMSE Parameter Value
0:30
Lowitart 1086-021
022
Looe Punth Tra YoAMM
925
0.20
0.333
0.10
0.
40
25 Paramataa Value
100
322
23
40
35
1.5
Paaramater Valte
Figure 2: RMSE versus parameter (bottleneck dimension m for Adapter and rank for Truncation) budget for the two-layer MLP adapter (yellow) and the low-rank truncation baseline (orange). Left: transonic dataset (M 1.0). Right: low-Mach dataset (M =
rank truncation baseline obtained by replacing AW with its optimal rank-r SVD approximation, which corresponds to an idealized LoRA module. Both models are evaluated by the root-mean-square error 0.1). RMSE (|YI-111(())
2 on the held-out set.
Figure shows the RMSE achieved by two-layer MLP adapter against the best rank-r SVD truncation of AW (idealized LoRA) under identical parameter budgets and supplementary experimental results can be found at Appendix C.10 The advantage for Adapters is already noticeable at extremely small budgets (m, < 16) and widens as capacity grows. In the transonic case (M = 1.0) the adapter with only m 64 hidden units halves the error a rank- 128 truncation.
3.3 Spectral Energy Concentration in Low-Frequency Bands
DPST Med Rand Drop High
DPOT Tubb Drop High Average vs. Cut
To systematically gauge the relative importance of different spectral bands to predictive accuracy, we conduct a
cude
12 A2Awccaae Cut
1.
Aaerraee Emnr
deg Dark
12g toa)
spectral drop-high experiment on
fully fine-tuned DPOT model evaluated on the diree-dimensional Navier-Stokes benchmarks with random initial conditions at M 0.1 and M 1.0
0.00
03
DPOT Mud.1 Rand Drop-Hight Average Error vs. CLe Off Index
and Turbulence initial conditions at M- 1.0.
Prg Br Test
4ecrooe
A fixed mini-batch taken from the test set is transformed to Fourier space and
its spectrum is uniformly partitioned into N, non-overlapping frequency bands. For each cut-off index & E (O...., N₂ 17 we zero out every coefficient that lies in a band whose index is > k perform an inverse FFT, and pass the filtered representation through the network. The mean I.2 relative error between the network output and the ground-truth target is then recorded. Sweeping k from low to high therefore reveals how progressively discarding higher- frequency content influences the model's 'ss accuracy.
Figure 3: Mean L2 error versus the cut- band index & for Rand M 1.0 (upper left), Turb M 1.0 (upper right), and Rand M- = 0.1 (bottom).
5

---

The resulting error curves for all datasets are presented in Figure As the cut-off index & increases, meaning progressively higher spectral bands are excised, the average Lg error falls steeply at first and then flattens, eventually becoming almost insensitive to the removal of additional bands. The error curves suggest that most predictive power captured once a relatively small set of low-frequency bands is retained; beyond this point, excising additional high-frequency bands yields only marginal further degradation. This behaviour is consistent with the ppectral-truncation heuristics commonly used in FNO-style models [38 54
To place the empirical finding on firmer ground, we extend classical harmonic analysis time- continuous PDE solutions and, using Proposition 3.3 show that the cumulative energy of high- frequency modes decays polynomially with the cut-off radius, which implies that low-frequency modes carry the dominant share of the energy. The proof is provided in Appendix B
-rrequeig- Energy Split for PDE Solution). Let >>>>> & and Proposition 3.3 (Quantitative Low-/High Frequency suppose f E C( (T))) suprejo.r] [t)(()) I M. Let f(t, x) = ΣkEZ f(t, k) e dkr then each k + 0,
IF(1, - M (1 + |
anr every integer K 22 1 there exists C = C(d, s) such that
k)t S KCM |
Σ
= OkK-220) |A
Σ
(5)
Proposition 3.3 shows that, in Fourier space, PDE solutions concentrate most of their energy in relatively low-frequency modes while the high-frequency tail decays as O(Kd-2s).
Taken together, these findings indicate that low- frequenquccy bands predominantly convey the global, energy-rich structure of fluid flows, arguing for higher-capacity adapters in that regime. Conversely, high-frequency bands are comparatively sparse and noise-prone; representing them with lightweight, low-rank transformations not only suffices for detail reconstruction but also serves as effective spectral regularizer. Leveraging the complementary information carried by different frequency bands more effectively therefore represents a critical avenue for carrying out PEFT.
4 Methodology
For parameter-efffcieet fine-tuning, retrofit each Fourier-domain mixing layer of the LOM with Frequency-Adaptive Adapters [F-Adapters)-bbttleeee band according to governed formula (Eq. (6)) The design is model-agnostie: it can be plugged into any FFT-based layer without altering the host architecture's 's training recipe. The overall F-Adapter pipeline is illustrated in Figure
MLPs whose width varies per frequency
Fourier Representation. Given an input tensor BxCxxxxxxx we first perform a real 3-D FFT: & = rFFTN (F. e CB MixxMMXMaXX where M +1 for real transforms. The last dimension (C' channels) is split into K non-overlapping blocks with equal width d = C/K.
Band Partitioning. We retain only the lowest M 111mmmn;M (M, Mz) spatial modes and partition them into contiguous frequency bands 0 = by < by < <.. < by = M, B, = {bb-1,.... by 1}, so that every block- band slice can be processed independently. For most cases we set B = 4 and choose by = M].
Band-Specific Bottleneck Allocation. Let fo 1 [bb (/bbbiibb be the centre frequency of band b. We allocate a bottleneck width ra according to
Tb Tmin + (Tmax Tmin )
M 1n }"
where Tmins Tmax and p are while higher bands shrink toward "min-
controling the curvature. Lower bands receive wider Tb-
F-Adapter Micro-Architecture In the DPOT-H [14] backbone, for each block E [K] and band be [B] in Fourier Attention Layer we attach three tiny adapters:
A
Rd
RRd
mid,
dd
Ag.: Rd +R

---

Algorithm F-Adapter PEFT Forward Pass in DPOT's 4
Fourier Attention Layer
X + rFFTN((x
Step I: FFT
X - reshape X, (K, dK, (M,
Step 2: reshape channels/modes b Step band loop
for - tok do
for 6- toB do
(i:j) <- band indices(()
D. 3-a: compute slice indices
2 f x/: i:j, i:j, 0:M,, k]
b 3-b: extract complex slice
ZR - R(z): 2ς (z)
& 3-c: split real/imag
ZR < ( ; A (ZT)
D. 3-d: input adapter
(UR, un) 4- -ourierrmixx(ze
W (1)
3-e: first Fourier mixing 3-f: activation
UR < -GELU(UR): : u1 f GELU(U)
un < Amid (uR)) (uR):
u1 Amid (ur)
D 3-g: mid adapter
(VR. v1) < fourrrirrrrr ponrier_mi((ure ur (
3-h: second Fourier mixing 3-i: output adapter
VR. + Agut Ag (VR): VI 4 (v))
x[: å:j, itj, 0:M,, VR +ivi
D 3-j: scatter back b Step 4: IFFT
x < iRFFTN(X)
return x'
Step 5: residual
Adapter Architecture
down
up
Frequency Band Division
Fourier Layer
Adapter
Pre-trained Modules
Adapter
FFT
IFFT
Adapler
Adapter
Figure 4: Pipeline for inserting Frequency-Adaptic Adapters (F-Adapters) between consecutive pre-trained Fourier sub-modules in a Fourier layer in LOMs.
where he is the number of retained temporal modes. Each adapter implements the canonical bottleneck residuum
Zdown
W z + down
Zast [Zdown);
Zup
W Zuct + b,"
+ Sb Zup
(8)
with = GELU and is scalar. The matrices have shapes W W < w E Pxn, where DE (d, d h depending on the adapter stage. The Parameter-Efffiientt Fine-Tuning forward pass of the F-Adapter within the DPOT backhone is outlined in Algorithm [
Initialization and Training Details. We adopt zero-initialization for every W and b that, at the start of fine-tuning, the adapted path is an exact identity and does not perturb the pre-trained backbone. Down- proneprojcction weights are initialized with Kaiming-uniform initialization. All spectral kernels W (2)) follow the scale definition dhh from the Fourier Attention Later in DPOT.
Computational Cost. Let N₂ BK denote the total number of block- slices in one Fourier- Attention layer. The additional parameters introduced by the F-Adapters can be written as
B Σ
B Σ
AdAAter
(2dr
2dr + rb) x K (2+ h2) = (24 + K2
Tb- hmll
(9)
Defining the average bottleneck width f = EB Tb. this simplifies to
O Addateee (2d + 1) B K h4) f- O(dBKh.r)
(10)
For typical settings, this overhead remains below 2% of the host model's total parameter count.
7

---

Rand
Turbulence
Scheme
% Params
Mem (GB)
12RE (M=1.0)
L2RE (M=0.1)
Mem (GB)
L2RE
AdaLoRA [58]
0.69%
15,83
0.6726
0,6275
27.08
0.6795
HydraLoRA 48
0,85%
22.14
0.6333
0,6164
33.85
0.6888
Prompt Tuning
20
1.03%
19.82
0.6378
0.6127
23.37
0.6651
Vanilla Adapter
17
1.16%
15.85
0,5496
0,4893
25.41
0,4696
FiLM Adapter
43
1.30%
15.85
0,5655
0,5054
26.76
0,4987
RandLoRA [
1.36%
15.86
0.6370
0,6125
24.69
0.6893
LoRA 18
1.37%
15.85
0.6395
0.6211
25.03
0.6842
F-Adapter (Ours)
1.91%
15.88
0.5329
0.4639
26.90
0.4523
SVFT 25
2.31%
15.91
0.6375
0.5984
23.36
0.6655
Full Fine- Tuning
100.00%
25.27
0.5391
0.4002
37.06
0.2382
Table 2: PEFT results on the IB IB-parameter DPOT-H backbone for 3D Navier-Stokes forecasting.
Webcyy Pollt Compu to fp
Wrady feld Tauch
wiy hold Carpp sane
SPPPEE
Figure 5: Side-by-side velocity field comparisons for Turbulence epoch 500. From left right: Vanilla Adapter, F-Adapter (Ours), and LoRA. Each panel shows ground-truth compared with prediction.
Plug-and-Play Deployment. Figure depicts the drop-in procedure that enables an F-Adapter to be grafted onto any FFT-based Fourier layer appearing in FNO-style Large Operator Models (LOMs) without disturbing the host training recipe. First, the input field x Bx x xL is mapped the spectral domain pre-trained and frozen Fourier-mixing kernels of the backbone, upon which we retain the lowest M spatial modes and slice them into contiguous radial frequency bands B,..... BB according to 0 by < b1 <... < bg = M. For every block & E [K] and band b [B] we attach a bottleneck MLP whose width rb computed by Eq. ( Each adapter performs the nown-activation. transformation defined in Eq. (8). writes the adapted coefficients back to their original spectral locations, and leaves all surrounding FFT logic untouched. Finally, the spectrum is converted back to physical space via x' = iRFFFNN the procedure relies solely on the presence of an FFT/FFFT pair, applies verbatim to every LOM that scales out of the FNO family. If a Fourier layer in LOM contains multiple pre-trained sub-modules that require fine-tuning, one can interleave F- T-Adapters between those sub-modules following the workflow depicted in Figure E Algorithm Toffers representative instantiation of such a
rFFTN((x). The resulting complex tensor is then passed through the
(TFFFNN((uapeee)) (adapted) and added residually to the original signal. Because
F-Adapter deployment.
5 Experiments
5.1 Main Experiments
Our principal comparison of the proposed F-Adapter against suite of PEFT approaches widely used for LLMs follows the experimental protocol described in Section 3.1 In the 3D Navier-Stokes forecasting task in Table our method lowers L2RE by from 16.7 % 25,4 % while using nearly the same GPU memory as standard LoRA and its variants. Relative to the strongest baseline (Vanilla Adapter), F-Adapter improves accuracy by from 3.0 5.2 % with a comparable parameter budget, which supports the value of
capacity allocation. Side bade-by-side slice plots in Figure 5 of velocity magnitude show that F-Adapter retains filamentary vortical structures and reproduces both low- and high-energy regions with small local amplitude deviations. LoRA yields coarse block-like patches with muted intensities and loses fine-scale features, which consistent with its collapsed spectrum.

---

Table further reports results for PEFT methods on the 2D shallow-ward equations (SWE-2D) and the 3D maggetohyydrodnnmmic task (MHD-3D), two settings that demand high-frequency fidelity and broad spectral coverage. For MHDD-DD, we follow the data processing protocol of Du et al. 9 but train on only 24 trajectories, creating a severe data-scarcity scenario. The results Table show that even under figh-frequency regimes and limited data, F-Adapter maintains substantially higher accuracy, whereas alternative PEFT schemes struggle to adapt. These findings indicate that, instead of overlooking high-frequency content, F-Adapter partitions the spectrum so that all frequency bands are handled more efficiently and more effectively.
Full fine-tuning unsurprisingly achieves the best raw accuracy in most cases, but it requires x50 more trainable parameters and -1.4x the memory during training. Hence, F-Adapter strikes accuracy-efficinnyy
favourable accuracy- efficiency trade-off and represents a practical alternative when computational resources or deployment budgets are constrained.
These findings substantiate our central claim: explicitly matching adapter capacity to the spectral characteristics of scientific operators is critical for effective and economical adaptation, whereas LoRA or prompt-based methods designed for language do not readily transfer to the SciML regime. For more Spectral analysis, please refer Appendix C.12
5.2 Ablation Studies
Ablation on the Effect of the Dimension-Frequenee Schedule. To isolate the effect of the dimension fimeueion-frequencyy schedule, we introduce an F-Imverse-Adapter for ablation study. In the F-Inverse-Adaptee we reverse the capacity allocation, giving higher-frequency bands larger bottleneck dimensions while matching F-Adapter's 's overall parameter count and memory consumption.
Concretely, its bottleneck size is number of modes and p is the hyperparameter exponent.
Tb = Tmin + (Tmax Tmin)
fu 3
P
where M is the total
The F-Adapter, which assigns larger dimensions to low-frequency bands and smaller ones to high-frequency bands, outperforms both the vanilla Adapter and the reversed F-Inverse- Adapter in T This confirms that our dimension dimension-frequency schedule is both reasonable and effective.
SWE-2D
MHD-3D
Scheme on DPOT
L2RE
% Param
L2RE
% Param
AdRL 58
0.1061
0.70%
1.0022
0.69%
HydraLoRA 48
0.0956
0.88%
0.9440
0.85%
Prompt Tuning
20
0.1050
0.11%
0.9950
1.03%
Vanilla Adapter
17
0.0902
0.48%
0.7226
1.16%
FiLM Adapter 43
0.0162
0.57%
0.7593
1.30%
RandLoRA 01
0.1568
1.05%
0.9800
1.36%
A Ablation on Different Types of Adapters for Fourier Domain. We explore alternative strategies to address the challenges of applying Adapters in the Fourier domain. Motivated by Xiao et al. [52]. which amortizes fourier-kernel parameters with KAN 30 accommodate arbitrary fighfrreueecy modes, we propose the Fourier Adapter. We substitute the adapter's 's down-projection and up- ppopreeection layers with FourierKAN layer [53], which parameterizes edge functions truncated Fourier series, and integrate this module directly into the Fourier domain. Motivated by Zhang et al. (57] who applied Chebyshev polynomial bases within KANs to enhance PINNs for PDE solution, we substitute the adapter's linear layers with Chebyshey-based KAN modules to form the Chebyshev Adapter. Motivated by Tripura and Chakraborty (50). who incorporate wavelet transforms into operator learning, and by Zhao et al. [60], who employ the learnable wavelet-based activation function WaveAct for solving PDEs, we introduce the WaveAct Adapter-an adapter module that uses WaveAct as its nonlinear activation throughout the down- and up- up-projection layers. Architectural details for all the aforementioned Fourier-domaii adapter improvements can be found in Appendix C.B.
LoRA 18
0.1081
1.40%
0.9845
1.37%
F-Adapter (Ours)
0.0116
1.24%
0.6341
1.91%
SVFT
0.0975
0.84%
1.0004
2.31%
Full Fine- Tuninnunig
0.0023
100%
0.4190
100%
Table 3: PEFT results the 1B-parameter DPOT-H backbone for 2D shallow-water equations (SWE-2D) and the 3D magnetohydrodynamic (MHD-3D) in data scarcity conditions.
Comparing the results in Table 4 with Table we can conclude that most of the aforementioned adapters specifically designed for the Fourier domain do indeed outperform the Vanilla Adapter and LoRA. But Chebyshev Adapter slightly lags behind F-Adapter in accuracy and increases latency

---

Scheme
% Params
Mem (GB)
FLOPPs (G)
Time (ms)
1.2EE (M=1)
1.2EE (M=0.1)
F-Adapter (Ours)
1.91%
15.88
548.53
90.38
0.5329
0,4633
Chebyshev Adapter
2.18%
16.19
554.80
268.02
0,5409
0.4757
Fourier Adapter
1.93%
20.57
546.85
1449.54
0.6584
0.6053
WaveAct Adapter
1.16%
15.85
547.47
92.69
0,5566
0.4691
Table 4: Computational cost and accuracy of different types of frequency-domain adapters on the IB- 1B-parameter DPOT-H backbone.
by many times, reflecting the overhead of the dense Chebyshev polynomial expansion within the KAN architecture. The Fourier Adapter's 's costly FourierKAN edge-function evaluations increase memory use by 29% and slow runtime tenfold, and its accuracy falls sharply. illustrating that naively adding high-order Fourier series exacerbates spectral aliasing. WaveAci Adapter equals F- F-Adapter in memory and nearly in speed, yet its accuracy lags, implying that learnable wavelet activations alone cannot fully capture the high-frequency dynamics of PDE solutions in Fourier domain.
This ablation study demonstrates that adaptively allocating the lightweight adapter's low-rank dimension according to spectral content (F- Adapter) is more effective than replacing the projection layers with heavier functional bases. Other ablation studies that assess how hyperparameter choices affect F-Adapter performance are reported in Appendix C.14
Scheme
Rand
Rand
Turb
f-Inverse-Adaper
0.5664
0.4983
0.4747
Vanilla Adapter [17
0.5496
0.4893
0,4696
F-Adapter (Ours)
0.5329
0.4639
0.4523
Table 5: Ablation on adapter dimension schedules. Columns report L2 relative error (L2RE) on the Rand dataset at M = 1.0 (first column), the Rand dataset M = 0.1 (second column), and the Turb dataset at M 1.0 (third column).
5.3 Discussions
In this section, we discuss the application of frequency-based capacity allocation other non-FNO based LOMs. Although an FNO backbone provides direct access to frequency features, our main focus is on assigning each frequency band its own proper bottleneck dimension rather than strictly performing convolution in the frequency domain. This insight allows our F-Adapter to extend naturally non-FFT architectures. On the pure transformer-basse Poseidon (16) model, estimate frequency energy for each Linear layer from adjacent-toke -ooenn differences, and for each Conv2d layer we perform local real 2-D FFT on the convolution output to obtain an energy spectrum that guides the adapter's is weight generation. The PEFT adapter itself still operates in the native spatial domain. Capacities are allocated to bands according to their energy following Equation (6) which equips the model with frequency awareness. Table repoorss the resulting performance gains on SWE-2D.
We observe that adapters deliver strong results when the base model an FNO, yet their effectiveness declines sharply on transformer backbone. In contrast, LoRA and its variants demonstrate robust performance on transformer backbones, reflecting established best practices in fine-tuning LLMs. But our F-Adapter still narrows this gap by significantly improving adapter performance on transformers. Building on this insight, we introduce F-LoRA: it preserves the frequency-base based capacity allocation of F-Adapter while replacing the bottleneck MLP with LoRA-style low-rank linear updates. For detailed design. please refer to Appendix CC.55 F-LoRA achieves SOTA performance across broad suite of PEFT methods in this setting.
Scheme on Poseidon
L2RE
% Param
Prompt Tuning
20
> 1.0
0.07%
LoRA [8]
0,4010
2.07%
Randl .oRA 1
0.3134
2.07%
Vanilla Adapter
17
0.6231
2.18%
AdaLoRA [58]
0.3756
2.32%
HydraLoRA 448
0,3474
2.57%
FiLM Adapter 43
0,4567
3.19%
SVFT 25
0.6742
4.22%
F-Adapter (Ours)
0.4311
4.17%
F-LoRA (Ours)
0.2746
4.78%
Full Fine- Tinein
0.1534
100%
Table 6: PEFT results on the Poseidon
backbone for 2D shallow-water equations (SWF-2D).
Conclusion
We provide the first systematic PEFT study for pretrained LOMs, exposing depth amppliheppiieed spectral error floor in LoRA. We prove that adapters avoid this limit and, guided by Fourier energy analysis, design F-Adapters that match capacity to modal energy. Updating at most 2% of weights, they set SOTA records in 1.2REs on several challenging partial differential equation tasks, validating a principled and efficient route for fine-tuning LOMs.
10

---

Acknowledgements
We would like to thank the anonymous reviewers and area chairs for their helpful comments. This work is supported by NSFC 62306252, Hong Kong ECS award 27309624, Guangdong NSF 2024A1515012444, and the central fund from HKU IDS.
References
[1] Paul Albert, Frederic Z Zhang, Hemanth Saratchandran, Cristian Rodriguez-Opazo, Anton den Hengel, and Ehsan Abbasnejad. Randlora: Full-rank parameter-efficient fine-tuning of large models. arXiv preprint arXiv:2502.00900 2025.
[2] Kamyar Azizzadenesheli, Nikola Kovachki, Zongyi Li, Miguel Liu-Schiaffini, Jean Kossaifi, and Anima Anandkumar. Neural operators for accelerating scientific simulations and design. Nature Reviews Physics, 6(5):320-328, 2024.
[3] Boris Bonev, Thorsten Kurth, Christian Hundt, Jaideep Pathak, Maximilian Baust, Karthik Kashinath, and Anima Anandkumar. Spherical fourier neural operators: Learning stable dynamics on the sphere. In International conference on machine learning, pages 2806-2823. PMLR, 2023.
[4] Mats Carlsson, Viggo Н Hansteen, Boris V Gudiksen, Jorrit Leenaarts, and Bart De Pontieu. publicly available simulation of an enhanced network region of the sun. Astronomy & Astrophysics, 585:A4, 2016.
[5] Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. Advances in Neural Information Processing Systems, 35: 15664644-6677, 2022.
[6] Tianyu Chen, Haoyi Zhou, Ying Li, Hao Wang, Chonghan Gao, Rongye Shi, Shanghang Zhang, and Jianxin Li. Omniarch: Building foundation model for scientific computing. arXiv preprint 2001 2024.
[7] Woojin Cho, Kookjin Lee, Donsub Rim, and Noseong Park. Hypernetwork- -base meta-learning for low-rank physics-informer neural networks. Advances in Neural Information Processing Systems, 36:1 36:112199111 11231, 2023.
[8] Harald Cramér and Herman Wold. Some theorems on distribution functions. Journal of the London Mathematical Society, 1(4):290-294, 1936.
[9] Yutao Du Qin Li, Raghav Gnanasambandam, Mengnan Du, Haimin Wang, and Bo Shen. Global-local fourier neural operator for accelerating coronal magnetic field model. In 2024 IEEE International Conference on Big Data (BigDala), pages 1964-1971. IEEE, 2024.
[10] Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychometrika, 1(3):111211, 1936.
[11] Robert Joseph George, Jiawei Zhao, Jean Kossaifi, Zongyi Li, and Anima Anandkumar. Incremental spatial and spectral learning of neural operators for solving large-scale pdes. arXiv preprint arXi::2211.15188 2022.
[12] John Guibas, Morteza Mardani, Zongyi Li, Andrew Tao, Anima Anandkumar, and Bryan Catanzaro. Adaptive fourier neural operators: Efficient token mixers for transformers. arXiv preprint arXiv:2 13587, 2021.
[13] Demi Guo, Alexander M Rush, and Yoon Kim. Parameter-efficient transfer learning with diff pruning. arXiv preprint arXiv:20/2.07463 2020.
[14] Zhongkai Hao, Chang Su, Songming Liu, Julius Berner, Chengyang Ying, Hang Su, Anima Anandkumar, Jian Song, and Jun Zhu. Dpot: Auto-regressive denoising operator transformer for large-scale pde pre- pre-training. arXiv preprint arXivv2403.03542. 2024.
11

---

[15] AmirPouya Hemmasian and Amir Barati Farimani. Pretraining neural operator in lower dimensions. arXiv preprint arXiv:2407.17616. 2024.
[16] Maximilian Herde, Bogdan Raonic, Tobias Rohner, Roger Käppeli, Roberto Molinaro, Emmanuel de Bézenac, and Siddhartha Mishra. Poseidon: Efficient foundation models for pdes. Advances in Neural Information Processing Systems, 37:72525-72624 2024.
[17] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-efficient transfer learning for nip. International conference on machine learning, pages 2790-2799. PMLR, 2019.
[18] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, (2)33, 2022.
[19] Jiaqi Huang. Zunnan Xu, Ting Liu, Yong Liu, Haonan Han, Kehong Yuan, and Xiu Li. Densely connected parameter-efficient tuning for referring image segmentation. arXiv preprint arXiv:2501.0S5 085500, 2025.
[20] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efffcient prompt tuning. arXiv preprint arXiv:2104.0866 086901, 2021.
[21] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv: 00//0/01000000 2021.
[22] Zijie Li, Dule Shu, and Amir Barati Farimani. Scalable transformer for pde surrogate modeling. Advances in Neural Information Processing Systems, 36:28010-28039, 2023.
[23] Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. arXiv preprint ar arXiv:2010.08832 2020.
[24] Zongyi Li, Daniel Zhengyu Huang, Burigede Liu, and Anima Anandkumar. Fourier neural operator with learned deformations for pdes general geometries. Journal of Machine Learning Research, 24(388):1-26, 2023.
[25] Vijay Chandra Lingam, Atula Neerkaje, Aditya Vavre, Aneesh Shetty, Gautham Krishna Gudur, Joydeep Ghosh, Eunsol Choi, Alex Dimakis, Aleksandar Bojchevski, and Sujay Sanghavi. Svft: Parameter-efficient fine-tuning with singular vectors. Advances in Neural Information Processing Systems, 37:41425-44446, 2024.
[26] Dongqi Liu and Vera Demberg. Rst-lora: A discourse-aware low-rank adaptation for long document abstractive summarization. arXiv preprint arXiv:2405.0063: 2024.
[27] Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in- contetee learning. Advances in Neural Information Processing Systems, 35:1950-1965, 2022.
[28] Ning Liu, Siavash Jafarzadeh, and Yue Yu. Domain agnostic fourier neural operators. Advances in Neural Information Processing Systems, 36:47438 47450, 2023.
[29] Yen- Sen-Cheng Liu, Chih- Yao Ma, Junjiao Tian, Zijian He, and Zsolt Kira. Polyhistor: Parameter- efficient multi-task adaptation for dense vision tasks, Advances neural information processing systems, 35:36889-3666 -39901 2022.
[30] Ziming Liu. Yixuan Wang. Sachin Vaidya, Fabian Ruehle, James Halverson, Marin Soljačić, Thomas Y Hou, and Max Tegmark. Kan: Kolmogorov-amold networks. arXiv preprint arXiv:2404.0 19756, 2024.
[31] Sebastian Loeschcke, David Pitt, Robert Joseph George, Jiawei Zhao, Cheng Luo, Yuandong Tian, Jean Kossaifi, and Anima Anandkumar. Tensorgrad: Tensor gradient robust decomposition for memory-effiiient neural operator training. arXiv preprint arXiv:2501.02379, 2025.
12

---

[32] Imad Eddine Marouf, Enzo Tartaglione, and Stéphane Lathuilière. Mini but mighty: Finetuning vits with mini adapters. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1732-1741, 2024.
[33] Michael McCabe, Bruno Régaldo-Said Saint Blancard, Liam Holden Parker, Ruben Ohana, Miles Cranmer, Alberto Bietti, Michael Eickenberg, Siavash Golkar, Geraud Krawezik, Francois Lanusse, et al. Multiple physics pretraining for physical surrogate models. arXiv preprint arXtv:2310.02994, 2023.
[34] Sidharth Menon and Ameya D Jagtap. Anant-net: Breaking the curse of dimensionality with scalable and interpretable neural surrogate for high-dimensional pdes. arXiv preprint arXiv:2505.03595 2025.
[35] Thomas O' LLLaay-Rosebere Peng Chen, Umberto Villa, and Omar Ghattas. Derivative-informed neural operator: an efficient framework for -imenssinn parametric derivative learning. Journal of Computational Physics, 496: 196:11255 12555, 2024.
[36] Marinela Parović, Alan Ansell, Ivan Vulié, and Anna Korhonen. Cross-lingual transfer with target language-ready task adapters. arXiv preprint arXiv:2306.0276. 2023.
[37] Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay, Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Azizzadeneshelii et al. Fourcastnet: global data-driven high-resolution weather model using adaptive fourier neural operators. arXiv preprint 22XXi:2202..12144 2022.
[38] Shaoxiang Qin, Fuyuan Lyu, Wenhui Peng, Dingyang Geng, Ju Wang, Naiping Gao, Xue Liu, and Liangzhu Leon Wang. Toward better understanding of fourier neural operators: Analysis and improvement from spectral perspective. arXiv e-prints, pages arXiv-2404, 2024.
[39] Md Ashiqur Rahman, Robert Joseph George, Mogab Elleithy, Daniel Leibovici, Zongyi Li, Boris Bonev, Colin White, Julius Bemer, Raymond A Yeh, Jean Kossaifi, et al. Pretraining codomain attention neural operators for solving multiphysics pdes. Advances in Neural Information Processing Systems, 37:104035-104064 2024.
[40] Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Mhyyics-informen neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686-707, 2019.
[41] Hayden Schaeffer, Russel Caflisch, Cory D Hauck, and Stanley Osher. Sparse dynamics for partial differential equations. Proceedings of the National Academy of Sciences, 110(17): 663446639, 2013.
[42] Junhong Shen, Tanya Marwah, and Ameet Talwalkar. Ups: Efficiently building foundation models for pde solving via cross-modal adaptation. arXiv preprint arXiv:2403.07187, 2024.
[43] Aliaksandra Shysheya, John Bronskill, Massimiliano Patacchiola, Sebastian Nowozin, and Richard E Turner. Fit: Parameter efficient few-shot transfer learning for personalized and federated image classification. arXiv preprint (XXx:2200600867 2022.
[44] Sidharth SS, Keerthana AR, Anas KP, et al. Chebyshev polynomial-based kolmogorov-arnot networks: An efficient architecture for nonlinear function approximation. arXiv preprint arXiv:2405.07200, 2024.
[45] Shashank Subramanian, Peter Harrington, Kurt Keutzer, Wahid Bhimji, Dmitriy Morozov, Michael W Mahoney, and Amir Gholami. Towards foundation models for scientific machine learning: Characterizing scaling and transfer behavior. Advances in Neural Information Processing Systems, 36:71242-71262. 2023.
[46] Makoto Takamoto, Timothy Praditia, Raphael Leiteritz, Dan MacKinlay, Francesco Alesiani, Dirk Pflüger, and Mathias Niepert. PDEBench Datasets, 2022. httpsstss: //dd:/.or////000//// 18419/ SAA0///ARUS2200
13

---

[47] Makoto Takamoto, Timothy Praditia. Raphael Leiteritz, Daniel MacKinlay, Francesco Alesiani, Dirk Pflüger, and Mathias Niepert. Pdebench: An extensive benchmark for scientific machine learning. Advances in Neural Information Processing Systems, 25:1596-1611 2022.
[48]
Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Cheng-Zhong Xu. Hydralora: An asymmetric lora architecture for efficient fine-tuning. Advances in Neural Information Processing Systems, 2024.
[49] Alasdair Tran, Alexander Mathews, Lexing Xie, and Cheng Soon Ong. Factorized fourier neural operators. arXiv preprini arXiv:211 1.ee22111.138000 13802, 2021.
[50] Tapas Tripura and Souvik Chakraborty. Wavelet neural operator: neural operator for parametric partial differential equations. arXiv preprint 2rX1v:2205.02191, 2022.
[51] Yizheng Wang. Jinshuai Bai, Mohammad Sadegh Eshaghi, Cosmin Anitescu, Xiaoying Zhuang, Timon Rabezuk, and Yinghua Liu. Transfer learning in fine-tuning, lightweight fine-tuning, and low-rank adaptation. arXiv preprint arXiv:2502.000 00782, 2025.
neural networks: Full
[52] Zipeng Xiao, Siqi Kou, Hao Zhongkai, Bokai Lin, and Zhijie Deng. Amortized fourier neural operators. Advances in Neural Information Processing Systems, 37:115001-115020, 2024.
[53] Jinfeng Xu, Zheyu Chen, Jinze Li, Shuo Yang, Wei Wang. Xiping Hu, and Edith C-H Ngai. Fourierkan-gef Fourier kolmogorov-arnold network-an effective and efficient feature transformation for graph collaborative filtering. arXiv preprint 2riiv22006.10044, 2024.
[54] Zhi-Qin John Xu, Lulu Zhang, and Wei Cai. On understanding and overcoming spectral biases of deep neural network learning methods for solving pdes. arXiv preprint arXiv:2501.0995 2025.
[55] Zhilin You, Zhenli Xu, and Wei Cai. Mscalefno: Multi-scale fourier neural operator learning for oscillatory function spaces. arXiv preprint arXiv:2412.20183, 2024.
[56] Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked models. arXiv preprint 100.:22006.10199 2021.
[57] Hangwei Zhang, Zhimu Huang, and Yan Wang. Ac-pkan: Attention-chhaneed and chebyshev polynomial-basee physics-informe kolmogorov-arnold networks. arXiv preprint arXiv:2505.0866 2025.
(58) Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget allocation for parameter- efficient fine-tuning, arXiv preprint arXiv:2303. 105 2023.
[59] Xuan Zhang, Navid Rajabi, Kevin Duh, and Philipp Koehn. Machine translation with large language models: Prompting, few-shot learning, and fine-tuning with qlora. In Proceedings of the Eighth Conference Machine Translation, pages 468-481, 2023.
[60] Zhiyuan Zhao, Xueying Ding, and B Aditya Prakash. Pinnsformer: A transformer-based framework for physics-infornee neural networks. arXiv preprint arXiv:2307.1 11833, 2023.
[61] Anthony Zhou, Cooper Lorsung, AmirPouya Hemmasian, and Amir Barati Farimani. Strategies for pretraining neural operators, arXiv preprint arXiv:2406.08411 2024.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction accurately capture the paper's core contribution contribution--fiqqeeeee--- -requency-aware adapters that attain state-of-the-art parameter-effciient accuracy on 3D Navier-Stokes tasks taskstthoulh they slightly over-generalize the method's applicability by under-stating its reliance on FFT-based backbones.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Appendix D explicitly reflects on the strong www-freuueccy-oominee assumption that underpins the theory, acknowledges that highly non-linear multi-physics flows (e.g. MHD turbulence, reactive plasmas) may violate this premise, and identifies the need for future work to characterise such regimes-thereby satisfying the checklist's requirement discuss assumptions, scope and robustness. Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
15

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: All major formal claims are explicitly stated with their required hypotheses and accompanied by full proofs in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Section 3.1 discloses the exact PDEBench datasets, DPOT-H backbone, optimizer, schedule, hardware, and PEFT placement, while Appendix C lists the complete hyper-parameter table, data splits, layer configurations, and motrii-together with commitment to release code upon acceptance, this provides all information needed to replicate every reported result.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
16

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: The PDEBench datasets publicly available and the code will be made publicly available upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ mnips.cc/public//
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 3.1 specifies the data splits, backbone, optimizer (AdamW), learning rate schedule, batch size, epochs, and hardware, while Appendix parameter table and evaluation protocol, giving readers all training and test details needed to interpret the results.
LirixxCClss Clists the complete hyper-
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Experimental results are tested multiple times to ensure stability and reliability,
17

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state The authors should preferably report a 2- 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative етгог rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Compute resources are thoroughly described and evaluated both the experimental setup and the experimental results sections.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http::////urrppp.ccp///////eeeee
Answer: [Yes]
Justification: The study operates exclusively on publicly available scientifeessiuultion data, involves no human subjects or sensitive personal information, and poses no foreseeable dual-use or societal-risk concerns, thereby conforming to the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there is a special consideration due to laws regulations in their jurisdiction).
10. Broader impacts
18

---

Question:
Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Appendix E highlights positive outcomes (e.g., greener, more accessible high-resolution scientific forecasting) and also cautions about dual-use risks for weapons design and bias propagation in safety-critical deployments, thus addressing both sides of the pocietal-impact spectrum.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Appendix D commits to releasing the code under a research-only licence and to adding provenance logging, providing concrete safeguards to limit potential dual-use misuse of the released surrogate models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
19

---

Answer: [Yes]
Justification:
The paper cites the original paper that produced the code package or dataset.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in the thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswithcode
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The code will be made publicly available upon acceptance.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The study uses solely publicly available 3D-Naaier-Sttokes simulation data
and involves human participants crowdsourced tasks, so the question is not applicable.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
20

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
The study uses only publicly available simulation data and includes experiments involving human participants, so IRB review is not applicable.
Justification:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non non-standart components.
Please refer to our LLM policy http://arrrrrp..cooooooooaaraaae for what should or should be described.
21

---

A Preliminary
Let Rd denote the hidden activation transformer sub-layer and Wo E IRddd be the frozen projection learned during pre-training.
Low-Rank Adaptation (LoRA) frozen:
LoRA
18
injects a rank-r correction into Wo while keeping
hout
W hW),,
AW a BA,
(11)
A AER R.dd
B BER Bddr
a
(12)
so that only A and totalling 2rd parameters are updated.
Bottleneck Adapter Adapters 17 append a two-layer bottleneck MLP with residual gate
hdown W down h + bdowns
hact f(hdow)),
hup-
W bact + bup
hout + s haps
(13)
where W Frxd Wap Rdxr, f(-) GELU, < d Ryo. The adapter adds
2rd + d + trainable parameters and reduces to the identity when
Both LoRA and adapters thus enable parameter-efficient fine-tuning by confining learning to small, task-specific subspaces while preserving the frozen pre-trained backbone.
B Mathematical Proofs
Lemma 1 (LoRA error lower bound). Let AW Raxd admit the singular-vales decomposition AW UEVT with singular values 2 2 ... 2 oa 2 0 and orthonormal singular vectors Ul..... ual. V (u....., val. For any factorization Rdxx, A E Fxx and any RR
|w x BArlI 2
of x)2
(14)
i=7+1
Furthermore, for the worst-case lower bound of the LoRA enror we obtain
sup |a]||=11
|AW x- BAx Z 0r41.
(15)
Proof. Because AW = UEVT, every x E Rd satisfies
AWr U.V x Σ as (v)a)) u4
(16)
By the orthogonality of V = [e1..... val. one has V = I. hence
= ==rr- ==1r= (VTV)== (Vz) \ (
(17)
Since rank(BA) S (r,im(AA) C spanful...... U1 }. }. so there exist scalars a1...., or with
BAr
04 u4
(18)
Define the error e(x) = AWr - BAr. Substituting T6 and 18 gives
e(x)
Σ
[oo x) asju au
Σ σi (v. u.
(19)
t=1
22

---

The orthonormality of {u:} implies
d
||e())]
-
-as +
Σ ,)(()
(20)
i=1
1=++1
Because the first sum can be made arbitrarily small by a suitable choice of a, the second sum furnishes unavoidable contribution:
|AWx BAx| Z
Σ a)
(21)
completes the proof of LoRA error's lower bound.
Next, we present the proof establishing the worst-case lower bound for the LoRA error.
Let S == (A) C IRd be the column space of B.A. Since rank S we have: dim(S) < and dim(S-) 2 d r.
(22)
Consider the (r + dimeeniioa subspace spanned by the first r + right singular vectors:
Vr+1 == span
(23)
By the subspace intersection theorem:
dim(S- V++1) 2 dim(V++1) + dim(s+) - d 2 (r+ (d (d-r)) r) d=
(24)
Thus, there exists unit vector 2o E S n
Decompose To in the singular vector basis:
0g
with
Σ i=1
a
=1
(25)
1=1
The approximation error satisfies:
||W xu B BAr = P (AW ) + (Ass - BAxo
(26)
2 IPs (AW xo)
(27)
Using the SVD of AW:
AW
(28)
i=1
Since Ta E - and BAr E S, we have Ps- (BAxo) = 0, thus:
r+1
(AWW))
Σ o,a,Psuus ug
(29)
From the optimality of AW, for any unit y Rdd AAWy - AW 2 0r41.
(30)
Specifically for y = E V++1
r+1
AAW 20
Σ
oe 2 0 o
(31)
i=1
Combining 27 and 31 :
AW 2o
BA
|| (AW 2l)
(32)
|A xo 2
(33)
2 o Σ oi
(34)
1=1
2 0. - of Σ a
(35)
i=1
2 2-11-000 - of a2
(36)
23

---

The maximum is achieved when d,+1 = 1, giving
AWax zo BAzoll 2 0,+1-
(37)
Since there exists at least one for which Eq. 37 holds, the supremum over all * 0 satisfies:
sup 2=40
AWW x BAx 2 0,+1-
(38)
Remark 1. The worst-Cate lower bound coincides with the optimal spectral-norn error || |AW AW,Il2 Jr+1 given by the classical Eckart Eckart-Young- Youkg-Mirsky theorem(10), and is therefore tight. Remark 2. Lemmaa demonstrates that LoRA cannot attain zero approximation error; while its worst-case error is governed by the (r+1)-st singular value, LoRA yields meaningful improvement only when this singular value is sufficiently small.
Proposition 3.1 Block-wise LoRA lower bound ). Let AW; g blockdiag AWtiiagg@WWWW AW ..... AW(K)) be the biock-wise model parameter updates and BA = blockdiag ((") (()) ,...., B(K) A(K) be the (")ikddig(((()))A)
block-wise low-rank approximation, where B(b) E Cáxr, A Crxd. Then, for any input x, the approximation error for block- wise LoRA satisfies
K Σ
1/2
(AW, BAA) N
E i-11
oh (v),
(1)
In particular, the worst-case operator norm error obeys
sup [I BA)&l12 2 (Ar+)(AW,)
(2)
||
Proof. Since V& [bk,1,..... uk.al E IRdxd is orthogonal,
VoVV la.
(39)
For any 2% Rd.
xs = VoVVVT xx) Σ (v)) Uk.s.
(40)
Set
e := AW B
(41)
With AW; = V VI and im(B,Ad) C spanfuu.1.... urrr there exist scalars ag, (1 <igr) such that
AW Σ
one (u,, up.s
(42)
B&A Σ Okii Uk.i.
(43)
Hence
Ck =
Σ (ox,((TT, - ax.]uus uk, +
Σ
aka(UK (T)) uk.i-
(44)
1=41
Orthonormality of { implies
|| Σ [on.(( an.1]+ Σ
o >
Σ
(45)
24

---

Since (AW, - BA)x = (e1...., ex)" and the blocks are orthogonal,
d
II BA)x| - Σ e 2
Σ k=1 imr+1
(46)
and taking square roots of (46) yields the stated bound.
Next, present the proof establishing the worst-case lower bound for the block-wise LoRA error.
Define
AAW, BAllop
sup (AW,, BA))
(47)
1
For any unit vector (x1....,
K (48)
(AW, - BA) = (Di D IIA BA) Σ
Da
Writing the SVD AW; U with Eg diag(o ........ ok.d), any E IRd expands as (1) a) UR, and
D&I Elone(ohaa)) akii] uk Σ
(k,((1 Uk,i
imp+1
for some (fp,1 Okr-- Choosing x = Upi++1 gives
|Dxllop 2
D Tkr
(49)
Taking the maximum over and using yields 2
Finally, since AW, block-diagonal its singular values the multiset { holds and the proof is complete.
Lemma 2 (Universal Approximation Theorem for Adapters). Let K C RDD be compact, a: R - IR continuous non-affine function, and C(K;R.) For every & > 0, there exist parameters m E N. V E mD U E Dxm b Rm and E ID' such that
h(z) = = UVV
satisfies
sup |1((x) - f(x)lloo <E
(50)
zEK
Proof. Define the hypothesis classes:
Ho
E (w) x + bi) us RR RD E
E
= Ho {c}.
(51)
Step 1: Density Argument by Contradiction. Assume H + CKK: By Hahn-Banach theorem, there exists non-zero continuous linear functional (C(K;RD)) such that:
L(h) vh H.
(52)
Step 2: Riesz-Markov- Kakutani Representation. For vector- valued continuous functions, there exists a R P---vlledd Radon measure ..., such that:
L(g) = Σ / (,(2)dp5(a), Vg E C(K:RD)
(53)
3=1
The annihilation condition becomes:
Σ I hy(z)du;(x)= =0 Wh E H.
(54)
K
25

---

Step 3: Constant Function Elimination. Testing with constant functions h(z) co E RP:
(k)) Vea E R.P.
(55)
This implies the total mass vanishes for each component:
(k) = AA visisD,
(56)
Step 4: Single Neuron Analysis. For any direction w RR, bias b E RR and basis vector ej, consider:
h(x) ejo(w To x+ ++b) Ho- 54 gives:
(57)
Substitution into
aw x (z)
Yu YWERD,be bE R, R,
(58)
Step 5: Projection to ID Measures. For each w R.,, define projected measures V/ R by:
Vi = Uj ((1 K E A)) for Borel A R.
(59)
Equation
58
becomes:
[ all b)des, 0 44 E R.
(60)
Step 6: Fourier Analytic Argument. Let F denote the Fourier transform. For tempered distributions:
Fla (w) [lo](w) Flu (w) = =((w) V.) = Ve E R.
(61)
(62)
Lemma 6.1 (Non-vanishing spectrum). For non-affine a C(R) \ Pi, 7 is not identically zero. Specifically:
If a is sigmoidal: (ww) has exponential decay but supp(ō) =
For ReLU: (ww) = Tó(w)) + (in distribution sense)
GeLU: (w) is analytic and non-zero on R \ (0)
Thus Vjj.. = 0 in 62
implying Vj = O.
Step 7: Cramér-Wold Device. For any w E IR., the projected measure satisfies:
(A) = Hy (o E K TT xE A) = AAÇR. R.
(63)
By Cramér-War Wold theorem g this implies:
(B) = vBorel B K, 1 15j5D.
(64)
Contradicting L # 0 in 53 Therefore H = D):
Step 8: Approximation Construction. Given f E C(K; PP and 0, by density there exists h H with:
|| fllc() sup zek ISDD
h,)(x) [x) <
(65)
This completes the universal approximation property.
Remark (Measure-Theoretic Details). All measures are Radon measures by the Riesz-Markon Kakutani theorem. Fourier transforms are interpreted in the distributional sense. The Cramér-Wir Wold theorem applies to finite Borel measures
Remark (Activation Function Spectrum). The critical requirement ā # 0, satisfied by:
Non-polynomial finction:: xtt) et e'
26

---

Piecewise linear functions with o" + 0 distributionally
Functions with non-vanishing generalized spectrum
Remark 5. By contrast to Lemma pemmapproves universal approximation theorem for adapters: the presence of nonlinear activation endows them with strictly greater expressive power than LoRA. Lemma (Spectral decay from Sobolev regularity). Let 9: T -> C be 2x-periodic with all weak derivatives up order N in L ( where > d/2. Denote its Fourier coefficients by
gk (2T)² J g(x))==eee ik. dx, ke
Then there exists C such that
1gke - C(1 + |k:3))a/2 Ak zd.
Proof. Let N with a d/2 and fix a multi-index m (mi,.-. ma) satisfying [m]] Σj= my = O. For any test function Φ E C (TT), integration by parts in the distributional sense yields
J 8t g(z))(())d (-1) mm [fr g(x)0m(()))) (())dr.
(66)
Applying this to d(x) ghi and noting that e -ik-
= (--ik)"e-th we derive
[ g(x)e she dr (-ik))m [ 0t ( x dx.
JTu
(67)
Consequently, the Fourier coefficients satisfy
[ 8th 0"h g())ee ik- da.
9h (2 iik)) JTM
(68)
Taking absolute values and applying Hölder's inequality, we obtain
Lok S (2 kmml JL (0")(==)dd==
g|0mg (2m)<Am²)*
(69)
To bound |k= = -I[, Ky " observe that by the arithmetic-geometric mean inequality,
|
II TDs
(70)
Vd
where ||||| = //g + + k2. Substituting this into the estimate for [gx] gives
gkl 5 d d0/21100m gllss
||:
(71)
(2mrd
For low-frequency modes with || < the bound Igul I |1llla holds trivially. Combining both cases by defining
d d2 g (2Trd
C = max
sup [1]
lgkl.
(72)
we achieve the unified decay estimate
Igul < C(1 + 3k]3))-(1 a/2
4k E gd
(73)
Sharpness follows by considering test functions g(x) = II cos 2, with 3 > a, where direct calculation shows [9kl
||||| -28
Lemma 4 (Spatial- (Spatial-domain domain adapter error bound). Let Moingle-layer adapter in the spatial domain produce perturbations ö, at grid points E I C .l....,Ny with 16, S Em uniformly, and suppose #I Kd, Then the global G eerro
N2
6b i I
0, i i
27

---

satisfies
||l| = Σ
1 1
1/2
S
Vdd 6m = O(K422).
(74)
ST
so in the absence of any decay in the perturbations one only obtains the spatial-domain rate O (Kd/)).
Proof. By definition,
||e|12
Σ
10112
K E mm
IEI
and taking square roots yields ||e|| & Kd/2 Em O(K4)3)
Proposition 3.2 preequency-selectivv approximation of adapters). Let l9a| S (k()) with >> For any E> there exist frequency truncation radius K and adapter bottleneck width m E such that the
adapter obeys
||e|| OK + OK (m
||l| = || (g) - FF (1)l2 <s
(3)
Proof. Let
e = F (g(t) g()))
(75)
By the unitarity of F-1 we have
||e|12 Σ1 (ax( -
(76)
Split the sum in (76) into the high-freyuency part (k) > K and the low-frequency part (k) < K. For the high-frequency truncation use [g 0 and the decay hypothesis Ig (+x)) S C(k) -a:
Σ lax (ix) (K)>K
S C²
Σ (K)>
(k)-20 20 S C2 f
(1+12)a dr.
d
(77)
Setting r = v8 with dr
1 2/2 gives
d-2
r d-1
1 1 3 2
1
00
Co
dr
ds
<
Ca
8 a--a-1
ds
kd-2a (78)
K
K
r2+
2 K2 (1 + s)a
2
K2
2(a 2)
so that
(+x)) d-Kaa.
(79)
(
Taking square-roots yields the high-frequency contribution
mnl') 1/2
C Ca
Σ <
gk
AAKE
A
V 2( d)
(80)
For the low-frequency part (k) < K, the adapter achieves exponential uniform accuracy:
1gx (fx) - Gr (
S 6m
(81)
hence
Σ 19x Ōk 2 < Kdd 2 20m
(82)
and after taking square-roots the low-frequency contribution is
1/2 1)
Σ KK
B K Ze cm
= mm
(83)
K
28

---

Combining these two estimates with (76 gives
d
||l| f AK + BKBB 2e om
(84)
To ensure ||e|| < E, it suffices to choose K and such that
1/(a $1
A 2
K
24
(85)
d B 2
mm
cm
m >
2B
(86)
e
Since > d/2, the exponent 1/(a- d) is positive, and thus one can always pick finite K and then m to satisfy both inequalities. This yields
||e|| = ( + O Be
(87)
which for suitably growing m is strictly faster than the spatial-domain rate O(K4/2) in Lemma 4
Remark (Error Component Interpretation). The spectral truncation term O(K2-9) " reflects the accelerated decay granted by a > d/2 in the Fourier domain. The parametrization term D(K2e-om demonstrates that, by increasing the adapter width m, one obtains exponential control over the low-frequency upproximation error.
Proposition 3.3 (Quantitative Low- Low-/High Frequenyr Energy Split for PDE Solution). Let s > d and suppose J C[O, 7); H (T))), suPrEJo,T] (1)())) I M. Let (t, = ΣxEZ s(e, k) e k- then for each k + 0,
[f(t,k] S M (1 + 1|A|12) ) and every integer K 2 there exists C- C(d,s) 8) such that
(4)
Σ |
k) < SCM2
Σ = -(K
(5)
|
Proof. By assumption one has
ll = ([+1113)**((tt..) s M2.
(88)
kEZd
Hence for each nonzero k; A)) - A (1 + 11&1))) [f(t,K] s M (1 + [|A||3) -/2.
(89)
Define the frehefryyyncyy tail
(x = Σ |f(t,k) S M Σ (1+ ||()1))-
(90)
||
||||>
Partitioning {k: |||||| > K} into shells m - 1 < ||k|| S m and writing N(rr) = #(k:
||]|| S rr.
we get
Σ (1+ [(|13)+
[(m) - -N(m 1 13] (1+ + (m - 1)2)-3
(91)
||||
m=[K|+]
Since ((m) N(m 1) < Cam and (1 + (m 1)²)-a S (m 10-11 -1) 1)-2, one obtains
Σ (1+ | < Ca |
m 1-122 < Co
1 K
pd-1-22 dr = Ca K d-20
922)
2s d
K +1
follows that
Ax(1) _ Cs M² K d-22
(93)
2s d
Finally, Parseval's identity gives
Σ
=1000) Ax(1) 11(()) -
(94)
J
which completes the proof.
29

---

C Experimental Settings and Supplementary Results
C.1 DPOT Backbone
The huto--eereesivve Denoising Operator Transformer (DPOT) a Fourier Nourier-transformet backbone designed for large-scale pre-training on heterogeneous PDE trajectories [14), Its architecture (Figure 2 of the original paper) is factorised into four principal Temporal Aggregation, Fourier Attention, and Output Projection-which together convert raw spatiotemporal grids into operator-valued predictions while retaining full frequency information.
Embedding,
Patch/Positioning Embedding Layers. Each input trajectory urt E R is first patchified: a Px convolution groups neighbouring cells and lifts them to a D-dimensional token space. Learnable positional encodings Wp W2( f). are then added channel- -isee, producing per- uy
H
time-step embeddings Z E xD that remain cesolution-annosia across datasets.
Temporal Aggregation Layers. To condense the temporal context, DPOT employs a weighted temporal MLP with complex Fourier features. For each spatial location (i, and channel c, the layer forms weighted sum
T Σ
(i,.cc) AgB
w!(o)
2.1555))) ix 2p
(95)
where W(O)) and %o are learnable and shared across datasets. This operation implicitly encodes fume-freueeccy signatures that help the model infer PDE type and latent parameters from short sequences.
Fourier Attention Layers. The core stack consists of L Fourier-attentinn layers. Each layer lifts its input to frequency space via an FFT, applies a two-layer multi-head MLP K₁ to the complex coefficients, and reverts to the spatial domain with inverse FFT before point-wise MLP Mf:
(++1) ( + (K(
(96)
This frequency-spar space mixing acts efficient global kernel integral transform and scales linearly with sequence length in httpr/cccceeteeeeeeeeeeeeoiiiiii
Output Projection Layers. Finally, point-wise projection Q RR - RRout maps the latent field back to the physical variable space, optionally preceded by up- up-sampling or padding to match the desired resolution. Because Q operates channel-wise, it is independent of grid size and can be re-used for variable-sized oomm::/:.cootttttteoeeooooooooor (oneeerrreeeececciiiiiooooeeeeed
Discussion. The modular design makes DPOT both flexible-hhanding irregular resolutions, channel counts, and temporal lengths-aud and scalable: model width D and depth L can be increased to the B- parameter regime with ncar-linear FLOP growth. Moreover, the FFT-IFFT symmetry of the Fourier attention stack enables lightweight fine-tuning strategies such as our Frequency-Adappe (Section 4 which can be inserted without modifying pre-trained weights or training schedules.
Adapters
C.2 Poseidon Backbone
The scalable Operator Transformer (scOT) T is the backbone of Poseidon [6., designed to approximate solution operators S(t, a) time-dependent PDEs by jointly encoding the lead time and the input function in a hierarchical, multiscale rision-transfommer architecture with shifted-windor (Swin attention. In contrast to next-step predictors, scOT directly learns the operator that maps initial data to the entire solution trajectory and supports continuous-in- evaluation through limecconditioed layer normalization.
Patch and Embedding Layers. Inputs a E C(D:R") are first partitioned into non-overlapping pxp patches and linearly embedded into C-dimensional latent field C(D; RC). This discretizes patching operator that averages within patches and lifts to token space; the embedding is immediately normalized by a (lead-time) conditioned layer norm (see below). The construction resolution- agnostic and serves the interface between function-space data and transformer tokens.
30

---

Layer Norm. To enable real-values time queries, scOT replaces standard layer norm LN with lead-time conditioned variant that modulates the affine parameters by
(o)(), alt) v(œ) (o(x) (x(
+ ((t), a(t) at a, ((t) 3, 3, (97)
with Pv. By the channel-wise mean and standard deviation. This simple conditioning yields continuousiin-time evaluations for S(t,-) within a single network.
Shifted-Windov (SwinV2) Attention Blocks. At each scale, tokens pass through SwinV2 blocks that apply windowed multi-head self-attention within fixed spatial windows; windows are shifted across layers allow global information exchange with linear complexity in the number of tokens. Each block follows residual "attention-MLPP stack with time-conditioned layer norms on both sublayers.
Hierarchical Encoder-Decoder with Skip Connections. Swin stages are arranged in a U- Set-style hierarchy with patch merging for down-scaling and patch expansion for up-scaling. Encoder and decoder stages at matching resolutions are connected by lightweight ConvNeXt blocks that preserve multi-scale features while keeping the bottleneck convolution-free.
Output Recovery and Mixup. After decoding, a recovery head reassembles the latent tokens back to the physical domain, optionally with mixup in the output space; this step is independent of the grid size and thus compatible with variable-resolution domains.
Training Objective and all2all Supervision. Given trajectories (S(ls, a scOT can be trained with standard operator loss
L(O) = M(K +1
(1), S(1a. (L)) (p (p= 1).
(98)
and, crucially, with an all2all variant that exploits the semigroup property S(t,,a) = S(t--,, S(t, a)) to form supervision from all intra-trajectory time pairs (fx.tg) with k < k, yielding O(K2) training pairs per trajectory and markedly improved data efficiency. At inference, scOT supports direct t-queries variable-step rollout via successive applications of Sg-
Discussion. The modular design-patee embeddings, time-conditioned normalization, shifted- makes scOT both flexible (heterogeneous PDE
window attention, and multi-scale inputs, resolutions, boundary conditions via masking) and scalable (depth/width and token counts). In POSEIDON, this backbone underpins large gains in sample efficiency and accuracy across diverse downstream PDE operators after pretraining on a compact set of quid-dynamics operators.
C.3 Fine-tuning Protocol
We begin with DPOT backbone that was pre-trained on diverse two-dimensional PDE trajectories and adapt new datasets of arbitrary dimensionality.
Dimensional adaptation. Only the Fourier-Attention layers and Patch-Embeddiig kernels are replaced with their one-, two-, or three-dimensional counterparts that match the target grid. Positional embeddings resized with trilinear interpolation, while all remaining weights are loaded unchanged.
Parameter-efficiee updates. All newly added PEFT modules are initialized with (near)-zero up- projection weights, so the network initially behaves like the frozen backbone and gradually routes learning into the adapters as training proceeds.
Apart from these structural switches, the fine-tuning follows the same recipe used during pre-training, but updates only the lightweight adapter weights and few normalisation parameters.
scheduler, and so forth-
31

---

C.4 Placement of PEFT Modules
PEFT Modules are inserted at four important positions of the network:
1. Patch/Positioning Embedding Layers- after each convolution that maps raw input patches into the latent space.
2. Temporal Aggregation
after the first patchifying layer.
3. Fourier Attention Layers-before, before, between, and after the linear transforms operating in Fourier space.
4. Output Projection Layers- to the final transposed-convolttion path that reconstructs the physical field.
This arrangement grants every major transformation pathway a low-rank, trainable side route, enabling the model to specialise to PDE systems with minimal additional parameters.
C.5 Evaluation Metric
The L2 Relative Error (L2RE) is adopted as the sole evaluation metric. Given the test set D (())) L2RE is defined as
15 y:lle
L2RE
(99)
This ratio normalizes the prediction error by the energy of the ground-truth signal, yielding a dimension- -oenne quantity whose smaller value indicates better performance.
C.6 PDEBENCH 3D Compressible Navier-Stokes (CFD-3D) Dataset
The CFD-3D benchmark released with PDEBENCH 47
targets the forward prediction of turbulent, compressible flows in three spatial dimensions. It now comprises three distinct subsets, each recorded on 128 Cartesian grid and sharing identical solver parameters and output format:
Subset name
Initial condition
(n,5,
MS-3D-turk 1S-3D-rande
divergence- turbulence
(10-8, 110-8,1088,10 1.0)
random Gaussian field low-Mach random field
110-8,10-8,100
NS- rand
(10-$,10-8.0.1)
Each subset contains 100 simulation trajectories of the full compressible Navier-Stokes (CNS)
equations (100) 1022..
0.p + V. (pv) 0,
(100)
p(0.v vv. -VVp Vn + (s+ 3
V(V.v)
(101)
1 0 E 50llv11) + +v. v. {ollv(p) v v- 0.
(102)
Every trajectory provides 21 equally-spaced snapshots (t E [0, 1) stored six-channel tensors [p. u, w. p, E 11118112812122 128x6
Initial and boundary conditions. The three subsets differ only by their initial velocity field and Mach number. NS-3D-turt seeds divergence-Fo Prregeence-free Kolmogorov-type spectrum, whereas NS-3D-rand and NS-3D-rand draw velocity, density and pressure perturbations from isotropic Gaussian random fields (extended from Equation (8) in (7) before adding uniform background. Periodic boundaries are enforced all directions, mimicking homogeneous isotropic turbulence and simplifying spectral learning methods. The mandom-field subsets include a fixed-Mach configuration at M 1.0 and a nearly inviscid, low-Mach compressible configuration M 0.1; full specifications of these variants provided in the official dataset card.
32

---

Scientific and ML challenges. Beyond the previously noted high dimensionality and shock- capturing difficulties, the extended CFD-3D benchmark now stresses surrogate models along two diversity (turbulent vs. random fields) and (ii) Mach-number
additional axes: (i) initial-conditee variation spanning an order of magnitude (M=0.1 1.0). Successful models must therefore exhibit robust generalisation across both flow regimes and acoustic compressibility scales.
Splits. Following the original protocol, we reserve 90 trajectories for training/validation and for held-out testing within each subser. We utilize stratified sampling to preserve the subset ratios.
Quantity
Symbol
Value
Notes
Spatial resolution
N₂ x Ng x N₂ 1283
Cartesian grid
Time steps per run
N₂
21
At=0..5
Number of runs
samples (1),))
100
90/10 train/test split
Viscosity pairs
10-2
Two regimes Isothermal EOS
Mach number
M
1.0
Boundary condition
Periodic
All faces
Stored channels
6
p. U. v, W, P,
Table 7: Core statistics of the CFD-3D dataset.
C.7 PDEBench 2D Shallow Water Equations (SWE 2D) Dataset
The SWE 2D benchmark in PDEBENCH targets forward prediction of free-surface flows in two spatial dimensions. Each trajectory is simulated on a 128x 128 Cartesian grid with nonperiodic Neumann boundaries and is provided an HDF5 array following the convention (N, T, X, Y, V). The benchmark offers 1000 distinct runs and, for each run, 100 stored time steps that capture nonlinear wave fronts and shock-like features typical of shallow-water dynamics. Baseline solvers and dataset packaging follow the official PDEBENCH specification.
Governing equations. The two-dimensional shallow water system is supplied in conservative form with bed-slope source terms:
0,h + 0,
+ (ghe)
(103)
(huu) + 0n hu² + ggh²²)
+ (yure) = g 0,b,
(104)
(hv) + 02 ho² + ggh + O2 (huv) = g 0,b.
(105)
where is water depth, (u, e) are horizontal velocities, b(x, y) is the bathymetry, and g is gravitational acceleration. In this representation the prognostic variables are (h, Jeu, lvo). which makes conservation properties explicit and well defined even the presence of discontinuities.
Problem setup and data generation. The dataset instantiates radial dam-break scenario square domain 52 [-2.5, with an initial water mound centered at the origin,
2.0,
v <r, u(0, 20, = v00...v) 0,
1.0,
otherwise,
where the radius r is sampled uniformly per run from J0.3, 0.7] to diversify initial conditions. Simulations use a finite-volume solver from PyClaw to generate reference trajectories that are then downsampled to the released resolution and schedule.
Scientific and MI challenges. SWE 2D stresses emulators through sharp fronts, wetting and drying interfaces, and reflection at nonperiodic boundaries. Accurate surrogates must conserve mass and handle momentum coupling while remaining stable over multi-step rollouts. The benchmark exposes these difficulties in controlled setting with standardized storage and splits.
33

---

Quantity
Symbol
Value
Notes
Spatial resolution
N₂ x Ny
1282
Cartesian grid
Time steps per run
N
100
Stored steps per trajectory
Number of runs
Naamples Ω
1000
90/10 train/test split
Domain
2.5, 2.5]
Square box Nonperiodic
Boundary condition
Neumann radial mound h. hu, hve PyClaw
Initial condition
(0.3, 0.7); u =
Stored channels
Conservative variables Finite-volume reference
Solver
Table 8: Core statistics of the SWE 2D dataset in PDEBENCH.
Splits. Following the official protocol, reserve 90% of runs for training and validation and 10% for held-out testing. The same split policy is used for baseline models reported in the dataset paper.
DPOT uses this SWE 2D benchmark when reporting pre-training and transfer results, which motivates our choice to adopt the same data conventions and splits.
C.8 Bifrost Chromosphere-Corona MHD-3D Dataset
We use the publicly released Bifrost enbanced-netwond simulation distributed by the Hinode Science Data Centre Europe (ID: en024048 hi on). provides time-indexed 3D magnetic-fiele cubes
(Bz. By. B.) on Cartesian grid of 504 504 x 496 that spans physical volume of 24 Mm x 24 Mm -2.4 Mm to (numerical range) 14.4 Mm. The standard release contains 157 snapshots at cadence of 10s, from t = 3850 s to t = 5410s. These specifications are consistent with the original description of the run and subsequent studies that use the same source.
Governing equations. Bifrost advances the full resistive MHD system on staggered mesh with high-order finite differences and explicit time stepping. The code solves, in conservative form, mass continuity, momentum balance with Lorentz force, magnetic induction, and total-energy evolution:
0p
+ (ou) = O, 0t
(106)
(pu) 0t
+ V
(pu & u - = Vp + x B B+gg.
(107)
²B = V x (ux B V (nJ).
(108)
Öt
de
+ v a
(eu) = p u u-Q
(109)
with _ B. Boundary treatment uses ghost zones with problem-depentee conditions; radiation, conduction, and non-equilibrium ionization are included through Q and the equation of state.
Initial and boundary conditions. The simulation represents an enhanced network with two opposite magnetic polarities separated by about Mm the photosphere. Convective driving shears and braids the field, producing realistic chromosphere-come corona coupling. Lateral and top boundaries are nonperiodic in the production run used by en024048 hion, implemented through ghost zones in Bifrost.
Scientific and MI challenges. MHD-3D stresses operator learners through strong anisotropy along field lines, steep gradients near the photosphere, and nonperiodic boundaries that complicate spectral assumptions. Accurate surrogates must reconstruct 3D structure from minimal boundary information and remain stable across height with respect to physically derived metrics such as [B] and [J].
Splits and extreme-scarcity protocol. To probe learning under severe data scarcity, we treat each time index as a trajectory and select 24 snapshots for training. The remaining snapshots are held out
34

---

Table Core statistics of the MHD-3D dataset used in our study.
Quantity
Symbol
Value
Notes
Spatial resolution
N₂
x No N
504 x 504 496
Cartesian grid
Physical domain
24x24x166 16.8 Mm 157
E [-2.4 Mm, 14.4 Mm]
Snapshots per run
N₂
10 s cadence; E [3850s, 5410s)
Stored channels
B2, By: B; Nonperiodic
Magnetic field
Boundary condition
Ghost-zone implementation
Downsampled target
504 x 504 x
Height subsampling
Train set (ours)
Minin
24
Extreme scarcity
Source archive
en024048 1a002404__hionno
Hinode SDC Europe
Effective Rank of AW per Block (Mi)1)
è DPoT Number
Figure 6: Spectral diagnosties after full-rank fine-tuning at M 1.0. Top row: aggregated singular-value value spectrum (left) and cumulative-energy curve (right). Bottom: block-wise effective ranks of AW.
for testing, Unless otherwise noted, inputs are the bottom boundary magnetogram (504 x 504) and targets are the downsampled (504 504 x 99) interior volume for the same time index, following the GL-FNO data interface.
Provenance and access. The en024048 hion cubes are curated at Hinode SDC Europe; the underlying simulation is documented by Carlsson et al. [4]. Recent ML work on coronal-field reconstruction from this source provides a consistent pre-processing recipe and confirms the 3D geometry and cadence figures listed above.
C.9 Spectral Diagnostics of AW after Full-Rank Fine-Tuning
To assess the intrinsic rank of the updates obtained via unconstrained fine-tuning we perform full- full-rank adaptation of the B B-parameter DPOT-H backbone the 3D-NS at Mach numbers M = 1.0 and M = 0.1. After convergence, we extract the complex-value Fourier Attention Layer weights (real and imaginary concatenated) (Wit k,p) for each block i: E [0,20 26) and projection p E {Wi, w compute the deltas
datasets
AW k.p
W k.P
Wre k.p'
(110)
flatten each A WR.p to a matrix, and perform singular-value decomposition. We define the effective rank the number of singular values o; satisfying a; > 0.01
35

---

Agpnepted pour
Effective Pankk (M
Figure 7: Spectral diagnosties after full-rank fine-tuning M 0.1. Top row: aggregated singular-value spectrum (left) and cumulative-energy curve (right). Bottom: block-wise effective ranks of AW.
Diagnostics at M = 1.0. Upper left panel Figure 6 reveals a shallow spectral decay, while upper right panel shows that only 80% of the Frobenius norm is captured by the first 100 modes. Bottom Panel further indicates that nearly every block requires 2 240 modes (of 256), confirming the high-rank nature of the update and the large approximation gap faced by rank-constrained adaptation such as LoRA (see Proposition 3.1.
Diagnostics at M = 0.1. Although the low-Mach setting leads to slightly steeper spectral decay (upper left panel Figure 7 and faster accumulation energy (upper right panel), the transformation remains far from low-rank: capturing 90 % of the energy still requires ~ 140 modes for W1 and ~ 103 modes for W2. Bottom Panel shows that the wwwn-projection pathway retains an average effective rank of s 246, while the up-projection weights still average s 226 modes, underscoring that the intrinsic update is still high-rank. Consequently, even at M = 0.1 rank- constrained linear adapters suffer from irreducible spectral bias.
Table 10 summarizes the effective-ranke statisties of AW at .M 1.0 and M 0.1, confirming that both Mach regimes yield inherently high-rank updates. Likewise, Table reports the number of singular components required to capture 90 % of the total energy: at M 1.0, 126.6 (resp. 97.8) modes are needed for wi (resp. W2), compared to 2 138.6 and R 103.3 at M = 0.1. Although compressible flows M = 0.1 exhibit slightly steeper spectral decay, the updates remain far from low-rank. These diagnostics substantiate that full-rank fine-tuning induces intrinsically high-rank transformations, thereby imposing an irreducible spectral bias on rank-constrained linear adapters.
M
1.0
M
Parameter
Mean
Std.
Min
Max
Mean
Std.
Min
Max
W1
250.7
11.4
200
256
245.9
20.3
187
256
W2
225.0
23.6
168
256
225.7
23.0
164
256
Table 10: Effective-rank statisties of AW whose dimension is 256 M 1.0 and M 0.1 (across
all Fourier- Attention blocks).
C.10 RMSE Comparison for MLP Adapter vs. Low-Rank Truncation
The numerical results reported the main text were obtained on a diagnostic set built from the first Fourier-Attention block of DPOT. Specifically, we collect.V 200 000 real Fourier activations
36

---

M
1.0
M
0.1
Parameter
Mean
Std.
Min
Max
Mean
Std.
Min
Max
W1
126.6
20.4
67
155
138.6
27.6
65
170
W2
97.8
20.8
138
103.3
26.5
52
154
Table 11: Number of singular components required reach 90 % cumulative energy M 1.0 and
M 0.1.
H E EERXN Nxd and compute their targets Y = H AWW,, where AW is the exact full-rank weight update after fine-tuning. The data are split 90/10 into training and validation subsets before fitting either surrogate. Table 12 lists the root-mean-square error (RMSE, reported in 10 units) for all budgets considered. Consistent with the curves F the two-layer ML.P adapter dominates the low-rank SVD baseline across the entire budget spectrum and for both Mach numbers.
Mach
Method
8
16
64
128
Adapter (MLP)
30.48
22.05
17.64
6.02
5.70
5.09
M 1.0
Low-Rank Trunc.
29.10
27.99
26.08
22.06
16.83
8.96
Adapter (MLP)
24.76
27.67
18.38
8.43
4.04
4.46
= 0.1
Low-Rank Trunc.
22.95
21.92
20.23
17.37
13.53
7.34
Table 12: Held-out RMSE (x10-2) for each parameter budget. Adapter budgets correspond to hidden widths m, low-rank budgets correspond to SVD ranks r.
Implications. Because the MLP trained directly on the (H, mapping it can exploit non-linear interactions in the representation space that any linear low-rank approximation of AW must ignore. The result corroborates our spectral analysis: even aggressive rank truncation leaves non-negli error floor, whereas modest non-linear adapter is able to emulate the full-rank update with far fewer tunable parameters. This finding further reinforces the case for Adapter-style PEFT in operator learning tasks characterized by pronounced physical complexity.
C.11 FLOPs and Inference Time for Experiments in Section
Scheme
FLOPs (G)
Step Inference Time (ms)
AdaLoRA
158
543.384
75.631
HydraLoRA
48
547.039
155.784
Prompt Tuning 20
540.838
28.649
Vanilla Adapter
547.469
81.823
FiLM Adapter
43
548.318
93.676
RandL 1
545,458
73.432
LoRA [18]
551.008
71.651
F-Adapter (Ours)
548.531
90.383
SVFT 25
630.880
93.026
Chebyshev Adapter
554.797
268.022
Fourier Adapter
546.849
1449.544
WaveAct Adapter
547.469
92.694
Full Fine-Tuning
540.838
27.427
Table 13: Computational cost and single-step latency of parameter-efficient fine-tuning strategies.
Table 13 presents a comparative overview of computational overhead incurred by each PEFT scheme in the 3D NS Rand M = 1.0 experiment. Our F-Adapter executes 548 G FL.OPs-only about 0.2% more than Vanilla Adapter and comfortably within the typical LoRA budget-while yielding single-step inference latency of 90 ms. well below the 100 ms threshold commonly regarded
37

---

Scheme
RMSLE&
RelErrs
% Param
Vanilla Adapter
0.9186
10.55%
1.16%
LoRA
1.9356
435.09%
1.37%
F-Adapter (Ours)
0.9095
6.12%
1.91%
Full Fine-Tuning
0.3208
0.21%
100%
Table 14: Spectrum-level accuracy and parameter efficiency. F-Adapter achieves the best spectral fidelity among PEFT methods while retaining a small parameter footprint.
as interactive. Although Prompt Tuning and full fine-tuning achieve slightly lower latencies (28 ms and 27 ms, respectively), the runtime premium of F-Adapter modest and offset by its richer representational capacity. Crucially, our approach is an order of magnitude faster than hydra-style LoRA or higher-order spectral adapters specifically designed for Fourier domain, underscoring the efficiency of its frequency-adaptive design. Overall, F-Adapter strikes favorable balance between computational cost and adaptation power, making practical drop-in replacement for existing adapter families in latency-sensitive scenarios.
C.12 Spectral Analysis of PEFT Methods on DPOT
Setup. We evaluate how well different PEFT methods recover multi-scale 3D turbulence in 3D experiment by comparing the predicted isotropic kinetic-energy spectrum against DNS on the test set, and by inspecting 3D visualizations of velocity magnitude. For spectra, compute E(k) from the three velocity components using a 3D FFT and shell averaging in wavenumber space; prediction and DNS are processed by the same pipeline.
Metrics.
To quantify agreement across scales we report two spectrum-level metrics.
(i) Root mean square logarithmic error (RMSLE) of the spectrum
Σ log Epred (k:) log EDNS 2
RMSLEø
(111)
where the sum runs over wavenumber shells { This measures shell-wise discrepancy on logarithmic scale so that low and high & bands contribute comparably.
(ii) Relative error of the total kinetic energy
Amaax
Exot.pred
DNo
Exst
E(k) dk;
RelErrg
X 100%.
(112)
F
The integral equals the metric captures conservation of total energy content.
turbulent kinetic energy up to a constant factor, so this
Findings. Figure & presents E(k) logarithmic axes together with the DNS and k-5/3 reference slope for the inertial range. F-Adapter tracks the DNS spectrum closely over broad band of k and preserves the correct decay higher wavenumbers. Vanilla Adapter exhibits noticeable deficit in mid-to mid-to-high k. LoRA underestimates energy by orders of magnitude across most shells and shows noisy behavior the largest ht consistent with the very large RelErrg.
C.13 Discussion on Different Types of Adapters for Fourier Domain
Motivated by prior work 52 which leverages the frequency-domain expressivity of Chebyshev polynomials within FNO, we propose the Chebyshev KAN Adapter (Chebyshev Adapter). It utilizes the spectral expressivity of Chebyshev-hased Aolmogorov-Arnold Networks 44 by replacing the standard linear up- up-orojection with ChebyKAN layer. Given input
Chebyshev Adapter
38

---

10-3
DNS (Ground Truth (Solmaguros)
Energy spectrum E00
13a
10
10'
DMS foround Iruth)
10a
Model (adepter)
Energy spectrum EUK
10"4
(Kolmogorod)
10
1
10:
10
16a
DNS (Ground Truthi
10"
Model (adapter)
Energy spectrum EIK)
10-4
(Koimogerov)
10"1
10-1
10
10:11 11
10a
10
Weve number
Figure 8: Energy spectra E(k) on logarithmic axes for three PEFT methods, each compared to DNS (blue) and k 5/3 553 reference slope (gray). From top to bottom: LoRA, F-Adapter (Ours). and Vanilla Adapter. F-Adapter follows the DNS spectrum more closely across a broad range of wavenumbers, while Vanilla Adapter shows a deficit at mid-to-high & and LoRA substantially underestimates energy with noisy behavior at large k.
activation x E Rdd,, the Chebyshev Adapter computes
tanh W down bdowa),
(113)
dioctteneck
N
Σ Ckia Tn (2:), n=0
k= 1, dnn-
(114)
1=1
where W downn& IR dRoosseeencc dhn is the learnable down-projection, &; = tanh(z;) normalizes into [-1, 1], and (x) = cos( arccos(x)) is the n-th Chebyshev polynomial of the first kind. The coefficient tensor C Rd xduotilencc x(N+1) is learned end-to-end. Finally, residual connection with learnable scalar a restores the original dimension:
Chebyshev-Adapteer
a
(115)
We initialize Wdown by Kaiming uniform and set all Ck.i, = O to start training from the identity mapping. The scalar is also learnable, allowing the model to adaptively control the adapter's contribution.
The Chebyshev Adapter leverages the enhanced Chebyshey-KAN Layer to boost approximation power without the dense spline-grid storage required by standard Kolmogorov-Aroold Networks 30].
39

---

Fourier Adapter Motivated by the amortised Sourier-kerne formulation of Xiao et al. [2 and the expressive FourierKAN layer of Xu et [53]. we introduce the FourierKAN Adapter (Fourier Adapter) as frequency-domain alternative to the vanilla bottleneck Adapter used in LOMs. Given an input activation x E Rdd,, the module first performs linear dimension reduction
z W down + balon),
W down E Bootteececk Xduu.
(116)
where a.)) denotes GELU unless stated otherwise. To restore the original width we replace the standard linear up-puor projection with FourierKAN layer that expands each scalar 2; into truncated Fourier series of order K:
dnotelenack
Vk
An.i, cos(n Bu.i,n sin(nzz)):
== 1, I....,.. din
(117)
y= (11..... Vdd..)
(118)
Here the learnable coefficients A, B lie in Rd Xduottienokk x Because cos and sin are 2n 2i-periodic and globally supported, Eqs. (117) (118) endow the adapter with a strong inductive bias for periodic, high-frequency phenomena that commonly arise in PDE spectra, while avoiding the dense spline grids required by classical KANS [30]. The series order K is typically I 256 to curb aliasing and memory, yielding an (dinnhhotttencck K) cost.
learnable LayerNorm followed by residual scaling finishes the block:
Fourier-Adapter (x) LN(y) +x,
(119)
where learnable parameter a is initialized to 0, so training begins from the identity map. We initialize (A, B) with W(O, K -11/2d bottlenocc)) and attenuate high frequencies by (12 1) to ensure smooth
scalar functions at start-up, following our implementation practice.
The Fourier Adapter offers parameter efficiency comparable F-Adapter while directly modelling spectral bases; however, its global trigonometric kernels incur greater FLOPs and memory (Table[3) and can amplify aliasing when K is large, echoing the empirical findings in Table 2 Nonetheless, for tasks dominated by periodic boundary conditions or sharp oscillations, serves as a principled, physics-aware drop-in replacement for projection-based adapters.
WaveAct Adapter Building on the learnable wavelet-based activation proposed in Zhao et al. [60] and the success of wavelet transforms in operator learning (50], devise the WaveAct- Activated Adapter (WaveAct Adapter). Unlike functional-basic adapters that alter the projection layers themselves, WaveAct Adapter keeps the standard bottleneck architecture but replaces the pointwise non-linearity with a paraneter-efffcieet WaveAct gate that superposes local sine and cosine responses. Formally, for an input activation x E Ed we compute
z W down X + bdown:
W down E I dehnttionncc xdin
(120)
E = a sin(2,) + cos(2z),
I,... dbottlenecks
(121)
dbostlonock
Uk
Wupl R 2, up
k 1, din
(122)
where a, b E are two learnable, shared scalars that modulate the sine / cosine mixture, and W E IR Xdbostiencck WaveAct thus provides compact spectral gate whose frequency content is dynamically tuned during training, requiring only 2 extra parameters irrespective of width.
Finally, a residual path equipped with learnable gain a restores the original dimensionality:
WaveAct- Addaptrrxxx) y X, a E TR.
(123)
We set (a. = 11, to start from an identity-like activation (sin + cos ~ 1 near the origin). Following Houlsby al. [17], Wap and its bias are initialised at zero so that = yields an exact identity map the beginning of training: Wdown follows Kaiming-uniform initialisation. Eq. [12l) equips each bottleneck coordinate with an adaptive wavelet kernel that can synthesise both low- and high-frequency components, while preserving the memory- and FLOP-profile of the vanilla adapter
40

---

p
Tmin
Tmmx
B
L2RE
% Param
FLOPs (G)
B dim
B2 dim
B3 dim
Ba dim
Bs dim
Bg dim
2
4
16
4
0.4523
1.91%
548.5307
13
8
5
2
16
6
0.4509
2.45%
548.7430
10
4
2
32
0.4191
3.40%
555.3716
22
11
1
16
32
4
0.4203
4.38%
556.8579
29
23
17
11
4
16
64
4
0.3885
6.76%
569.9026
44
23
16
16
1
64
4
0.4152
8.00%
572.8753
58
46
34
22
Table 15: Performance and configuration across bandwidth settings.
(Table 4). Empirically, WaveAct Adapter matches F-Adapter in memory usage and runtime yet trails slightly in L2RE (Table[5), suggesting that wavelet activations alone are insufficient to fully model the extreme high-freqeeney dynamics present in the Fourier domain. Nevertheless, its negligible parameter overhead and strong locality make it an attractive drop-in replacement when compute budgets are tight or periodicity is weak.
C.14 Ablation Study over Diverse Hyperparameter Settings for F-Adapter
We conducted extensive ablation studies on diverse hyperparameter settings using DPOT and F- Adapter on the 3D-Turbulance dataset.
Results in Table 15 indicate that hyperparameters primarily influence performance by modulating adapter capacity allocation across frequency bands. This adjustment effectively governs the model's overall capacity. Crucially, hyperparameters do not directly affect performance. Their impact is mediated through capacity allocation. Consequently, selecting appropriate capacity based computational resource constraints enables predictable performance outcomes. The magnitude of this impact remains relatively limited. This finding aligns with the insight presented in Table [ which shows that adapters nearly obey the scaling law. Our Band-Specific Bottleneck Allocation framework in Equation (6) maintains robust generalization across diverse tasks, while hyperparameters retain flexibility to accommodate available computational resources.
C.15 Design Details of the Frequency- Capacity Allocation Paradigm in the Transformer-Basee Poseidon Backbone
Motivation. Poseidon's backbone scOT learns solution operators Stt, a) with time-conditioned layer normalization and hierarchical shifted-window awareness without rewriting the model to operate in Fourier space. We estimate frequency content per layer, allocate a capacity budget across frequency bands, and realize the budget with two parameter- efficient routes: F-Adapter and F-LoRA. The design preserves Poseidon's 's native spatial pipeline and matinuous-inn-ine interface.
window attention stack. Our goal is to inject frequency
Preliminaries on frequency signals. Neural operator models such as FNO expose frequency channels through explicit spectral layers. In our transformer-based Poseidon backbone we keep all computations in the native spatial domain and recover frequency cues with lightweight probes: adjacent-token differences provide a proxy for high- versus fww.uereqeccy content in Linear layers, and a local real 2D FFT on Conv2d outputs yields a monibatch-normalizn
energy spectrum. The resulting per-band energies E, E [o. 1] act as data-dependent gates for our adapters. Bands are defined concentric partitions of the frequency plane, and the capacity assigned to each band follows the same rule as Equation (6), which allocates larger bottlenecks to low frequencies while reserving nonzero capacity for higher bands; the same schedule parameterizes ranks in F-LoRA.
Frequency estimation on Poseidon. We keep the base scOT layers intact. (i) For a Linear layer, we treat the token axis as a short ID sequence, compute adjacent- adjacent-token differences, and convert their magnitude to scalar frequency score per example, which we softmax into band weights т&. (ii) For Conv2d layer inside Poseidon's 's downsample or upsample paths, we run a local real 2D FFT on the convolution output, accumulate power within each annular band, and normalize to wp The estimates used only to gate adapters; the base layer remains purely spatial.
41

---

F-Adapter attaches bank of lightweight per-band adapters to each target layer while freezing the base weights.
F-Adapter.
B-1
ybase [(x),
Yb = Aol (huse)
yhase
+
Σ
b=0
Here C is the original Linear or Conv2d, and A, is a bottleneck MLP for Linear layers or xx1 conv bottleneck for Conv2d layers with width d,. The per-band outputs are combined by the data- dependent weights Th. Only the adapter parameters are trainable. This keeps inference identical to the base scOT path plus small residual branch and does not alter Poseidon's 's time conditioning.
F-LoRA. F-LoRA keeps the same frequency banding and gating but replaces each bottleneck adapter with LoRA-style low-rank updates that live on the frozen weight path. For Linear weight W E Rmxn
B-1
(ELORA(T) = Wx +
Σ Tbb BE,, b=0
A Bg ER
with trainable As, BG and fixed scale a. The rank r, follows the same capacity rule as d., which concentrates rank low frequencies while keeping nonzero rank for higher bands. For Conv2d use equivalent 1 X 1 factorizations per band in channel space. F-LoRA inherits the strong optimization behavior of LoRA transformer backbones and maintains small trainable footprint.
Implementation notes. Both mechanisms freeze the original Poseidon weights. F-Adapter attaches per-band residual branches whose last projection is initialized at zero avoid training instability at warm start. F-LoRA A initializes B₁ from a truncated normal and A, zero, which recovers the base in standard LoRA. The energy estimator and the gating T. differentiable
model at step but do not introduce global FFTs, so training throughput is close to that of the base model. The loss follows Poseidon's operator objective sampled times with L norm, which keeps supervision aligned with operator learning rather than single-step forecasting.
D Limitations
Our theoretical guarantees rest Now-freeneency-dmminane condition- namill,,, that the Fourier energy spectrum of the target operator decays sufficiently fast so that most variance is captured by the first few modes. This premise is supported across broad class of dissipative PDEs, including incompressible and compressible Navier-Stokes, reaction reaction-diffusion diffusion, shallow-water, and adifetionn systems, all of which exhibit steep inertial-range energy spectra. Nevertheless, its universality remains to be fully established-strongyy non-linear, multi-physics flows (e.g.. MHD turbulence reactive plasmas) may display flatter spectra that subtly stretch the separability premise in Proposition 3.2 3.3 A theoretically rigorous characterization how non-monotone or multi-modal spectra influence our approximation error bounds, and whether adaptive frequency-awan capacity allocation can re-establish similar guarantees in these settings, remains an open and compelling direction for future work.
E
Broader Impacts
The proposed frequency-adaptive adapter framework lowers the computational and memory footprint required to fine-tune large operator models (LOMs) for complex partialdifferentill-quutoo systems, potentially democratizing high-resolution scientific forecasting in climate science, aerospace design, and renewable-enee eneeryy optimization by enabling researchers with modest hardware to customise state-of state-of-the-e th- solvers. By concentrating learnable capacity on the most energetic spectral modes, our method also reduces training energy consumption relative to full fine-tuning, contributing to greener Al practice. At the same time, accelerated surrogate models for fluid and plasma dynamics could misused for strategic weapon design or proprietary industrial processes; we therefore commit to releasing code under rescarch-only licence and to incorporating provenance logging to
42

---

discourage dual-use. Finally, because adapter-based based surrogates may still propagate modelling bias when extrapolating beyond their training spectra, we encourage downstream practitioners couple workflows and to validate predictions against
our models with established trusted baselines before deployment in safety-critical settings.
43


Reviews:
[{"review_id": "5nUgMsyTQw", "reviewer": "Reviewer_G4t8", "summary": "This paper presents F-Adapter, a frequency-adaptive, parameter-efficient fine-tuning method specifically designed for pre-trained Large Operator Models (LOMs) in scientific machine learning. Unlike existing approaches such as LoRA, which are widely used in NLP and vision tasks, F-Adapter leverages the spectral structure of PDE solutions by allocating adapter capacity based on the distribution of energy across frequency bands. The authors provide both theoretical justifications 1) highlighting limitations of LoRA in Fourier-based architectures and 2) extensive empirical evaluations on 3D Navier–Stokes forecasting tasks. The proposed method consistently outperforms prior PEFT baselines while tuning fewer than 2% of the model parameters.", "questions": "1. Beyond FFT-Based models: Can F-Adapters be extended to non-Fourier architectures, such as graph-based operators or other models?\n\n2. Sensitivity to adapter scheduling: How sensitive is performance to the bottleneck allocation schedule? Would a learned or dynamically adapted schedule perform better?\n\n3. High-frequency regimes: While PDE solutions are often low-frequency dominated, some scenarios (e.g., turbulence, shocks) require high-frequency fidelity. How does F-Adapter perform in such regimes? Would be great to see comparison in those tasks.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Nice motivation. The paper offers a compelling analysis of why LoRA struggles in this domain, backed by provable lower bounds on approximation error and a universal approximation argument for adapters.\n2. The frequency-adaptive allocation of adapter capacity is elegant and grounded in the spectral properties of PDEs, which are clearly laid out both empirically and theoretically.\n3. Strong empirical results: across several challenging benchmarks, F-Adapter achieves state-of-the-art performance among PEFT methods, all with minimal parameter overhead.\n\nWeakness:\n1. Limited scope. The method and experiment mainly focus on a specific architecture (DPOT) and PDE family (Navier–Stokes). It would be helpful to understand how broadly the method applies to other operator learning settings, especially those not based on FFTs.\n2. Hyperparamter and tuning cost. The dimension–frequency scheduling is manually set and fixed across tasks. It might be case by case and require manual tuning. An adaptive one will be better.", "was_revised": true, "final_justification": "The authors addressed my questions and concerns, and thus I am willing to raise my score.", "rebuttal": "Thanks for the thoughtful comments!\n\n**Reply to W1 & Q1**\n\n**Firstly, FFTs are still the cornerstone of most LOMs, much as Multi‑Head Attention is for Large Language Models.** Representative FFT‑based LOMs include\n1. PreLowD (TMLR 2024), which relies on a factorized Fourier Neural Operator whose core loop is FFT, inverse FFT, and spectral convolution.\n2. UPS (TMLR 2024), whose first stage applies FNO spectral blocks to map the physical field into a token sequence processed by a language‑model body.\n3. CoDA‑NO (NeurIPS 2024), where on a uniform grid the key operators K, Q, V, M, and the integral operator I are all implemented with FNO components.\n4. OmniArch (ICML 2025), which uses a Fourier encoder to move spatial field values to the frequency domain.\n5. DPOT (ICML 2024), our main backbone, which is the only open‑source model in this family with more than 1B parameters and outperforms purely transformer‑based baselines such as MPP in their original paper.\n\nSecondly, although an FNO backbone provides direct access to frequency features, our main focus is on assigning each frequency band its own proper bottleneck dimension rather than strictly performing convolution in the frequency domain. This insight allows our F‑Adapter to extend naturally to non‑FFT architectures. On the pure transformer‑based **Poseidon** model, we estimate frequency energy for each Linear layer from adjacent‑token differences, and for each Conv2d layer we perform a local real 2‑D FFT on the convolution output to obtain an energy spectrum that guides the adapter’s weight generation. The PEFT adapter itself still operates in the native spatial domain. Capacities are allocated to bands according to their energy, following Eq(6) of our paper, which equips the model with frequency awareness. The accompanying table reports the resulting performance gains on 2D Shallow Water Equation Dataset.\n|Scheme on Poseidon|SWE-2D L2RE|%Param|\n|-|-|-|\n|Prompt Tuning|>1.0|0.07%|\n|LoRA|0.4010|2.07%|\n|RandLoRA|0.3134|2.07%|\n|Vanilla Adapter|0.6231|2.18%|\n|AdaLoRA|0.3756|2.32%|\n|HydraLoRA|0.3474|2.57%|\n|FiLM Adapter|0.4567|3.19%|\n|SVFT|0.6742|4.22%|\n|F-Adapter|0.4311|4.17%|\n|**F-LoRA**|**0.2746**|4.78%|\n|Full Fine-Tuning|0.1534|100%|\n\nWe observe that adapters deliver strong results when the base model is an FNO, yet their effectiveness declines sharply on a transformer backbone. In contrast, LoRA and its variants demonstrate robust performance on transformer backbones, reflecting established best practices in fine-tuning LLMs. But our F‑Adapter still narrows this gap by significantly improving adapter performance on transformers. **Building on this insight, we introduce F‑LoRA: it preserves the frequency‑based capacity allocation of F‑Adapter while replacing the bottleneck MLP with LoRA‑style low‑rank linear updates.** F‑LoRA achieves SOTA performance across a broad suite of PEFT methods in this setting.\n\n**Reply to W2 & Q2**\n\nWe would like to clarify that the hyperparameters that determine the model’s dimensionality may be difficult to make adaptive or learnable, since they fix the shape of weight tensors and cannot be updated during training. Nonetheless, we conducted extensive ablation studies on diverse hyperparameter settings using DPOT and F-Adapter on the 3D-Turbulance dataset. \n\n|p|r_min|r_max|B|L2RE|%Param|FLOPs(G)|B1 dim|B2 dim|B3 dim|B4 dim|B5 dim|B6 dim|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|2|4|16|4|0.4523|1.91%|548.5307|13|8|5|4|/|/|\n|2|4|16|6|0.4509|2.45%|548.7430|14|10|8|6|4|4|\n|2|8|32|4|0.4191|3.40%|555.3716|22|11|8|8|/|/|\n|1|16|32|4|0.4203|4.38%|556.8579|29|23|17|11|/|/|\n|4|16|64|4|0.3885|6.76%|569.9026|44|23|16|16|/|/|\n|1|16|64|4|0.4152|8.00%|572.8753|58|46|34|22|/|/|\n\nResults indicate that hyperparameters primarily influence performance by modulating adapter capacity allocation across frequency bands. This adjustment effectively governs the model's overall capacity. **Crucially, hyperparameters do not directly affect performance. Their impact is mediated through capacity allocation.** Consequently, selecting appropriate capacity based on computational resource constraints enables predictable performance outcomes. The magnitude of this impact remains relatively limited. This finding aligns with the insight presented in Table 1 of our original paper, which shows that adapters nearly obey the scaling law. Our Band-Specific Bottleneck Allocation framework (Eq 6) maintains robust generalization across diverse tasks, while hyperparameters retain flexibility to accommodate available computational resources.\n\n**Reply to Q3**\n\nFirstly, we have reported turbulence experiments in both Table 2 and Table 3. Now we supplement our study with a new set of comparative tests on the 2D shallow‑water equations (SWE-2D) and the 3D magnetohydrodynamic (MHD-3D) equations. In the SWE setting, phenomena such as hydraulic jumps and sharply non‑uniform interfaces generate shock‑like discontinuities and rich high‑wavenumber content, requiring high‑frequency fidelity for accurate capture. The MHD equations represent a class of PDEs that are characterized by broadband spectral behavior. Moreover, we only extracted 24 trajectories for training, aiming to evaluate the performance under extreme data scarcity conditions.\n\n|Scheme on DPOT|SWE-2D L2RE|SWE-2D %Param|MHD-3D L2RE|MHD-3D %Param|\n|-|-|-|-|-|\n|AdaLoRA|0.1061|0.70%|1.0022|0.69%|\n|HydraLoRA|0.0956|0.88%|0.9440|0.85%|\n|Prompt Tuning|0.1050|0.11%|0.9950|1.03%|\n|Vanilla Adapter|0.0902|0.48%|0.7226|1.16%|\n|FiLM Adapter|0.0162|0.57%|0.7593|1.30%|\n|RandLoRA|0.1568|1.05%|0.9800|1.36%|\n|LoRA|0.1081|1.40%|0.9845|1.37%|\n|F-Adapter(Ours)|**0.0116**|1.24%|**0.6341**|1.91%|\n|SVFT|0.0975|0.84%|1.0004|2.31%|\n|Full Fine-Tuning|0.0023|100%|0.4190|100%|\n\nOur results demonstrate that F‑Adapter still achieves SOTA. Intuitively, this stems from our frequency‑adaptive allocation of model capacity. **Rather than neglecting high‑frequency features, we partition the spectrum so that all frequencies are processed more efficiently and effectively.** F‑Adapter combines three complementary mechanisms: wide channels for low‑frequency components, lightweight residual connections for high‑frequency components, and a nonlinear universal approximator to concentrate limited parameters on the most critical features. High‑frequency information, although important, is energetically sparse and can be captured with small bottleneck modules. These modules both preserve essential details and impose an implicit regularization. Consequently, F‑Adapter excels in turbulent flows and other scenarios dominated by high‑frequency dynamics."}, {"review_id": "vObCjikDvI", "reviewer": "Reviewer_dyWR", "summary": "This paper studies the fine-tuning problems of large operator models in scientific machine learning (SciML). It provides an empirical analysis and theoretical insights on the spectral structure of PDE solutions. On the methods and experiments side, the authors consider DPO, which uses the Fourier Neural Operator (FNO) as a backbone, as an example for analyzing the fine-tuning performance. They introduce F-Adapter, a frequency-adaptive adapter for fine-tuning. The empirical results have shown the superiority of the proposed method in terms of generalization and spectral fidelity compared to the existing fine-tuning methods.", "questions": "- LoRA is not designed for improving the fine-tuning performance but to improve computational efficiency. How does the computational efficiency look in the proposed F-Adaptor method compared to LoRA?\n- Since the authors focus on the fine-tuning of large models, why don’t you also consider supervised fine-tuning (SFT)? There are some SFT works in SciML. [1]\n\n[1] Chen, Wuyang, et al. \"Data-efficient operator learning via unsupervised pretraining and in-context learning.\" Advances in Neural Information Processing Systems 37 (2024): 6213-6245.", "limitations": "The authors have addressed their limitations and under-explored aspects in the paper, such as the generalizability of the premise for F-Adapter.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n- This paper investigates an important problem of post-tuning in large-scale scientific machine learning. \n- This paper provides an empirical analysis and theoretical insights into the fine-tuning methods in terms of the energy distribution of PDE solutions.\n- This paper is well-written and well-organized.  \n\nWeaknesses\n- The scope of the studied large models in SciML is relatively limited. There are a bunch of large SciML models currently, as the authors also mentioned in Related Works. Some of them are based on transformer architectures without the Fourier transform or FNO. The empirical analysis and the proposed methods are more focused on DPOT, which uses FNO as a backbone. Therefore, I am not sure if the authors still have the same observations or conclusions after trying some other transformer-based models. \n- A follow-up concern is that the definition of larger operator models is kind of unclear to me. I was thinking of a large-scale model using neural operators as backbones. However, in the related work, the authors also include MPP and Poseidon, which use transformers as backbones, as large operator models. In other words, I was confused by this paragraph, especially “Building on this, …, these large-scale operator models, built upon the FNO framework,...” Many of the cited papers clearly didn’t use FNO. [1,2,3]\n\n[1] McCabe, Michael, et al. \"Multiple physics pretraining for physical surrogate models.\" arXiv preprint arXiv:2310.02994 (2023).\n\n[2] Hao, Zhongkai, et al. \"Dpot: Auto-regressive denoising operator transformer for large-scale pde pre-training.\" arXiv preprint arXiv:2403.03542 (2024).\n\n[3] Li, Zongyi, et al. \"Fourier neural operator for parametric partial differential equations.\" arXiv preprint arXiv:2010.08895 (2020).", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the thoughtful comments!\n\n**Reply to W1**\n\n**Firstly, FFTs (FNO) are still the cornerstone of most LOMs, much as Multi‑Head Attention is for Large Language Models.** Representative FFT‑based LOMs include\n1. PreLowD (TMLR 2024), which relies on a factorized Fourier Neural Operator whose core loop is FFT, inverse FFT, and spectral convolution.\n2. UPS (TMLR 2024), whose first stage applies FNO spectral blocks to map the physical field into a token sequence processed by a language‑model body.\n3. CoDA‑NO (NeurIPS 2024), where on a uniform grid the key operators K, Q, V, M, and the integral operator I are all implemented with FNO components.\n4. OmniArch (ICML 2025), which uses a Fourier encoder to move spatial field values to the frequency domain.\n5. DPOT (ICML 2024), our main backbone, which is the only open‑source model in this family with more than 1B parameters and outperforms purely transformer‑based baselines such as MPP in their original paper.\n\nSecondly, although an FNO backbone provides direct access to frequency features, our main focus is on assigning each frequency band its own proper bottleneck dimension rather than strictly performing convolution in the frequency domain. This insight allows our F‑Adapter to extend naturally to non‑FFT architectures. On the pure transformer‑based **Poseidon** model, we estimate frequency energy for each Linear layer from adjacent‑token differences, and for each Conv2d layer we perform a local real 2‑D FFT on the convolution output to obtain an energy spectrum that guides the adapter’s weight generation. The adapter itself still operates in the native spatial domain. Capacities are allocated to bands according to their energy, following Eq(6) of our paper, which equips the model with frequency awareness. The accompanying table reports the resulting performance gains on 2D Shallow Water Equation Dataset.\n|Scheme on Poseidon|SWE-2D L2RE|%Param|\n|-|-|-|\n|Prompt Tuning|>1.0|0.07%|\n|LoRA|0.4010|2.07%|\n|RandLoRA|0.3134|2.07%|\n|Vanilla Adapter|0.6231|2.18%|\n|AdaLoRA|0.3756|2.32%|\n|HydraLoRA|0.3474|2.57%|\n|FiLM Adapter|0.4567|3.19%|\n|SVFT|0.6742|4.22%|\n|F-Adapter|0.4311|4.17%|\n|**F-LoRA**|**0.2746**|4.78%|\n|Full Fine-Tuning|0.1534|100%|\n\nWe observe that adapters deliver strong results when the base model is an FNO, yet their effectiveness declines sharply on a transformer backbone. In contrast, LoRA and its variants demonstrate robust performance on transformer backbones, reflecting established best practices in fine-tuning LLMs. But our F‑Adapter still narrows this gap by significantly improving adapter performance on transformers. **Building on this insight, we introduce F‑LoRA: it preserves the frequency‑based capacity allocation of F‑Adapter while replacing the bottleneck MLP with LoRA‑style low‑rank linear updates.** F‑LoRA achieves SOTA performance across a broad suite of PEFT methods in this setting.\n\n**Reply to W2**\n\nYou raise a valid point. In our paper, we define a Large Operator Model (LOM) as a pre‑trained foundational model that reaches the parameter scale commonly associated with “large models” and learns solution operators for PDEs. The ambiguity you noted comes from our wording: we meant only those LOMs built on the FNO backbone, not models using transformer‑based architectures. We will revise this sentence in the camera ready version to make it clear that “large‑scale operator models” in that sentence refers exclusively to FNO‑based LOMs. \n\n**Reply to Q1**\n\nWhile LoRA is not designed to improve performance, it serves as a viable option when memory and computational budgets are limited. Therefore, the goal of this work is to find a better parameter-efficient fine-tuning method that can achieve stronger performance under the same memory and computational constraints, rather than seeking improvements over the full-parameter fine-tuning method. **Please refer to Table 9 in the appendix of our paper.** When compared to LoRA, F‑Adapter achieves comparable FLOPs and per‑step inference time, while delivering superior performance.\n\n**Reply to Q2**\n\nSFT adapts a pre‑trained large model to a downstream task by updating all of its parameters so as to minimize the supervised loss on labeled data. PEFT pursues the same objective using labeled data but only introduces or updates a small set of extra parameters, while keeping the backbone weights fixed. In other words, SFT adjusts every model weight, whereas PEFT makes only minimal changes, making it a more resource friendly variant of supervised fine tuning. Both methods share the goal of maximizing task performance; their sole distinction lies in the fraction of parameters that are trainable. We acknowledge the relevance of Chen et al. (2024) [1], which explores data-efficient operator learning via unsupervised pretraining and in-context learning (ICL) in SciML. While their work focuses on reducing labeled data dependency through ICL, a non-fine-tuning approach, our study prioritizes PEFT for scenarios where task-specific labeled data exists but full parameter updates are computationally prohibitive. **As shown in Table 2 and Table 9, we have empirically compared the performance and computational cost of full fine tuning against several PEFT techniques.**"}, {"review_id": "JsXLjSZvnF", "reviewer": "Reviewer_gNLD", "summary": "### Summary\nThe paper targets parameter-efficient fine-tuning (PEFT) for large Fourier-Neural-Operator (FNO) models.  \n1. Low-rank LoRA severely underperforms when adapting a 1 B-parameter pretrained FNO on 3-D Navier–Stokes forecasting, whereas a simple two-layer adapter converges faster and attains lower error.  \n2.  The authors prove a depth-amplified approximation lower-bound for stacked low-rank updates, while two-layer MLP adapters possess universal approximation capacity in Fourier layers and achieve exponentially decaying error when focused on low-frequency modes.  \n3.  They propose a Frequency-Adaptive Adapter (F-Adapter). Each Fourier layer is partitioned into radial frequency bands; the adapter hidden width  $ r_b = r_{\\min} + (r_{\\max}-r_{\\min}) \\Bigl(1-\\tfrac{f_b}{M}\\Bigr)^{p} $  assigns larger capacity to low-frequency modes carrying most PDE energy, smaller capacity to high-frequency modes.  \n4.  On multiple 3-D Navier–Stokes benchmarks, F-Adapter tunes \\< 2 % of weights yet surpasses LoRA and other PEFT baselines, establishing new state-of-the-art L2 relative errors.", "questions": "### Questions for Authors\n1. How would F-Adapter perform on operator networks without explicit FFTs, or on PDEs with broadband spectra (e.g., MHD)?  \n2.   How robust is performance to the number of bands and the exponent \\(p\\)? Could these be learned automatically?  \n3.  What is the accuracy of full-parameter fine-tuning on the same tasks, and how close does F-Adapter come?  \n4. Could combining LoRA (for linear layers) with non-linear adapters reduce overhead further while mitigating LoRA’s depth error?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n1.  First systematic PEFT study for large operator models in scientific ML.  \n2.  Frequency-based capacity allocation is justified by spectral energy analysis and proved beneficial.  \n3.   Formal lower-bound for LoRA and universal-approximation result for adapters clarify why non-linear adapters excel.  \n4. Comprehensive benchmarks, ablations, FLOP analyses, and convergence plots demonstrate robustness.  \n5.  < 2 % trainable parameters with negligible FLOP increase enables low-resource adaptation of 1 B-parameter PDE solvers.\n\n### Weaknesses\n1.  Experiments cover only Fourier-based Navier–Stokes; generality to other PDEs or non-spectral backbones is untested.  \n2.  Band count, \\(r_{\\min}, r_{\\max}, p\\) require tuning; sensitivity analysis is brief.  \n3.  No full fine-tuning reference; few comparisons to alternative PEFT schemes (e.g., BitFit).  \n4.  Combines known components (adapters, spectral splits); methodological leap is domain-specific rather than fundamentally new.\n\n### Novelty\nNovel within SciML: the first PEFT method that exploits Fourier energy profiles to size adapters. Theoretical framing and empirical validation are original for this domain, though the underlying ideas (adapters, frequency banding) are incremental relative to broader ML literature.", "was_revised": true, "final_justification": "The questions and concern have been addressed. I believe the sucessful application of parameter fine-tuning in operator learning is interesting.", "rebuttal": "Thanks for the thoughtful comments!\n\n**Reply to W1**\n\nThanks for your concerns. We would like to discuss our exploration on non-spectral backbones here.\n\n**Firstly, FFTs (FNO) are still the cornerstone of most LOMs, much as Multi‑Head Attention is for Large Language Models.** Representative FFT‑based LOMs include\n1. PreLowD (TMLR 2024), which relies on a factorized Fourier Neural Operator whose core loop is FFT, inverse FFT, and spectral convolution.\n2. UPS (TMLR 2024), whose first stage applies FNO spectral blocks to map the physical field into a token sequence processed by a language‑model body.\n3. CoDA‑NO (NeurIPS 2024), where on a uniform grid the key operators K, Q, V, M, and the integral operator I are all implemented with FNO components.\n4. OmniArch (ICML 2025), which uses a Fourier encoder to move spatial field values to the frequency domain.\n5. DPOT (ICML 2024), our main backbone, which is the only open‑source model in this family with more than 1B parameters and outperforms purely transformer‑based baselines such as MPP in their original paper.\n\nSecondly, although an FNO backbone provides direct access to frequency features, our main focus is on assigning each frequency band its own proper bottleneck dimension rather than strictly performing convolution in the frequency domain. This insight allows our F‑Adapter to extend naturally to non‑FFT architectures. On the pure transformer‑based **Poseidon** model, we estimate frequency energy for each Linear layer from adjacent‑token differences, and for each Conv2d layer we perform a local real 2‑D FFT on the convolution output to obtain an energy spectrum that guides the adapter’s weight generation. The adapter itself still operates in the native spatial domain. Capacities are allocated to bands according to their energy, following Eq(6) of our paper, which equips the model with frequency awareness. The accompanying table reports the resulting performance gains on 2D Shallow Water Equation Dataset.\n|Scheme on Poseidon|SWE-2D L2RE|%Param|\n|-|-|-|\n|Prompt Tuning|>1.0|0.07%|\n|LoRA|0.4010|2.07%|\n|RandLoRA|0.3134|2.07%|\n|Vanilla Adapter|0.6231|2.18%|\n|AdaLoRA|0.3756|2.32%|\n|HydraLoRA|0.3474|2.57%|\n|FiLM Adapter|0.4567|3.19%|\n|SVFT|0.6742|4.22%|\n|F-Adapter|0.4311|4.17%|\n|**F-LoRA**|**0.2746**|4.78%|\n|Full Fine-Tuning|0.1534|100%|\n\nWe observe that adapters deliver strong results when the base model is an FNO, yet their effectiveness declines sharply on a transformer backbone. In contrast, LoRA and its variants demonstrate robust performance on transformer backbones, reflecting established best practices in fine-tuning LLMs. But our F‑Adapter still narrows this gap by significantly improving adapter performance on transformers. **Building on this insight, we introduce F‑LoRA: it preserves the frequency‑based capacity allocation of F‑Adapter while replacing the bottleneck MLP with LoRA‑style low‑rank linear updates.** F‑LoRA achieves SOTA performance across a broad suite of PEFT methods in this setting.\n\n**Reply to W2**\n\nWe conducted extensive ablation studies on diverse hyperparameter settings using DPOT and F-Adapter on 3D-Turbulance dataset. \n|p|r_min|r_max|B|L2RE|%Param|FLOPs(G)|B1 dim|B2 dim|B3 dim|B4 dim|B5 dim|B6 dim|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|2|4|16|4|0.4523|1.91%|548.5307|13|8|5|4|/|/|\n|2|4|16|6|0.4509|2.45%|548.7430|14|10|8|6|4|4|\n|2|8|32|4|0.4191|3.40%|555.3716|22|11|8|8|/|/|\n|1|16|32|4|0.4203|4.38%|556.8579|29|23|17|11|/|/|\n|4|16|64|4|0.3885|6.76%|569.9026|44|23|16|16|/|/|\n|1|16|64|4|0.4152|8.00%|572.8753|58|46|34|22|/|/|\n\nThe first row details the default hyperparameter configuration applied across all experiments in our original manuscript. Results indicate that hyperparameters primarily influence performance by modulating adapter capacity allocation across frequency bands. This adjustment effectively governs the model's overall capacity. **Crucially, hyperparameters do not directly affect performance. Their impact is mediated through capacity allocation.** Consequently, selecting appropriate capacity based on computational resource constraints enables predictable performance outcomes. The magnitude of this impact remains relatively limited. This finding aligns with the insight presented in Table 1 of our original paper, which shows that adapters nearly obey the scaling law. Our Band-Specific Bottleneck Allocation framework (Eq 6) maintains robust generalization across diverse tasks, while hyperparameters retain flexibility to accommodate available computational resources.\n\n**Reply to W3**\n\n**In Table 2, we present a comprehensive comparison between full fine‑tuning, our F Adapter, and 8 alternative PEFT schemes.** This evaluation includes methods such as LoRA, Adapter, and Prompt Tuning. Across most benchmarks, all eight approaches perform better than BitFit. Since our main manuscript focuses on frequency‑domain PEFT designs and BitFit only updates bias parameters without delivering targeted improvements or new insights, we did not include BitFit as a baseline.\n\n**Reply to W4**\n\nWe appreciate this perspective on methodological novelty. While our framework integrates established concepts like adapters and spectral decomposition, its core contribution lies in **the first unified theoretical and empirical framework** comprehensively addressing domain-specific fine-tuning challenges in SciML. Critically, our frequency-aware parameter allocation principle (Eq. 6) extends beyond Fourier-based architectures. When applied to transformer based models like Poseidon through F-LoRA, it still achieves SOTA results, demonstrating broader applicability beyond spectral architectures. This evidence confirms that our high-level methodology generalizes to diverse architectures and tasks.  The details have been shown in **Reply to W1.** **These findings reveal a transferable design principle: task performance in SciML is closely linked to the efficient division of labor across features of varying frequencies, regardless of the base architecture.** While instantiated here for PDE-solving, this paradigm provides actionable insights for other tasks with strong frequency dependencies.\n\n**Reply to Q1**\n\nFirstly, please refer to **Reply to W1** for the performance of F-Adapter on operator networks without explicit FFTs. Secondly, we supplement our study with a new set of comparative tests on the 2D shallow‑water equations (SWE) and the 3D magnetohydrodynamic equations (MHD). In the SWE setting, phenomena such as hydraulic jumps and strongly non‑uniform interfaces produce shocklike discontinuities and pronounced high‑wavenumber content. The MHD represents a class of PDEs characterized by broadband spectral behavior. Moreover, we only extracted 24 trajectories for HHD training, aiming to evaluate the performance under extreme data scarcity conditions.\n|Scheme on DPOT|SWE-2D L2RE|SWE-2D %Param|MHD-3D L2RE|MHD-3D %Param|\n|-|-|-|-|-|\n|AdaLoRA|0.1061|0.70%|1.0022|0.69%|\n|HydraLoRA|0.0956|0.88%|0.9440|0.85%|\n|Prompt Tuning|0.1050|0.11%|0.9950|1.03%|\n|Vanilla Adapter|0.0902|0.48%|0.7226|1.16%|\n|FiLM Adapter|0.0162|0.57%|0.7593|1.30%|\n|RandLoRA|0.1568|1.05%|0.9800|1.36%|\n|LoRA|0.1081|1.40%|0.9845|1.37%|\n|F-Adapter(Ours)|**0.0116**|1.24%|**0.6341**|1.91%|\n|SVFT|0.0975|0.84%|1.0004|2.31%|\n|Full Fine-Tuning|0.0023|100%|0.4190|100%|\n\nOur results demonstrate that F‑Adapter still achieves SOTA. Intuitively, this stems from our frequency‑adaptive allocation of model capacity. **Rather than neglecting high‑frequency features, we partition the spectrum so that all frequencies are processed more efficiently and effectively.** F‑Adapter combines three complementary mechanisms: wide channels for low‑frequency components, lightweight residual connections for high‑frequency components, and a nonlinear universal approximator to concentrate limited parameters on the most critical features. High‑frequency information, although important, is energetically sparse and can be captured with small bottleneck modules. These modules both preserve essential details and impose an implicit regularization. Consequently, F‑Adapter excels in turbulent flows and other scenarios dominated by high‑frequency dynamics.\n\n**Reply to Q2**\n\nWe would like to clarify that the hyperparameters that determine the model’s dimensionality may be difficult to make adaptive or learnable, since they fix the shape of weight tensors and cannot be updated during training. Nonetheless, we conducted extensive ablation studies on these settings in our **Reply to W2**. \n\n**Reply to Q3** \n\n**Table 2** in our manuscript includes the accuracy results of full parameter fine tuning for reference, and in several cases F-Adapter attains higher performance.\n\n**Reply to Q4**\n\nThis is a valuable suggestion that aligns with our interest in refining PEFT strategies for SciML. Notably, **we have already conducted preliminary experiments on frequency-adaptive LoRA specifically tailored for Transformer-based large operator models, with details available in *Reply to W1.*** We believe these frequency-adaptive LoRA components could potentially be integrated with our current framework to leverage the strengths of both linear and non-linear adaptations. However, such integration requires careful design and extensive validation to ensure compatibility and performance gains, which would demand additional time beyond the current scope. Thus, we plan to pursue this integration as a dedicated direction in our future research."}, {"review_id": "wf9gOrqjuS", "reviewer": "Reviewer_acdm", "summary": "This paper introduces F-Adapter, a frequency-adaptive, parameter-efficient fine-tuning method tailored for foundation models in scientific machine learning. The F-Adapter allocates more capacity to low-frequency modes and less to high-frequency ones. The authors provide both theoretical analysis and empirical validation, showing that F-Adapter outperforms LoRA and other baselines on challenging 3D Navier–Stokes tasks.", "questions": "- Have the authors considered directly comparing the predicted velocity spectrum to the ground-truth spectrum?\n\n- Since the proposed F-Adapter operates in the Fourier domain, it appears tailored to FNO-based architectures. Could the authors comment on the feasibility of extending this approach to non-Fourier models or more general operator learning frameworks?\n\n- Would it be possible to include visual comparisons (e.g., slices or isosurfaces) of the predicted and ground-truth velocity fields? This would help assess whether the model captures the spatial structure of the flow beyond what L2 error metrics reveal.", "limitations": "The authors have addressed some of the limitations in the appendix.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n- The explanation of the lower bound for block-wise LoRA in Section 3.2 is clearly presented and adds valuable theoretical insight.\n\n- The theoretical framework presented by the authors is both rigorous and well-motivated, offering an intuitive perspective. Overall, the paper stands out for its principled integration of spectral theory into adapter design. By aligning model capacity with the energy distribution of PDE solutions, the approach seem to be both physically grounded and practically effective.\n\n- The authors demonstrate the computational efficiency of their method, alongside strong empirical results that support its practical utility.\n\n- The ablation studies are well-designed, providing interesting insights into both the frequency-based adapter scheduling and the choice of Fourier-domain adapter types.\n\n\nWeaknesses\n\n- In the 3D Navier–Stokes setup, the DPOT-1B model is fine-tuned on a difficult dataset, and the relative errors are high ($\\geq 40\\%$). The paper lacks visualizations comparing predictions and ground truth, making it unclear whether the models (using either LoRA or adapter-based tuning) simply converge to the training distribution mean with limited spectral correction. Such large errors raise concerns about the limitations of the pretrained model itself, and whether the adapter's performance can be meaningfully assessed in this setting.\n\n- Although the method is benchmarked against several baselines, only two very challenging datasets are used for fine-tuning. Given that even full fine-tuning yields high errors, it remains unclear how the method would perform on a broader range of PDE problems.\n\n- The proposed method relies on injecting adapters in the spectral domain after FFT, meaning it is only applicable to FNO-style architectures. However, prior work [1] has shown that FNOs scale poorly with increasing training data, casting doubt on the broader applicability of this method compared to more flexible techniques like LoRA.\n\n- The paper does not evaluate performance under extreme data scarcity, such as fine-tuning with only 20-30 trajectories. It is unclear whether the method would remain effective under such low-data regimes, which is the most interesting regime in SciML, where we face data scarcity.\n\n- In turbulent flow modeling, capturing fine-scale structures is essential due to the multi-scale nature of the system, the energy cascade process, and the presence of extreme events. Prioritizing only the energy-dominant modes may hinder accurate prediction of critical small-scale dynamics. I recommend that the authors compare the predicted velocity spectrum with the ground-truth spectrum to assess whether the individual modes are accurately captured.\n    \n---\n\n[1] Herde, M., Raonic, B., Rohner, T., Käppeli, R., Molinaro, R., de Bézenac, E., \\& Mishra, S. (2024). Poseidon: Efficient foundation models for pdes. Advances in Neural Information Processing Systems, 37, 72525-72624.\n\\end{itemize}", "was_revised": true, "final_justification": "The authors addressed most of the points I raised in my review, so I am increasing my score to 4. However, I believe a more appropriate score would be 4.5, as it is stronger than a borderline accept.", "rebuttal": "Thanks for the thoughtful comments!\n\n**Reply to W1 & Q3**\n\nFirst, we evaluated a simpler test case for the base model: the 2D shallow water equations (SWE-2D) from PDEBench. **This choice removes any bias from base model limitations when assessing PEFT methods.** The results show that F Adapter achieves the lowest relative error (≈1.6%).\n\n|Scheme on DPOT|SWE-2D L2RE|SWE-2D %Param|\n|-|-|-|\n|AdaLoRA|0.1061|0.70%|\n|HydraLoRA|0.0956|0.88%|\n|Prompt Tuning|0.1050|0.11%|\n|Vanilla Adapter|0.0902|0.48%|\n|FiLM Adapter|0.0162|0.57%|\n|RandLoRA|0.1568|1.05%|\n|LoRA|0.1081|1.40%|\n|F-Adapter(Ours)|**0.0116**|1.24%|\n|SVFT|0.0975|0.84%|\n|Full Fine-Tuning|0.0023|100%|\n\nSecond, we carried out visual comparisons of predicted and ground truth velocity fields using slice plots and isosurfaces. Unfortunately, rebuttal policy prevents us from including these figures here. In broad terms, the F‑Adapter preserves the filament‑like vortical structures and reproduces both high‑ and low‑energy regions, with only small local deviations in amplitude. LoRA, in contrast, yields coarse block‑shaped patches with muted intensities and loses most fine‑scale features, suggesting a collapse toward a spatially averaged flow. These qualitative visual observations support our quantitative error analysis and show that the adapter retains considerably more spectral detail than LoRA in this challenging regime. **We will present the full set of visual results in the camera ready version.**\n\n**Reply to W2 & W4**   \n\nFirstly, our original manuscript actually evaluated on three very challenging datasets (see Appendix C.5). \n\nBesides the SWE-2D introduced in our **Reply to W1 & Q3,** we now add another experiment on the 3D magnetohydrodynamic equations (MHD-3D) under extreme data scarcity. We adopt the GL-FNO[1] data processing protocol but train on only 24 trajectories. \n|Scheme on DPOT|MHD-3D L2RE|MHD-3D %Param|\n|-|-|-|\n|AdaLoRA|1.0022|0.69%|\n|HydraLoRA|0.9440|0.85%|\n|Prompt Tuning|0.9950|1.03%|\n|Vanilla Adapter|0.7226|1.16%|\n|FiLM Adapter|0.7593|1.30%|\n|RandLoRA|0.9800|1.36%|\n|LoRA|0.9845|1.37%|\n|F-Adapter(Ours)|**0.6341**|1.91%|\n|SVFT|1.0004|2.31%|\n|Full Fine-Tuning|0.4190|100%|\n\nThe results show that even with such limited data, our F Adapter continues to achieve substantially higher accuracy than the alternative PEFT schemes which fail to adapt under these conditions.\n\n[1] Du, Yutao, et al. \"Global-local Fourier Neural Operator for Accelerating Coronal Magnetic Field Model.\" 2024 \n\n**Reply to W3 & Q2**\n\n**Firstly, FFTs (FNO) are still the cornerstone of most LOMs, much as Multi‑Head Attention is for Large Language Models.** With elaborate design, it can still stably scale up and unleash powerful pre-trained capabilities. Representative FFT‑based LOMs include\n1. PreLowD (TMLR 2024), which relies on a factorized Fourier Neural Operator whose core loop is FFT, inverse FFT, and spectral convolution.\n2. UPS (TMLR 2024), whose first stage applies FNO spectral blocks to map the physical field into a token sequence processed by a language‑model body.\n3. CoDA‑NO (NeurIPS 2024), where on a uniform grid the key operators K, Q, V, M, and the integral operator I are all implemented with FNO components.\n4. OmniArch (ICML 2025), which uses a Fourier encoder to move spatial field values to the frequency domain.\n5. DPOT (ICML 2024), our main backbone, which is the only open‑source model in this family with more than 1B parameters and outperforms purely transformer‑based baselines such as MPP in their original paper.\n\nSecondly, although an FNO backbone provides direct access to frequency features, our main focus is on assigning each frequency band its own proper bottleneck dimension rather than strictly performing convolution in the frequency domain. This insight allows our F‑Adapter to extend naturally to non‑FFT architectures. On the pure transformer‑based **Poseidon** model, we estimate frequency energy for each Linear layer from adjacent‑token differences, and for each Conv2d layer we perform a local real 2‑D FFT on the convolution output to obtain an energy spectrum that guides the adapter’s weight generation. The adapter itself still operates in the native spatial domain. Capacities are allocated to bands according to their energy, following Eq(6) of our paper, which equips the model with frequency awareness. The accompanying table reports the resulting performance gains on 2D Shallow Water Equation Dataset.\n|Scheme on Poseidon|SWE-2D L2RE|%Param|\n|-|-|-|\n|Prompt Tuning|>1.0|0.07%|\n|LoRA|0.4010|2.07%|\n|RandLoRA|0.3134|2.07%|\n|Vanilla Adapter|0.6231|2.18%|\n|AdaLoRA|0.3756|2.32%|\n|HydraLoRA|0.3474|2.57%|\n|FiLM Adapter|0.4567|3.19%|\n|SVFT|0.6742|4.22%|\n|F-Adapter|0.4311|4.17%|\n|**F-LoRA**|**0.2746**|4.78%|\n|Full Fine-Tuning|0.1534|100%|\n\nWe observe that adapters deliver strong results when the base model is an FNO, yet their effectiveness declines sharply on a transformer backbone. In contrast, LoRA and its variants demonstrate robust performance on transformer backbones, reflecting established best practices in fine-tuning LLMs. But our F‑Adapter still narrows this gap by significantly improving adapter performance on transformers. **Building on this insight, we introduce F‑LoRA: it preserves the frequency‑based capacity allocation of F‑Adapter while replacing the bottleneck MLP with LoRA‑style low‑rank linear updates.** F‑LoRA achieves SOTA performance across a broad suite of PEFT methods in this setting.\n\n**Reply to W5 & Q1**\n\nBelow we present additional spectral results on the 3‑D Turbulence dataset.\n To quantify how well individual scales are reproduced, we report two metrics:\n\n1. Root mean square logarithmic error (RMSLE) of the energy spectrum\n$$\n\\text{RMSLE}\\_{E}\n   = \\sqrt{\\frac{1}{N}\\sum\\_{i=1}^{N}\n      \\bigl(\\log_{10}E_{\\text{pred}}(k_i)\n           -\\log_{10}E_{\\text{DNS}}(k_i)\\bigr)^{2}}\n$$\nWe compare the predicted and DNS energy spectra shell by shell in wave number space. This metric measures their discrepancy on a logarithmic scale and ensures that both low and high wave number bands receive equal weight.\n\n2. Relative error of the total kinetic energy\n$$\nE_{\\text{tot}}\n   = \\int_{0}^{k_{\\max}} E(k) \\mathrm{d}k,\n\\qquad\n\\text{RelErr}\\_{E}\n   = \\frac{\\lvert E_{\\text{tot,pred}} - E_{\\text{tot,DNS}}\\rvert}\n          {E_{\\text{tot,DNS}}}\\times 100\\\\%\n$$\nHere $E_{\\text{tot}}$ is the area under the spectrum. Conservation of total energy confirms that this quantity is directly proportional to the volume‑averaged turbulent kinetic energy.\n\n|Scheme|$RMSLE_E$|$RelErr_E$|%Param|\n|-|-|-|-|\n|Vanilla Adapter|0.9186|10.55%|1.16%|\n|LoRA|1.936|435.09%|1.37%|\n|F-Adapter(Ours)|**0.9095**|**6.12%**|1.91%|\n|Full Fine-Tuning|0.3208|0.21%|100%|\n\nIn addition, the camera‑ready version will include a plot of $E(k)$ and the DNS spectrum on logarithmic axes. **The experimental results demonstrate that our F‑adapter significantly outperforms both LoRA and the strongest competing baseline, the Vanilla Adapter, in accurately capturing the energy spectrum.**"}]

Author rebuttal:

Meta-review: This paper introduces F-Adapter, a frequency-adaptive PEFT method for large Fourier Neural Operator models. By allocating adapter capacity according to the spectral energy of PDE solutions, it prioritizes low-frequency modes while limiting high-frequency capacity.
After reviewing the rebuttal, the reviewers reached a consensus that the paper demonstrates a principled design, rigorous theoretical grounding, and effective empirical validation with extensive ablations. Most experimental concerns raised earlier were addressed. The authors are encouraged to incorporate the references suggested by the reviewers and discuss them in the related works.
Overall, the AC recommends acceptance.
Decision: Accept (poster)