DeltaPhi: Physical States Residual Learning for Neural Operators in Data-Limited PDE Solving
Authors: Xihang Yue
Yi Yang
Linchao Zhu
Venue: NeurIPS 2025 poster ()
Keywords: PDE
Operator Learning
Primary area: machine_learning_for_sciences
TL;DR: This work introduces a novel framework that learns physical sample residuals instead of direct mappings for PDE solving, significantly improving neural operator generalization in data-limited scenarios.

Abstract: The limited availability of high-quality training data poses a major obstacle in data-driven PDE solving, where expensive data collection and resolution constraints severely impact the ability of neural operator networks to learn and generalize the underlying physical system. To address this challenge, we propose DeltaPhi, a novel learning framework that transforms the PDE solving task from learning direct input-output mappings to learning the residuals between similar physical states, a fundamentally different approach to neural operator learning. This reformulation provides implicit data augmentation by exploiting the inherent stability of physical systems where closer initial states lead to closer evolution trajectories. DeltaPhi is architecture-agnostic and can be seamlessly integrated with existing neural operators to enhance their performance. Extensive experiments demonstrate consistent and significant improvements across diverse physical systems including regular and irregular domains, different neural architectures, multiple training data amount, and cross-resolution scenarios, confirming its effectiveness as a general enhancement for neural operators in data-limited PDE solving.

DeltaPhi: Physical States Residual Learning for Neural Operators in Data-Limited PDE Solving
Xihang Yue Yi Yang 1,, Linchao Zhu
College of Computer Science and Technology. Zhejiang University Intelligence, Zhejiang University
The State Key Lab of wwwp/yuexchanngeellaaaaaa. https: // /// git......... com
Abstract
The limited availability of high-quality training data poses a major obstacle in data- driven PDE solving, where expensive data collection and resolution constraints severely impact the ability of neural operator networks lear and generalize the underlying physical system. To address this challenge. we propose DeltaPhi, a novel learning framework that transforms the PDE solving task from learning direct input-output mappings to learning the residuals between similar physical states, fundamentally different approach neural operator learning. This refor- mulation provides implicit data augmentation by exploiting the inherent stability of physical systems where closer initial states lead to closer evolution trajectories. DeltaPhi is architecture-agnssii neural operators to enhance their performance. Extensive experiments demonstrate consistent and significant improvements across diverse physical systems including regular and irregular domains, different neural architectures, multiple training data amount, and cross-resolution scenarios, confirming its effectiveness general enhancement for neural operators in data-limited PDE solving.
and can be seamlessly integrated with existing
Introduction
Due to the lack of analytical solutions, solving complex Partial Differential Equations (PDEs) e.g. Navier-Stokes traditionally relies on numerical simulations based on ultra-fine grid division, which consumes expensive computational resources and time. This computational bottleneck has motivated the development of machine learning based PDE solving approaches [1, 2, 3, 4, 5]. In particular, neural operator learning [1, 2, 6] has emerged as promising direction by directly learning the mapping between input-output function fields, enabling efficient solution prediction through neural network forward computation.
critical challenge in neural operators the severe scarcity of high-quality training data [7, 8]. which manifests in multiple aspects: (1) collecting comprehensive training data often require expen- sive physical experiments or high-resolution numerical simulations; (2) computational and storage constraints may restrict the resolution of available data; (3) the collected data frequently exhibits distribution bias due to physical or experimental limitations. These data limitations fundamentally constrain the generalization capability of current neural operators. To address these challenges, we propose DeltaPhi, a novel framework that helps neural operators learn from scarce data.
The core innovation of DeltaPhi lies in reformulating the PDE solving task from learning direct input-output mappings to learning the residuals between similar physical states. This reformulation enables implicit data augmentation by leveraging key property of physical systems: states with similar initial conditions tend follow similar solutions. By learning residuals between pairs of
"Coresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

similar trajectories. DeltaPhi effectively expands the diversity of training samples without requiring additional data collection. This is fundamentally different from existing approaches that attempt to learn the straight mapping for each sample independently.
The proposed framework presents two advantages: (1) Enhanced Training Sample Diversity: The framework enables enhanced diversity in training labels through random auxiliary solution sampling. effectively mitigating distribution bias issues. This training distribution augmentation preserves physical validity while expanding the learned solution space. (2) Architecture Flexibility: DeltaPhi architecture- -agoossic and can seamlessly enhance existing neural operators [2. 9, 10] through unified residual learning mechanism, making it applicable across different PDE solving scenarios.
We conduct extensive experiments validate the effectiveness of physical states residual learning across various settings: (1) diverse physical systems including both regular domains (Darcy Flow, Navier-Stokes) and irregular domains (Heat Transfer, Blood Flow); (2) different neural architectures including spectral-basee and attention-based operators; (3) varied numbers of training samples to test robustness under data scarcity; and (4) cross-resolution scenarios to evaluate generalization capability, The results consistently demonstrate that DeltaPhi significantly improves the performance of base models, particularly in data-limited settings.
Overall, this work introduces a promising framework for enhancing neural operators in practical PDE solving applications where high-quality training data is scarce or expensive to obtain. The physical states residual learning framework could potentially benefit other high- nigh-dimensional regression prob- lems in scientific machine learning, especially when dealing with physically governed systems that exhibit stability properties.
2 Background and Related Work
2.1 Direct Neural Operator Learning
Neural operator learning aims to learn mappings between infiniteddimensional function spaces for solving parametric PDEs [1]. The operator mapping G A A->l ll maps between Banach spaces A and 4. For steady-state problems like Darcy Flow, the input function a(x) E AA typically represents coefficients while u(x:) U is the solution. For time-series problems like Navier-Stokes, a(x) commonly represents prior states, and u(r) represents future states [2].
Neural operators are constructed using various network architectures, with two prominent families being kernel-basee operators and DeepONet-style operators. KernelIntegral-fon architectures, such Graph Neural Operators (GNO) [1], often based on iterative kernel integration. Deep Operator Networks (DeepONet) [3] instead use separate branch and trunk networks to approximate the operator. These two general forms can be represented
Kernellntegral-formm: Go Q o QWo(Wi Ki) o...ooWW., K2)
DeepOnet-form: Go(a)()) Σ BA(a)TT(())
k=1
In the Kernellntegral-fon form, P and Q are projection layers, K., are learnable kernel integral operators, W; are local linear operators, and a is a nonlinear activation. In the DeepOnet-form, (a)a) represents the branch network outputs encoding the input function a, and Tp (y) represents the trunk network outputs encoding the output coordinates y. It is noted that the framework proposed in this work is applicable to both forms of architectures.
The components of these architectures are differently instantiated in previous works. Fourier Neural Operator (FNO) [2] utilizes the Fourier Transform based integral operation to efficiently learn the physical dynamics using limited training data. Factorized Fourier Neural Operator (FFNO) [10] factorize the Fourier Transform along each dimension, reducing the number of network weights. Clif- ford Fourier Neural Operator (CFNO) [11] employs the Clifford Algebra in the network architecture, incorporating geometry prior between multi-physicn fields. The various neural operators could be simply integrated with the proposed Physical States Residual Learning.
Other works investigate architecture improvements for different needs, including chaotic systems modeling [12], ahysical-informer instance- wise finetuning [13], accelerating computation [14, 15, 16, 17, 18), spherical fields processing [19], irregular fields learning [20, 21,, non-periodic boundary fields

---

modeling [22], large-scale pretraining [23, 24], etc. In addition, some works explore other network backbones, e.g. improved frequency-spatial domain transformation [25, 26, 27, 28, 29, 30, 31, 32], convolutions [33], graph neural network [34, 6, 35], attention mechanism [36, 21, 37, 38), and diffusion models [39, 40, 41, 42].
Despite theoretical approximation capabilities, practical limitations persist in generalization across resolutions [13]. handling data scareity [43], and managing biased training distributions [7]. Our work addresses these challenges through residual learning between physical trajectories.
2.2 Residual Learning in Previous Works
Some works [44, 45, 46, 47] learn residual between different data samples. Similar to ResMem [44], also utilize the training set as auxiliary memory during inference, enjoying its generalization enhancement property. The difference lies that (I) we memorize the auxiliary labels and learn the residuals, while ResMem memorizes the residuals and learns the labels, (2) our framework is end-to-end trained while ResMem is trained in separate stages, Similar external memory augmented strategies are popular in the Language Modeling community [48, 49].
Other works learn residuals between low resolution and high resolution representation for image super-resolution [50] and deep learning based CFD simulation [50]. Similar residual learning is investigated in time series prediction task [51, 45]. The primary distinction between these works to ours is that do not utilize the corresponding low-resolution solutions previous time-step solutions, but instead, learn the residuals between different trajectories.
Additionally, [52] studies learning residuals between PDE solutions with similar geometric domains. In contrast, our work focuses on a fundamentally different problem of learning residuals between similar physical states, enabling broader applicability. We establish this based on physical system stability and develop implicit data augmentation through similarity- based sampling during training, effectively addressing distribution bias and overfitting issues in data-limite limited scenarios.
3 Methodology
3.1 Preliminary: Stability of Physical Systems
Stability of PDEs. For a well-posed PDE system with solution operator G, stability means:
[[ ) ((a))llz S Cllai a2llA:
where 01, 02 are input functions and C' is positive constant. This property, also known as Lipschitz continuity of the solution operator, is one of Hadamard's criteria for pose-ddsssssss of PDEs. This stability property is well-established in various PDEs: elliptic PDEs satisfy stability under appropriate boundary conditions; parabolic PDEs exhibit stability through their dissipative mechanisms; and even certain chaotic systems maintain stability for suitable time intervals.
It's noted that this stability condition is not additional assumption imposed by our method, but rather a fundamental prerequisite for any data-driven operator learning approach to succeed [3, 2]. If small changes in the input function could lead to arbitrarily large, discontinuous changes in the solution, learning a generalizable mapping would be extremely challenging for neural networks.
Foundation of Residual Learning. Stability of PDEs establishes a relationship where the magnitude range of difference between output functions systematically changes with respect the difference between input functions. This relationship allows us to explicitly control the diversity of output function residuals by selecting input function pairs with varying similarity ranges. This establishes the foundation of residual neural operator learning and enables the effectiveness of implicit data augmentation, introduced in Section 3.5.
3.2 Residual Operator Mapping: A Unified Formulation
Residual Operator Mapping. Based on the stability of physical systems, we formulate the residual operator mapping that captures the differences between pairs of physical trajectories:
g A² - Au,
A² {(a,,, a,) as E A, E A],
(3)
Al: {u Uj | E EU EU},
3

---

Input Function
Inpat Function
Autillary Sarogle Hetriever
Auailiary Sample
Resolution Allenment
concat
Ausiliary Sample
RandSample
Step2: Randoally sample
trajectory.
Bay
Functiio
Sinilar Samples
Uk
Ncural
7100 Degolee GiST
arglopp
Stepl: top-
Curipar Function
Similarity Scones
Siuula, m.))
teajecteriss.
muinine
inforencc stager
Randon retiooal trabdee cublls
"Japtich Duxe Asquonaatio" Oau dirvet seagles (e. uj beronus dificee
Ourpat
?2
Ourput
reditaa eomeles (oo f Malh
Eaput Punction
Training Set
FLu
(hentor SM
(4) Dirrct Neural Operator
(b) Residual Neural Operator
(4) Auxiliaryy Sample Retriener-3Fel
Figure I: The overall architecture of Physical States Residual Learning. Given an imput functiin ai first sample similar auxiliary sample (ax, us,) from the training sample T. Subsequently, a, and (ak,, Wx,) are concatenated and fed into the the neural operator G producing the predicted states residual. Finally, the predicted solution û, is obtained by adding the predicted states residual with the auxiliary solution ukk.
where G represents the residual mapping operator. A² is the Cartesian product of A with itself, meaning it consists of all possible pairs of functions from A. Au represents the space of function residuals, which are the differences between pairs of functions from U. a, and u. are functions the space of A and U, respectively.
Solving PDEs by Residual Operator g In PDE solving, given imput function a, we first retrieve an auxiliary sample (ak; uk,) from the set of physical samples with known solution e.g. training set. Then the residual neural operator G predicts the solution residual between uy and uk,- The final solution un could be obtained by adding this predicted residual to uk;
Ur G (as, (na)) + Uk.
(4)
We introduce how to obtain auxiliary samples in Section 3.3, and the specific implementation of residual neural operators Section 3.4.
3.3 Auxiliary Sample Retriever Feret
Given an input function a; and the training set T = {ai, W we could retrieve an auxiliary sample (ak, u from T, formulated as follows:
k; = = T), k E (1,2, NJ, k +i,
(5)
where Fee is the sampling function. Below we present our implementation of Fret in detail.
Training stage. During training, given input function a; we randomly retrieve sample aj from training set with top-k similarity scores to a;:
F T) RandSample(argtoppK, Sim(a, az)),
0j
(6)
Sim(a;, az) ||: |
where RandSample(-) represents randomly sampling one element from the given set. K is the sampling range, hyperparameter that could adjusted for different PDEs. For a fair comparison, except for specific statements, we set K 20 for all experiments in this work.
Inference stage. During inference, given the input function test we select the most similar auxiliary sample from the training set:
Fist (a teat SeeasttTT .T) = argmax, Sim(a test a.))
While we take the most similar sample for inference, Appendix A.4 shows that the model robust to different auxiliary sample choices. When randomly selecting from the top 10 similar samples, repeated testing shows minimal variation a standard deviation of just 5.41e-5 across five runs. This indicates that neural operators effectively learn to handle varied auxiliary samples for the same input.

---

Algorithm Residual Operator Learning (DeltaPhi)
Input:
Training set T= = {(a,, / Neural operator G Number retrieval neighbors K Output: Trained residual operator G
while not converged do
Sample training pair (a,, u4) from T
Calculate similarities 8;j sim 0z), E ET,Ji
Find indices N of top-K similar samples
Randomly select k; from NG;
Predict residual Au; = G (la:) ax.])
i; = Au; Uk;
Update 0 by minimizing || us
end while
Cooss-Resolution Sample Retrieval. For cross-resolution scenarios where auxiliary samples have different resolutions from the input function, we employ Fourier Transform- based alignment. Specifi- cally, during inference with high-resolttion input function a downsample a to match training resolution via truncating high-frequency components for similarity calculation:
Frel aligned (a" T) F (DownSampling(a") a,));
(8)
where DownSampling is implemented through frequency-domain truncation:
DownSumpling(a*)- = F (TruncateFFrouss(aa")
(9)
Here Froutier and Tourier denote the Fourier transform and its inverse, respectively. The Truncate operation removes high-frequency components in Fourier space match the target resolution. This Fourier-based downsampling preserves the low-frequency structures of the physical field while ensuring consistent resolution for similarity computation. This resolution alignment enables effective utilization of auxiliary samples across different resolutions.
Retrieval Cost Analysis. The computational overhead of our retrieval process is minimal compared to neural network inference. The total complexity consists of similarity score calculation (O(N Ntrain)) and ranking (O(Nrrain log Ntrain)) )), where Nfeeed is the number of field points and Ntrain is the training set size. Our comprehensive experiments on Darcy Flow demonstrate the high efficiency: UPU--ased retrieval only takes 0.2-0.3ms across different training set sizes (100-900), and the entire residual learning pipeline, including both retrieval and data preparation, adds merely 0.5ms to the inference time. This negligible overhead makes our method highly practical for real-world applications. Detailed complexity analysis and timing experiments are provided in Appendix C.2.
3.4 Architecture of Residual Neural Operators G
Residual Neural Operator Backbone. Benefiting from the universal approximation capability of neural operator networks [2, 10] for direct operator learning, they could be quickly employed as residual neural operators G for learning residual operator mapping Go:
g Q (W, + K.)
a ( + K.)
(10)
where P and Q are projection layers, K., and W; are learnable operators, and o a nonlinear activation. Next, we introduce how to modify existing neural operators for learning residual mappings.
Concatenation of Auxiliary Samples. As aforementioned, the residual operator G requires additional auxiliary function ag; as input. In specific implementation, ag and dgs could be straightly concatenated along their channel dimension:
ainput ag t akgy
(11)
where b represents the concatenation operation of two vectors. Then the concatenated vector dinput feed into the neural operator network:
Up P (imput);
(12)
where the fully connected encoder in the operator network (Equation (10)), and Up represents the first latent function in the hidden space.
5

---

Concatenation of Additional Information. The residual operator mapping can be challenging
learn due to the variability of output residuals for different auxiliary samples. To mitigate such difficulty, we introduce additional information related to dl; as input:
Taking the corresponding wk; (ar) of as, (I) the input.
Utilizing calculated relation metrics, egg. similarity between a; and uk;, as inputs.
Our experimental results (Appendix A.3) indicate that such customized inputs could improve the performance of residual neural operators.
Physical Residual Connection. Based on Equation (4), to obtain the final predicted solution á, introduce physical residual connection at the final layer of the base operator network. This connection adds the auxiliary solution UA, to the predicted residual:
d = ((v)) + uk,
(13)
where Q is the last fully connected projector of the operator network, ey the last latent function in hidden space produced by a o + Kr) shown in Equation (10). This enables end-to-end training with existing frameworks while maintaining consistent setups (loss functions, trainable parameters, etc. with the base operator networks.
Resolueoon Sample Integration. For crossreesouution scenarios where we need to predict high- resolution outputs u" using low-resolution auxiliary trajectories (ak,, Ur.), we employ Fourier-based upsampling align the resolutions before integration:
Oinput (pSampling(at.) @ aH
(14)
Similarly, for the residual connection, we upsample the low-resolution auxiliary solution:
H = Q(er) + UpSampling(Mk:):
(15)
where Q is the decoder and ey is the final hidden state.
The upsampling is performed through zero- prro-padding in Fourier space to preserve the low cowffrequency structures while extending to higher resolution, formulated as follows:
UpSampling(a) FFoorier (FeroPad(Froonue(
(16)
where Flourier and Fourler denote the Fourier transform and its inverse, and ZeroPad extends the spectral representation with zeros to match the target resolution. This suuier-based approach ensures smooth upsampling while preserving the physical characteristics of the auxiliary trajectories.
3.5 How DeltaPhi Improves Generalization Capability
In data-limited scenarios, DeltaPhi enhances generalization by mitigating overfitting, which is achieved through a form implicit data augmentation enabled by our residual learning framework.
Mitigating Overfitting via Implicit Data Augmentation. The core benefit of residual learning is its effectiveness in mitigating overfitting, particularly in data-limited settings. This achieved through two complementary mechanisms.
First, DeltaPhi functions an implicit data augmentation strategy. For training set with N samples, direct learning provides only N input-output pairs. In contrast, DeltaPhi transforms each sample (at, U,) into K distinct residual pairs ((a;, ax), u, uy ) k= by pairing it with K different auxiliary samples selected via Section 3.3. The effectiveness of this augmentation stems from the stability property of PDEs (Section 3. which establishes a structured relationship between input differences and solution residuals. This property allows us to meaningfully control the diversity of the augmented training residuals by sampling auxiliary inputs with varying similarity levels. Therefore, with an opportune retrieval range, we can expand the sparse training distribution thus mitigating distribution while ensuring the residual learning task remains feasible and does not involve highly dissimilar, difficult-to-leary pairs. This process significantly increases the density and diversity of the training distribution, directly addressing the problem of sparse training data.
We name this implicit data augmentation because there are more explicit input-output physical function pairs, but directly training more diverse residual pairs.

---

(Previous)
(Deitalah)
NORM (Previous)
NORM (Previous]
NORM (Deltalhi)
NORM Ereur (Deluahi))
11,00
Error
Enver
Relative Guin: 7.14% (a) Imegular Darcy
Relative Gain: 40.99%1
Relative Guin: 00.00%
(b) Pipe Turbulance
(c) Heat Transfer
Figure 2: Prediction error visualization on irregular domain problems.
Second, the residual learning formulation itself discourages memorization. Because the predicted residual Gg (a,, (g.) must adapt to different auxiliary samples dg. for the same input ag the model is discouraged from simply memorizing a fixed input-output mapping. Thus, the model is less prone to memorizing labels for specific input functions in data-limited scenarios.
From a geometric perspective, this framework effectively shifts the learning paradigm from approx- imating isolated points (absolute solutions) on the solution manifold to learning tangent vectors (solution residuals) that describe the manifold's local geometry. Learning these relational vectors provides a richer learning signal. The stability property discussed in Section 3. ensures this task is tractable, as it establishes a structured relationship between input differences and solution residuals, allowing us to expand the training distribution while maintaining learnability.
Tractability of Residual Mapping Learning. It's noted that learning the residual mapping g can be more difficult than learning the direct mapping G. However, this increased difficulty does not arise from the introduction of spurious high-frequency noise; if two solution functions are smooth, their residual will also be smooth. Instead, the challenge stems from the inherent complexity of the residual mapping itself, which requires the model to synthesize information from two separate input functions (a; and as.)). Despite this increased complexity, our experiments demonstrate that existing neural operators are fully capable of learning this mapping effectively, even learning to respect its inherent symmetry without explicit constraints (see Appendix A.8)
Influence of Retrieval Range. While the retrieval range controls the diversity residuals for training. our results in Appendix A.1 show DeltaPhi is robust the selection of this hyperparameter. For experimental consistency, we set K = 20 throughout our comparisons. We further analysis its impact across different test splits in Appendix A.2 and discuss its selection in Appendix C.1. In addition, we introduce an alternative sampling strategy without K selection in Appendix A.6.
4 Experiment
This section provides the comparison various PDEs, comparison resolution generalization problems, and statistical analysis of residual operator learning. We present more experimental results and analysis in Appendix A.
4.1 Comparison on Various PDEs
We evaluate residual operator learning across irregular and regular domain PDEs, testing performance with different training data sizes. For fair comparisons, all comparisons use exactly identical hyperparameters, model weights, optimizer settings, and training steps.
Irregular domain problems. Following [9]. we evaluate five problems: 2D (Irregular Darcy Flow, Pipe Turbulence) and 3D (Heat Transfer, Composite, Blood Flow) scenarios, all defined on irregular domains represented by discrete triangle meshes. We compare against GraphS [53], DeepOnet [3]. POD-DeepOnet [54], FNO [2] and NORM [9] baselines, implementing residual learning version NORM for direct comparison. Appendix B provides more details.
Results in Table show that NORM (DeltaPhi) outperforms all baselines across irregular domain problems. Performance gains are most notable for Pipe Turbulence and Heat Transfer, showing 40-50% improvement over NORM, demonstrating the residual operator's effectiveness.

---

Table 1: Relative error comparison of operator learning on irregular domains.
Irregular Darcy (Train Size=1000)
Pipe Turbulence
Heat Transfer
Composite
Blood Flow
Model
(Train Size=300)
(Train Size=100)
(Train Size=400)
(Train Size=100)
GraphSAGE
6.73e-2
2.36e-1
2.09e-1
DeepOnet
1.36c-2
9.36c-2
7.20c-4
1.88c-2
8.93e-1
POD-DeepOnet
1.30e-2
2.59e-2
5.70e-4
1.44e-2
3.74e-1
FNO
3.83e-2
3.80e-2
NORM (Direct)
1.05e-2
1.0le-2
2.70c-4
9,99c-3
4.82c-2
NORM (DeltaPhi)
9.75e-3
5.96e-3
1.35e-4
9.18e-3
4.29e-2
Relative Gain
7.14% 1
40.99%
50.00%
8.11%
11.00%
In addition, we visualize the prediction error in Figure 2. The error value is calculated as the absolute difference between the predicted function å and ground-truth function u uGT i.e. lu-u Compared to previous direct learning methods, DeltaPhi significantly reduces the error scale.
Regular domain problems. We conduct the experiment two regular domain problems i.e. Darey Flow and Navier-Stokes, following [2]. The training data amount is 100. We validate DeltaPhi on different neural operators. including FNO [2], FFNO [10]. CFNO [11], GNOT [21]. Galerkin [36], MiOnet [55] and Resnet [56]. The metric relative 1.2 error. Appendix presents more details.
Table shows DeltaPhi improves performance across all base models for both Darcy Flow and Navier-Stokes equations, confirming its effectiveness on regular domains. The improvement on the Navier-Stokes, while modest compared to Darcy Flow, is particularly noteworthy given the inherently chaotic nature fluid dynamics at viscosity v = le 5. Even within this challenging scenario characterized by weakened correlation between similar initial conditions a(r) and their evolved states (x)), DeltaPhi still achieves consistent gains. This robustness to complex dynamics strengthens conclusions regarding the method's 's general applicability. Performance on temporal systems could be further enhanced through designs such as performing auxiliary sample retrieval at each step of time-series prediction mitigate the decay in correlation between input and output functions.
Table 2: Relative error comparison regular Darey Flow and Aavier-Stoker Equation.
Dorey Flow Residual Learning (DeltuPhi)
Navier- Equation Residual Learning
Model
Direct Learning (Direct)
Direct Learning
Relative Gain
Relative Gain
(Direct)
(DeltaPhi)
FNO
3.70e-2
3.31e-2
10.54%
2.24e-1
2.13e-1
4.86%
FFNO
5.22e-2
2.93e-2
43.76%
2,40e-1
2.20ee1
8,54% f
CFNO
4.79e-2
3.15e-2
34.23%
3.51e-1
3,35c-1
4.56% f
GNOT
6.74e-2
5.50e-2
18.39%
4.16e-1
4.06c-1
2.33%
Galerkin
6.78e-2
6.54e-2
3.60%
3.60e-1
3,54c-1
1.86%
MiOnet
8.61e-2
8.22e-2
4.53%
4.79e-1
4.61e-1
3.80%
Resnet
1.25e-1
1.07e-1
14.14%
4.39c-1
4.30c-1
1.87%
Different training scales. We train residual operators using FNO, FFNO, and Resnet Darcy Flow with varying training set sizes (100, 300, 500, 700, and 900) and test their performance on the additional 100 samples.
Table Comparison on different training scales.
Model
Differour Train Size
100
700
FNO (Direct)
3.70e-2
L.40-22
1.02e-2
8.37c-3
7.39c-3
PNO (DeluPhi)
3.31e-2
1.34e-2
9.64e-3
8.06e-
7.18c33
Relative Gain
10.54%
4.08%
5.99% 1
3.74% 1
2.83%
The experimental results shown in Ta- ble 3. On different training set sizes,
FTNO (Direct)
5.22e-2
L69e-2
9.73e-3
7.48e-3
6.16e-3
FFNO (DeltaPhi)
L11e-2
7.20c-3
6.22o-3
5,30--
Relative Gain
43.76%
34.47%
25.98% 1
16.79%
14.02%
DeltaPhi improves the prediction perfor- mances of neural operators. As the number of available training data decreases, the relative im- provement percent (the row "Relative Gain") consistently increases on FNO and FFNO. In addition, several residual neural operators even outperform direct neural operators using less training data amount, e.g. FFNO (DeltaPhi) with training size 500 outperforms FFNO (Direct) with training size 700, Resnet (DeltaPhi) with training size 700 outperforms Resnet (Direct) with training size 900. The results empirically demonstrate the effectiveness of DeltaPhi across different training scales.
4.2 Comparison on Resolution Generalization Problem
This section presents the experimental results of training-free resolution generalization.
Resolution generalization problem. Fourier neural operator [2] enables zero-shot resolution generalization inference. However, when the training data grid is excessively coarse, the inference

---

performance over high resolution data drops lot. The performance degeneration is more serious when learning operator mapping between fields with weaker spatial continuity such as Darey Flow.
Setup. We conduct the training-free resolution generalization experiment on Darcy Flow [2]. Specif- ically, we train the neural operators with low resolutions 85 x 85, 43 x 43, 31 31, and 22 x 22 respectively. then evaluate them using high resolution 421 x 421 test data. Both FNO [2] and FFNO [10] are compared the base models. The training set scale includes 100 and 900 trajectories.
Table 4: Relative error comparison on zero-shot resolution generalization.
Train Size= 100
Train Size=900
Model
85 x 85
43 43
x 31
22 22
85
43 x 43
x 31
22 22
FNO (Direct)
7.29c-2
1.19c-1
1.49e-1
1.70c-1
6.73e-2
1.1ee-
1.34e-1
1.76c-1
FNO (DeltaPhi)
6.91e-2
1.13e-1
1.33e-1
1.53e-1
4.91e-2 27.04% t
9.37e-2
1.15e-1
1.57e-1
Relative Gain
5.13%
5.06%
10.43%
10.08%
15.65%
14.29%
10.31% f
FENO (Direct)
6.64e-2
1.04e-1
1.24e-1
1.43e-1
4.89e-2
1.01e-1
1.27e-1
1.49e-1
FENO (DeltaPhi)
6.34e-2
9.90e-2
1.18e-1
1.36e-1
4.42e-2
9,00e-2
1.12e-1
1.35e-1
Relative Gain
4.53%
5.06%
4.84%
5.09%
9.48% f
10.72%
11.94%
9.29%
Performance comparison. Table presents
the resolution generalization results. Com- pared to FNO [2), FFNO [10] shows prefer- able generalization ability for its less num- ber of network weights. In all experimen- tal settings, the proposed physical residual models (tagged with "DeltaPhi") evidently outperform their counterpart base models. The performance improvement of residual operator learning is more noticeable when the training set size is 900. In this setting, FNO (DeltaPhi) managed to match, or even surpass FFNO [10] on resolution 85 x 85, 43 43, and 22 x 22, despite using the rel- atively weaker base network. The results empirically draw the conclusion that physical residual learning improves zero-shot resolution gener- alization performance.
High Resolution, FNO
Text Relative Ervor (X 100)
1
High Resolution, DP-FNO
Low Resolution, FNO
Resolution, DP-PNO
Mitigating Serious Ovediting
Training Epochts
Figure 3: Training curve comparison. "DP_*" HM denotes the proposed residual learning version of base models.
Training curve comparison. We also report the test loss curve during training in Figure 3. The training amount is 900 and the training resolution is 85 x 85. As the training proceeds, the relative error resolution 85 85 data continues decline. However, after some epochs, the loss on resolution 421 x 421 gradually increases due to overfitting. The proposed residual operator (named with DP-FNO) significantly mitigates this overfitting tendency of the base model (named with FNO).
4.3 Statistical Analysis of Residual Operator Learning
This section validates DeltaPhi's foundations through statistical analysis.
Similarity between output functions. We
validate the stability property in Section 3.1, which states that similar inputs lead to sim- ilar solutions. This property essential for
uix)) Distacee
0.40
Retrieval
Retileaal Set
Resioval Sizz: 700
Retioval Size: 900
Retrieval Set
0.30
0.25
our residual learning approach. In Figure 4, we show how the normalized distance be- tween output functions u fest (z)) and uk. (a) changes as the similarity rank of their input functions increases for Darcy Flow. (Ap- pendix D.1 provides visualization details.) The normalized distance grows consistently with similarity rank across all retrieval set sizes. This confirms the stability property when input functions become less similar, their solutions also become less similar. The relationship also shows that increasing the retrieval range K during training adds more diversity to the training samples by
0.20
0.15
10
20
30
40
70
80
a(x) Similarity Rank
Figure 4: The curve of distance between test (x) and us, (I) as retrieval similarity rank increases.

---

including wider range of residual patterns, thereby expanding the learned solution space while maintaining physical validity.
Training label distribution comparison. We demonstrate how DeltaPhi provides im- plicit data augmentation on Darcy Flow by visualizing label distributions. We project both training and testing labels (output func- tions for direct learning, output residuals for DeltaPhi) into 2D using PCA. Appendix D.2 details this process. Figure reveals the main benefits of our residual learning frame- work: In direct operator learning (left), lim- ited training data restricts coverage of the solution space, making generalization diffi- cult. This shows the fundamental challenge of distribution bias in data-limited scenarios. With DeltaPhi (right), we see that: (a) the testing label distribution becomes more con- centrated, showing that learning residuals is easier than learning absolute solutions, and (b) the training labels show greater spread, demonstrating the implicit data augmenta- tion claimed in Section 3.5. By creating diverse valid residual pairs through controlled auxiliary sample retrieval, DeltaPhi effectively enhances the training distribution while preserving physical validity.
Standaed Devistion
Totning Labek
BevsteeBBeatte
Initing Labil
Rang:
Lubds
Kang:
The divendly of tnaning buddi echazoe
Direct Opesstor Leaning
Residual Operator Leatning
Figure 5: Label distribution visualization. The points represent dimension-reduced labels (u. (z:) for direct learning, ur (z) - uk_ (x) for residual learning) through Principal Component Analysis. The ellipses with dot- ted lines and solid lines represent standard deviation and range. respectively. Green and red color corre- spond to training and testing set, respectively.
Limitation
Despite the comprehensive validation of DeltaPhi's superiority through an extensive range of experi- ments, we recognize some inherent limitations that, nevertheless, do not impact the solidity of our conclusions. First, although highly challenging, it is significant to study more realistic applications where training data acquisition is extremely costly, such as high-fidelity fluid modeling or biomedical simulations. Second, we acknowledge the potential constraints of using a general cosine similarity metric. While effective, this metric may not be optimal for all physical systems. This is particularly relevant for highly chaotic systems, such as the low-viscosity Navier-Stokes equations (Table 2), where DeltaPhi, while effective, showed a more modest relative improvement compared to other problems. This suggests that the complexities associated with such systems, where simple initial state similarity may not sufficiently capture the relationship between solution trajectories, warrant further investigation. Future work could explore these aspects, for instance, by developing more advanced auxiliary sample retrieval strategies better suited for complex dynamics, which may unlock greater performance gains in these challenging applications.
Conclusion
This work proposes to learn physical state residuals for PDE solving by reformulating the task from direct mapping to learning residuals between similar physical states, supported by physical system stability. This approach enables implicit data augmentation without requiring additional data collection, effectively addressing challenges in data-limited scenarios. We validate the effectiveness across various physical systems, neural architectures, and both regular and irregular domains. We hope this framework provides insights for future development of machine learning based PDE solving.
Acknowledgments
This work is partially supported by National Science and Technology Major Project (2022ZD01 17802). This work was supported in part by General Program of National Natural Science Foundation of China (62372403) and "Pioneer" and "Leading Goose" R&D Program of Zhejiang (No. 2025C02032). This work is also supported by the Fundamental Research Funds for the Central Universities (26-2025-00080) and the Earth System Big Data Platform of the School of Earth Sciences, Zhejiang University.
10

---

References
[1] Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces, arXiv preprint arXiv:2108.08488/ 2021.
[2] Zongyi LI, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differen- tial equations. arXiv preprint arXiv:2010.08899, 2020.
[3] Lu Lu, Pengzhan Jin, and George Em Karniadakis, Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators. arXiv preprint 1XXi::9100001933 2019.
[4] Maziar Raissi, Paris Perdikaris, and George E Karniadakis. brysscc--iffrmed neural networks: deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 2019.
[5] Zichao Long, Yiping Lu, Xianzhong Ma, and Bin Dong. Pde-net: Learning pdes from data. In International conference on machine learning, pages 3208-3216. PMLR, 2018.
[6] Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Graph kernel network for partial differential equations. arXiv preprint arXiv:2003. 033455 2020.
[7] Florent Bonnet, Jocelyn Mazari, Paola Cinnella, and Patrick Gallinari. Airfrans: High delity computational fluid dynamics dataset for approximating reynolds-averaged navier-stokes solutions. Advances in Neural Information Processing Systems, 35:23463 -34488, 2022.
[8] Jose Antonio Lara Benitez, Takashi Furuya, Florian Faucher, Anastasis Kratsios, Xavier Tric- oche, and Maarten de Hoop. Out-of-distributionaal risk bounds for neural operators with applications to the helmholtz equation. Journal of Computational Physics, page 113168, 2024.
[9] Gengxiang Chen, Xu Liu, Qinglu Meng, Lu Chen, Changqing Liu, and Yingguang Li. Learning neural operators on riemannian manifolds. arXiv preprint arXivv2302.088 066 2023.
[10] Alasdair Tran, Alexander Mathews, Lexing Xie, and Cheng Soon Ong. Factorized fourier neural operators. arXiv preprint 1rXxv:2111.3386 2021.
[11] Johannes Brandstetter, Rianne den Berg, Max Welling, and Jayesh K Gupta. Clifford neural layers for pde modeling. arXiv preprint arXiv:2209. 049344, 2022.
[12] Zongyi Li, Miguel I Liu-Schiaffini, Nikola Kovachki, Burigede Liu, Kamyar Avizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Learning dissipative dynamics in chaotic systems. arXiv preprint arXiv:2106.0687
2021.
[13) Zongyi Li, Hongkai Zheng, Nikola Kovachki, David Jin, Haoxuan Chen, Burigede Liu, Kamyar Arizzadenesheli, and Anima Anandkumar. Physics-informel neural operator for learning partial differential equations. arXiv preprint arXiv:21 1.03944, 2021.
[14] Michael Poli, Stefano Massaroli, Federico Berto, Jinkyoo Park. Tri Dao, Christopher Ré, and Stefano Ermon. Transform once: Elficient operator learning in frequency domain. Advances in Neural Information Processing Systems, 35:7947-7959, 2022.
[15] Colin White, Renbo Tu, Jean Kossaifi, Gennady Pekhimenko, Kamyar Arizzadenesheli, and Anima Anandkumar. Speeding up fourier neural operators via mixed precision. arXiv preprint arXiv:2307.15034 2023.
[16] Md Ashiqur Rahman, Zachary E Ross, and Kamyar Azizzadenesheli, U-no: U-shaped neural operators. arXiv preprint arXiv:2204 ///7 2022.
[17] Pengpeng Xiao, Muqing Zheng, Anran Jiao, Xiu Yang, and Lu. Quantum deeponet: Neural operators accelerated by quantum computing. Quantum, 9:1761, 2025.

---

[18] Wenhan Gao and Chunmei Wang. Active learning based sampling for high-dimensions nonlinear partial differential equations. J. Comput. Phys., 475:111848, February 2023.
[19] Boris Bonev, Thorsten Kurth, Christian Hundt, Jaideep Pathak, Maximilian Baust, Karthik Kashinath, and Anima Anandkumar. Spherical fourier neural operators: Learning stable dynamics the sphere. arXiv preprint arXiv:2306.03888 2023.
[20] Zongyi Li, Daniel Zhengyu Huang, Burigede Liu, and Anima Anandkumar. Fourier neural oper- ator with learned deformations for pdes on general geometries. arXiv preprint arXiv:220.005009 2022.
[21] Zhongkai Hao, Zhengyi Wang, Hang Su, Chengyang Ying, Yinpeng Dong, Songming Liu, Ze Cheng, Jian Song, and Jun Zhu. Gnot: A general neural operator transformer for operator learning. In International Conference on Machine Learning, pages 12556-12569. PMLR, 2023,
[22] Ziyuan Liu, Yuhang Wu, Daniel Zhengyu Huang. Hong Zhang, Xu Qian, and Songhe Song. Spfno: Spectral operator learning for pdes with dirichlet and neumann boundary conditions. arXiv preprint arXiv:23/2.06688) 2023.
[23] Thomas J Grady, Rishi Khan, Mathias Louboutin, Ziyi Yin, Philipp A Witte, Ranveer Chandra, Russell Hewett, and Felix J Herrmann. Model-parallel fourier neural operators as learned surrogates for large-scale parametric pdes. Computers & Geosciences, page 105402, 2023.
[24] Shashank Subramanian, Peter Harrington, Kurt Keutzer, Wahid Bhimji, Dmitriy Morozov, Michael Mahoney, and Amir Gholami. Towards foundation models for scientific machine learning:
Characterizing scaling and transfer behavior. arXiv preprint arXiv:2306.00 00258, 2023.
[25] Xihang Yue, Yi Yang, and Linchao Zhu. Holistic physics solver: Learning pdes in a unified spectral-physical space. In Forty-second International Conference on Machine Learning.
[26] Gaurav Gupta, Xiongye Xiao, and Paul Bogdan. Multiwavelet- based operator learning for differential equations. Advances in neural information processing systems, 34:24048-24062, 2021.
[27] Jiawei Zhao, Robert Joseph George, Yifei Zhang, Zongyi Li, and Anima Anandkumar. Incre- mental fourier neural operator. arXiv preprini arXiv:2211.151888 2022.
[28] Qianying Cao, Somdatta Goswami, and George Em Karniadakis. Lno: Laplace neural operator for solving differential equations. arXiv preprint arXiv:2303. 10528, 2023.
[29] Tapas Tripura and Souvik Chakraborty. Wavelet neural operator for solving parametric partial differential equations computational mechanics problems. Computer Methods in Applied Mechanics and Engineering, 115783, 2023.
[30] Tian Wang and Chuang Wang. Latent neural operator for solving forward and inverse pde problems. arXiv preprint 244iv::2006003992. 2024.
[31] Wenhan Gao, Jian Luo, Ruichen Xu, and Yi Liu. Dynamic schwartz-fourier neural operator for enhanced expressive power. Transactions on Machine Learning Research.
[32] Wenhan Gao, Ruichen Xu, Yuefan Deng, and Yi Liu. Discretization-ivvaaaneeee invariance? on the discretization mismatch errors in neural operators. In The Thirteenth International Conference on Learning Representations, 2025.
[33] Bogdan Raonic, Roberto Molinaro, Tim De Ryck, Tobias Rohner, Francesca Bartolucci. Rima Alaifari, Siddhartha Mishra, and Emmanuel de Bezenac. Convolutional neural operators for robust and accurate learning of pdes. In Thirty-sevntth Conference on Neural Information Processing Systems, 2023.
[34] Johannes Brandstetter, Daniel Worrall, and Max Welling. Message passing neural pde solvers. arXiv preprint 23X1v:2202.03376, 2022.
12

---

[35] Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Andrew Stuart, Kaushik Bhattacharya, and Anima Anandkumar. Multipole graph neural operator for parametric partial differential equations. Advances in Newral Information Processing Systems, $3:6755-6766, 2020.
[36] Shuhao Cao.
Choose a transformer: Fourier or galerkin. Advances in neural information processing systems, 2224924-249400 2021.
[37] Michael Prasthofer, Tim De Ryck, and Siddhartha Mishra. Variable-input deep operator networks. arXiv preprint arXiv:2205. 2022.
[38] Haixu Wu, Huakun Luo, Haowen Wang, Jianmin Wang, and Mingsheng Long. Transolver: A fast transformer solver for pdes on general geometries. arXiv preprint arXiv:2402.023666 2024.
[39] Jae Hyun Lim, Nikola B Kovachki, Ricardo Baptista, Christopher Beckham, Kamyar Az- izzadenesheli, Jean Kossaifi, Vikram Voleti, Jiaming Song, Karsten Kreis, Jan Kautz, al. Score-based diffusion models in function space. arXiv preprint arXiv:2302.074000 2023.
[40] Hang Zhou, Yuezhou Ma, Haixu Wu, Haowen Wang, and Mingsheng Long. Unisolver: Pde- conditional transformers are universal pde solvers. arXiv preprint 1r225:2405..7557, 2024.
[41] Siming Shan, Min Zhu, Youzuo Lin, and Lu Lu. Red-diffeq: Regularization by denoising diffusion models for solving inverse pde problems with application to full waveform inversion. arXiv preprint arXiv:2509.21659 2025.
[42] Siming Shan, Pengkai Wang, Song Chen, Jiaxu Liu, Chao Xu, and Shengze Cai. Pird: Physics- informed residual diffusion for flow field reconstruction. arXiv preprint arXiv:2404.08 12 2024.
[43] Ricardo Vinuesa and Steven L Brunton. Emerging trends in machine learning for computational fluid dynamics. Computing in Science & Engineering, 24(5):33-41, 2022.
[44] Zitong Yang. Michal Lukasik, Vaishnavh Nagarajan, Zonglin Li, Ankit Singh Rawat, Manzil Zaheer, Aditya Krishna Menon, and Sanjiv Kumar. Resmem: Learn what you can and memorize the rest. arXiv preprint a:Xiv:2302.0156, 2023.
[45] Chen Xu and Yao Xie. Sequential predictive conformal inference for time series. In International Conference on Machine Learning, pages 38707-38727. PMLR, 2023.
[46] Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. Residual dense network for image super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2472-2481, 2018.
[47] Yoav Freund and Robert E. Schapire. A tesicion- generalization of on-line learning and an application to boosting. In European conference on computational learning theory, pages 23-37. Springer, 1995.
[48] Shuohang Wang, Yichong Xu, Yuwei Fang, Yang Liu, Siqi Sun, Ruochen Xu, Chenguang Zhu, and Michael Zeng. Training data is more valuable than you think: A simple and effective method by retrieving from training data. arXiv preprint arXiv:2203.08773, 2022.
[49] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training In International conference on machine learning, pages 3929-3938. PMLR, 2020.
[50] Loh Sher En Jessica, Naheed Anjum Arafat, Wei Xian Lim, Wai Lee Chan, and Adams Wai Kin Kong. Finite volume features, global geometry representations, and residual training for deep searning-based cfd simulation. arXiv preprint arXiv:2311.14460 2023.
[51] Anastasios N Angelopoulos, Emmanuel J Candes, and Ryan J Tibshirani. Conformal pid control for time series prediction. arXiv preprint arXiv:2307.16895, 2023.
[52] Ze Cheng, Zhongkai Hao, Xiaogiang Wang, Jianing Huang, Youjia Wu, Xudan Liu, Yiru Zhao, Songming Liu, and Hang Su. Reference neural operators: Learning the smooth dependence of solutions of pdes geometric deformations. arXiv preprint 4rXhv:2405.17500 2024.
13

---

[53] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems, 30, 2017.
[54] Lu Lu, Xuhui Meng, Shengze Cai, Zhiping Mao, Somdatta Goswami, Zhongqiang Zhang, and George Em Karniadakis. A comprehensive and fair comparison of two neural operators (with practical extensions) based on fair data. Computer Methods in Applied Mechanics and Engineering. 393:114778, 2022.
[55] Pengzhan Jin, Shuai Meng, and Lu Lu. Mionet: Learning multiple- inpput operators via tensor product. SIAM Journal on Scientific Computing, A34900334 2022.
[56] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[57] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: wtt..4412..690 6980, 2014.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims have been included in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match thcoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have provided limitation section in the end of main text.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those
discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might violated practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't tt acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We have provided the assumption for the theoretical advantage of the proposed method in Section 3.1.
15

---

Guidelines:
If the contribution is either be a way to access this model for reproducing the results or a way to reproduce The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Sufficient experimental details have been provided for every base models and
datasets.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, if the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (c.g., in the case of a large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
Depending on the contribution, reproducibility
While NeurlPS does require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c)
model (e.g., large language model), then there should
the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case closed-source models, may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
16

---

Justification:
We have provided open access to the data and code to reproduce the main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
ooneooooissi for more details. httgggddd//oooeuunne
While we encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: htt////ooouusssiiooooooo
//nips.. wwtppbllc/g/tteeceeeceeeceeeeooe
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized ver- sions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
The experimental setup for every result has been stated.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Statistical analysis for random sample retrieval is provided in Appendix A.4.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main
claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
17

---

It is OK to report 1- 1-sigma enror bars, but one should state it. The authors should preferably report 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The computational resource is stated in Appendix B.3.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate
total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: ///urrisr....// cc
Answer: [Yes]
Justification: This work comprises the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed. If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
18

---

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mecha- nisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This work poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All the employed data and methods are open sourced and the original papers are stated.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (c.g.. CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- e-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged reach out the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
19

---

The answer NA means that the paper does not release new assets. Researchers should communicate the details of the dataset/code//modl as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country which research conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: :atttp::/neuriie h/tpe/rins....m wttpriiipp.ccccfeeeeeeeeeee / /LLM) for what should should not be described.
20

---

A Additional Experiment
A.1 Impact of Retrieval Range and Random Sampling
This section analyzes how different retrieval range values (K) affect the performance of our residual learning framework. Additionally, we investigate the importance of similarity-based retrieval by replacing it with random sampling (denoted as "w/o ".
Table 5: Relative enror comparison of FNO (DeltaPhi) with different retrieval ranges (K) and training
set sizes on Darey Flow. The symbols f and + denote performance increasing and decreasing respectively, compared to direct learning.
Training
ENO
ENO (DeltaPhi) with different K values
w/o Free
Set Size
(Direct)
K=5
K=10
K=20
K=30
K=40
K=50
100
3.57e-2
3.34e-2 f
3.31e-2 f
3.31e-2 f
3.31e-2 f
3.36e-2 f
3.30c-2
3.04e-2
300
1.4le-2
1,38c-2 f
1.41e-2 f
1.35e-2 f
1.36e-2 f 1.40e-2 f
1.37c-21
1.62c-2 4
500
1.02e-2
1.01e-2 f
1.01e-2
9.64e-3 f
9.96c-31 f
1.00e-2 1
1.01e-2
1.29c-2
700
8.37e-3
8.29e-31 f
8.55e-3 L
8.06e-3 f
8.04e-31
8.03e-31 f
8.19e-3
1.10e-2
Impact of retrieval range K. Table presents the relative error of FNO (DeltaPhi) with different retrieval range values across various training set sizes. We observe several important patterns:
In data-limited scenarios (100 to 500 samples), DeltaPhi demonstrates robust performance across different values, obtaining consistent performance gains across diverse selection of K.
With very limited training data (100 samples), larger K values (40-50) tend to perform slightly better, suggesting that expanding the retrieval range becomes beneficial when the training data is extremely sparse.
Conversely, with abundant training data (700 samples), moderate K values (20-40) generally outperform both smaller and larger retrieval ranges, indicating an optimal balance between sample diversity and learning difficulty.
The consistent performance across different K values demonstrates that our framework is not overly sensitive to this hyperparameter. making it practical for real-world applications where exhaustive hyperparameter tuning may not be feasible.
Importance of similarity- retrieval. The last column of Table 5 shows the performance when auxiliary samples randomly selected from the training without considering similarity ("Random Sampling"). Comparing this with similarity- retrieval reveals:
For most training set sizes (700, 500, and 300), random sampling performs significantly worse than similarity-based retrieval, with error increases compared to the best K value configuration.
This performance degradation confirms our discussion in Section 3.1: the stability property of PDEs makes learning residuals between similar states more tractable than between arbitrary states.
These results empirically validate the importance of similarity- auxiliary sample retrieval in our framework, particularly as the training set size increases. The optimal retrieval strategy may depend on the specific characteristics of the PDE system and the available training data, but similarity-basee retrieval provides robust default approach for most scenarios.
A.2 Performance Analysis on Different Test Distribution
To evaluate the robustness of DeltaPhi under varying test distributions, we conduct experiments on test samples with different distances from the training set. Specifically, we first compute the cosine similarity score between each test sample and its closest training sample on Darcy Flow. Based on these scores, we divide the test set into 7 overlvallapping splits, where lower average score indicates more challenging split due to greater deviation from the training distribution.
We evaluate both direct operator learning and residual operator learning using FNO as the base model across these splits. Additionally, test different retrieval range settings (K) analyze their impact on performance. The results are presented in Table 7.
The experimental results reveal several key findings:
21

---

Table 6: Statistics of different test splits based on similarity to training data.
Sample Number
Min Score
Max Score
Average Score
Split1
23
0.85
0.86
0.86
Split2
0.86
0.88
0.87
Split3
30
0.88
0,90
0.89
Split4
37
0.90
0.91
0,90
Split5
33
0.91
0.93
0.92
Split6
43
0.93
0,95
0,94
Split7
16
0.95
0.96
0.95
Table 7: Relative enror comparison on different test splits. Numbers in parentheses show relative improvement over direct learning.
Methods
Split1
Split2
Split3
Split4
Split5
Split6
Split7
Direct Learning
5.08e-2
5.00e-2
4.71e-2
3.31e-2
2.77e-2
2.73e-2
2.43e-2
DeltaPhi (K=20)
4.77c-2
4.68e-2
4.32e-2
3.14c-2
2.58c-2
2.46e-2
2.18c-2
(6.06%1)
(6.34%1)
(8.28%1)
(5.30%))
(6.83%1)
(10.09%t)
(10.46967)
DeltaPhi (K=50)
4.80c-2
4.66e-2
4.32c-2
3.13e-2
2.57e-2
2.47c-2
2.11e-2
().39%
(6.73%t)
(8.35%))
(5.51%1)
(7.24%61
(9.76%
(13.10%67
DeltaPhi (K=5)
4.83e-2
5.01e-2
4.39e-2
3.27e-2
2.52c-2
2,43e-2
2.10c-2
(4.78%/)
(-0.28%%4
(6.76%%)
(1.31%1)
(8.88%1)
(11.15%1)
(13.84%))
DeltaPhi with appropriate retrieval ranges (K=20 or K=50) consistently outperforms direct operator learning across all splits, even on the most challenging ones (Splitl, Split2, Split3) that significantly deviate from the training distribution.
The choice of retrieval range K has notable impact performance. A relatively small K value (K=5) leads to smaller improvements on challenging splits (Split1, Split2, and Split4), and can even underperform direct learning in some cases (Split2).
K values =K= or K=50) in appropriate ranges generally provide stable improvements across different splits, suggesting they enable more robust generalization under distribution shifts.
These results demonstrate that DeltaPhi maintains its effectiveness even when test samples signifi- cantly differ from training data, supporting its robustness in practical scenarios where training and testing distributions may not perfectly align.
A.3 Customized Auxiliary Input Ablation
This section conducts ablation experiments the customized auxiliary input.
Setup. We evaluate the influence of customized auxiliary input on Ravier-Stokes The training trajectory number is 100. FNO [2] is used the base model. We test 3 types input of au, (z), including complete as; (x), empty dx, (x) and partial (only input last trajectory steps) ax, (a). In addition, we experiment with the inclusion omission of us, (æ), well as the addition exclusion of Scores.
Customized Auxiliary Input Influence. The results are shown in Table 8. The first three rows indicate that utilizing partial ax, (x) yields the best results for ag, (x).. When comparing the third row with the fourth row, or the fifth row with the sixth row, it's evident that incorporating Score, is also meaningful. Additionally, when comparing the third row with the fifth row, as well as the fourth row with the sixth row, the inclusion of uk, proves to be extremely crucial. The above findings suggest that the utilization customized input beneficial, thereby confirming the conclusion that proper customized input could improve performance.
A.4 Auxiliary Sample Selection during Inference
As described in Section 3.3, we greedily select take the auxiliary sample with the top similarity score during inference. To validate the robustness of the trained residual neural operator on different
22

---

Table 8: Ablation of customized auxiliary input on Navier-Stokes. Bold font, underline and wavyline respectively denote the best, second best, and inferior results.
Customized Auxiliary Input
Relative Error
0k, (z)
uke uke (I)
Scorei,ks
All
2414
X
2.20e-1 1
Partial
/ 2.13e-1
Partial
2.13e-1
Partial
/
2,34e-1
Partial
2.37e-1
auxiliary samples, we experiment with randomly sampled auxiliary trajectories from the top-10 similarity scores. The experiment is conducted on Darcy Flow with FNO the base model.
Table 9: Results of repeat Evaluation using random auxiliary sample with Top-10 10 similarity score.
Relative Error of FNO (DeltaPhi)
Infer 1
3.3441876e-2
Infer 2
3.3364178e-2
Infer 3
3.3501464e-2
Infer 4
3.3489122e-2
Infer 5
3.3489122e-2
STD
5.40649e-5
Table 9 presents the relative error of every experimental run and the standard deviation (STD) across five repeated experiments. The low standard deviation value 5.41e-5 indicates that the trained residual neural operators are not sensitive to the selection of auxiliary samples. This concludes that the neural operator effectively learns the proposed residual operator mapping.
A.5
Different Similarity Function
We use Cosine Similarity for its two advantages:
Unaffected by the magnitude of the numerical value. Thus it could be conveniently applied to various PDEs without considering the numerical scale distribution of physical fields.
Taking normalized values in [o. 1]. We could simply use it an additional network input. As Table 8 shows, incorporating the similarity scores improves the model's performance on the Navier-Stokes equation.
To explore more retrieval metrics for scientific machine learning, we experiment with other discrep- ancy metrics (all the distance values are calculated after normalization), shown in Table 10.
Table 10: Results of DeltaPhi-FNO using different similarity functions.
Similarity Functions
Relative Error
Cosine Similarity Euclidean Distance
3.31e-2
3.31e-2
Manhattan Distance
3.28e-2
The results indicate that different distance functions yield subtle impacts. Euclidean Distance and Manhattan Distance are also appropriate choices. Additionally, inspired by the great success of the NLP community, conducting retrieval the latent space may yield superior results. We will explore this in future work.
23

---

A.6 Alternative Sampling Strategy: Importance Sampling
In addition to the similarity-based sampling strategy with a fixed range K described in Section 3.3, we also explored the importance sampling approach. Instead of selecting auxiliary samples from fixed-size neighborhood, this method assigns sampling probabilities to all training instances based on their similarity scores with the input function.
Specifically, given an input function a; the probability of selecting any training sample aj as the auxiliary sample is defined
(0))s/ k=1 exp(sim(aa, az)/T)
P(aşla;)
(17)
where sim(...) .) is the cosine similarity function defined in Equation (6), T is temperature parameter controlling the sharpness of the distribution, and N is the total number of training samples.
We evaluated this importance sampling strategy on two representative problems: Darey Flow with FNO and Heat Transfer with NORM. The results presented in Table 11.
Table 11: Performance comparison of different sampling strategies
Method
Heat Transfer
Darcy Flow
Baseline (Direct Learning)
2.70e-4 1.35e-4
3.70e-2 3.31e-2
DeltaPhi (K = 20)
DeltaPhi (Importance Sampling)
2.02e-4
3.22e-2
The experimental results demonstrate that importance sampling can effectively enhance model performance, achieving 25.04% and 12.95% relative improvement on Heat Transfer and Darcy Flow respectively. Notably, on Darcy Flow, importance sampling even outperforms the fixed-range sampling strategy (K 20). This suggests that adaptive probability- based sampling could be promising alternative to the fixed-range approach, particularly for problems where the similarity structure of the solution space is more complex.
A.7 Comparison of Interpolation Methods for Cross-resolution Residual Learning
In our framework, we employ Fourier-based interpolation for cross-resolution alignment and integra- tion (as discussed in Section 3.3 and Section 3.4). This choice is advantageous as zero-padding in the frequency domain, which equivalent to ideal sine interpolation in the spatial domain, effectively upsamples the signal while exactly preserving the original www-freuuency components without dis- tortion. This characteristic aligns with the core principles of FNO-like architectures for zero-shot resolution generalization, which rely accurately capturing the low-frequency dynamics of the physical solution.
Table 12: Comparison of different interpolation methods on Darcy Flow.
Interpolation Method
Relative Error
Fourier Interpolation
6.91e-2 7.36e-2
Bilinear Interpolation
Nearest Interpolation
7.39e-2
To validate this choice, we conducted a comparison with other common interpolation methods, specifically Bilinear and Nearest Interpolation. The experiment was performed on the Darcy Flow problem, training 85 x 85 resolution and testing on 421 x 421 resolution. The results, presented Table 12, show that Fourier-base based interpolation achieves the lowest error, confirming suitability for zero-shot generalization tasks.
24

---

A.8 Symmetry Analysis of Residual Operator
A key property of our residual learning formulation is the inherent symmetry between paired samples. To investigate whether the network naturally captures this property without explicit enforcement, we analyze the symmetry behavior of trained residual neural operators.
Symmetry Metric. We introduce Symmetry Loss metric to quantity how well the learned operator respects the expected symmetry for paired samples (a,, u,) and (ak, uk;))
|| (a;, (x;) + (ax; a:) 1 ||9/ (a,, ox, l12
Symmetry Loss
(18)
This metric measures the normalized discrepancy between residual predictions when the input order is reversed. A perfectly symmetric operator would yield value of zero, while larger values indicate asymmetric behavior.
Experimental Setup. We track this metric throughout the training process for our DeltaPhi-FNO model on the Darcy Flow problem. Importantly, do not explicitly optimize for symmetry in the training objective; we simply observe how the standard training process affects symmetry properties.
Table 13: Evolution of Symmetry Loss during training of DeltaPhi-FNO on Darcy Flow
Metrie
Epoch 0
Epoch 300
Epoch 600
Epoch 900
Epoch 1200
Epoch 1500
Epech 1800
Symmetry Loss
2.01c+0
1.16c-1
8.00e-2
4.52c-2
4.12c-2
3.84c-2
3.22c-2
Results and Analysis. The results in Table 13 demonstrate that the model progressively learns to respect the inherent symmetry the residual mapping without explicit guidance. Starting from a highly asymmetric state (Symmetry Loss of 2.01), the model naturally converges toward increasingly symmetric behavior, reaching a much lower Symmetry Loss of 3.22e-2 by the end of training.
This emergence of symmetry-aware behavior validates that our residual learning approach inherently captures physically meaningful properties of the underlying systems. The network effectively learns that the relationship between two physical states should maintain consistent properties regardless of which state is considered the reference, aligning with fundamental principles of physical systems.
These findings suggest potential future improvements through explicit symmetry enforcement during training or through specialized architectures designed leverage this property, which could further enhance the physical consistency and sample efficiency of neural operator learning.
A.9 Integration with Other Base Models
The formulated residual mapping general operator learning task, fundamentally independent of the specific base architecture. Consequently, any neural operator can be employed to learn this mapping. Table 14, we include two more base models, Transolver [38] and HPM [25], on the Irregular Darcy problem.
Table 14: Performance of DeltaPhi integrated with more base architectures.
Model
Relative Error
Transolver (Direct) [38]
8.54e-3 8.18e-3
Transolver (DeltaPhi)
HPM (Direct) [25]
7.39e-3
HPM (DeltaPhi)
6.67e-3
The results show that DeltaPhi consistently improves the base models. In the future, is significant to integrate DeltaPhi with more advanced architectures.
25

---

B Experimental Detail
B.1
Dataset
B.1.1 Regular Domain
Darey Flow. Darey Flow is steady-state solving problem. We conduct the experiment on Darey Flow using the same dataset as [2]. consisting of 421 421 resolution fields with Dirichlet boundary. The low resolution data e.g. 22 x 22 are obtained via uniformly downsampling operation.
Navier-Stokes. Navier-Stokes a challenging time-series solving problem. We the public dataset from [2]. The viscosity of trajectories is le - 5 (Re = 2000). The spatial resolution is 64 x 64. The input and output time step length are both 10.
B.1.2 Irregular Domain
For these irregular domain problems, we take the exact same setting with NORM, more details about the dataset can be found in [9].
Irregular Darcy. The irregular Darey problem is to solve the Darcy Flow equation defined on an irregular domain. The input function (ax) represents the diffusion coefficient field and the output function u(x) is the corresponding pressure field. The domain represented with a set of triangle meshes consisting of 2290 nodes. Following [9], train the neural operators with 1000 data and test them on additional 200 trajectories.
Pipe Turbulence. Pipe Turbulence is dynamic fluid system described by the Navier-Stokes equation. The computational domain is an irregular pipe shape represented as 2673 triangle mesh nodes. In this problem. the neural operator is required to predict the velocity field in the next frame given the previous velocity field. Same as [9], 300 trajectories are employed for training, and 100 data are used for evaluation in experiments.
Heat Transfer. In the Heat Transfer problem, the energy transfer phenomena due temperature difference is studied. The system evolves under the governing law described by the Heat equation. The neural operator is optimized to predict the 3-dimensional temperature field seconds given the initial boundary temperature state. The output physical domain is represented by triangle meshes with 7199 nodes. We use 100 data for training and the rest of 100 data for evaluation.
Composite. Composite problem is to predict the deformation field in high-temperature stimulation, which is greatly significant for composites manufacturing. The learned operator expected predict the deformation field given the input temperature field. Following [9] The studied geometry in this work an air-intake structural part of a jet, which composed of 8232 nodes. The training data size is 400 and the test data size is 100.
Blood Flow. This problem aims to predict blood flow in the aorta, which contains inlet and outlets. The blood flow is simulated as homogeneous Newtonian fluid. The computational domain completely irregular and represented by set of triangle meshes comprising 1656 nodes. The simulated temporal length is 1.21 seconds with the temporal step length of 0.01 seconds. The neural operator predicts the velocity field at different times given the velocity boundary at the inlet and pressure boundary at the outlet. In this problem, we have 400 data for training and 100 data for testing, same as [9].
B.2
Base Model
Fourier Neural Operator (FNO) [2]. Fourier Neural Operator (FNO) [2] utilizes the Fourier Transform based integral operation implement the neural operator kernel K,, We implement the FNO with the officially published code under the up-to-date deep learning framework. All model hyperparameters except the hidden channel width (64 in our experiment, in the original implementation) are kept the same as in the original manuscript. The incremental hidden channels enhance the model's representation capability, obtaining consistent performance improvement in all settings. The optimization setup (Adam [57] optimizer with initial learning rate 0.001 and weight decay 0.0001, StepI. scheduler with step size 100 and 7 = 0.5) consistent with official implementation.
26

---

Factorized Fourier Neural Operator (FFNO) [10]. Factorized Fourier Neural Operator (FFNO) [10] conducts the Fourier Transform along every dimension independently, reducing the model size and enabling deep layer optimization. We implement FFNO via the independent Fourier Transform along each dimension, the improved residual connections, and the FeedForward-basee encoder-decoder. All model hyperparameters and optimization setups are consistent with FNO.
Clifford Fourier Neural Operator (CFNO) [11]. Clifford Fourier Neural Operator (CFNO) [11] employs the Clifford Algebra in the neural network architecture, incorporating geometry prior between multiple physical fields. We employ the official implementation for the model architecture. The signature is set as (-1, and we pad lacked physical channels with zero. All hyperparameters except the channel width (32 in CFNO) remain consistent with FNO. The training setup (including the optimizer and scheduler) is the same as FNO.
Galerkin [36]. Galerkin proposes to learn the neural operator with linear attention. We employ the same model architecture as the original work. The network depth is 4, the head number in the attention layer is set as and the latent dimension in the feedforward layer is 256. We use the Galerkin attention mechanism and set the dropout value as 0.05. In our experiment, we use the same training setup as FNO.
GNOT [21]. GNOT is also an heterogencous normalized attention layer to handling various inputs such as multiple system inputs and irregular domain meshes. We employ the official architecture implementation. The network depth is set as 4, the hidden dimension is 64, and the activation function is GeL.U. The training setup is the same as FNO.
neural operator structure, which introduces the
MiOnet [55]. MiOnet [55] is an enhanced version of DeepOnet [3]. It processes multiple inputs by utilizing multiple branch networks and then merging all branch outputs with several fully connected layers. In our implementation, the depth of all branch layers, trunk layer, and merging layer are set as 4. The latent dimension is set as 128. We take the same training setup with FNO.
Resnet [56]. Resnet [56] is a classical network architecture in image processing. Although it fails in zero-shot resolution generalization inference for learning infinite dimension operator mapping, the powerful capability of capturing local details deserves attention. We implement Resnet by simply replacing the spectral convolution in FNO with x 3 spatial convolution operation. All model hyperparameters and optimization setups remain consistent with FNO.
B.3
Implementation Details
Except for specific statements, the experimental details are as follows: (a) Retrieval: For training, the auxiliary sample retrieval range K is set as 20. During inference, the auxiliary sample with the most similarity with a, is retrieved. (b) Input: The customized input function is partial (last channels) 0x and complete uk,- (c) Optimizing: We use the Adam [57] optimizer with initial learning rate 0.001 and weight decay 0.0001. The StepLR scheduler with step size 100 and 7 = 0.5 is used to control the learning rate. The batch size is set 8, All models are trained for 500 epochs. All experiments could be conducted on a single NVIDIA GeForce RTX 4090 device.
B.4 Metric
The evaluated metric is Relative L2 Error, defined as:
[ u4 2
L2
(19)
|
where û; and un are predicted solution and real solution respectively.
C Additional Method Detail
C.1 Selection of K Value
The optimal value of K hard to determine. It depends on the generalization problem type (limited training data problem and resolution generalization problem in this work), the base models, etc.
27

---

Empirically, the larger the value of K. the better the model's generalization range, but the learning difficulty also increases. A proper value of K should ensure enough diversity of training residuals for generalizing to the targeted pending solved physical trajectories. Thus, it is advisable to set larger K value when the discrepancy between the training set and test set evident, and the model's representation ability is considerably strong. Through extensive experiments, we find K" = 20 is an applicable value for the focused generalization problem this work, ie. the limited amount and resolution of training data.
For more challenging generalization problems (eg. serious data bias problems in realistic scenarios), we provide an empirical selection algorithm for the "initial value" decision of K (the "final value" of K is also related to the base model's 's capability). Given the training set (noted as {atrain fi=l ( suppose the imput functions (noted as {a } Nenet) of pending solved physical trajectories is available. The initial value of K could be calculated with the following steps: (1) Retrieve the most similar sample (noted as {at ( (e from the training set for every pending solving function. We denote the similarity values between ae and afea as slest (2) Calculate the maximum value steet of (stear) Ti=1- (3) For every training sample a arrein calculate its similarity with other training trajectories. Denote the index of y-th similar sample with afrain as ky. and the similar score between a arrain and arrain sT. (4) The K is calculated as the minimum satisfying maa 8 for any E [1, Nfrain].
l==1
C.2 Complexity Analysis of Auxiliary Sample Retrieval
The computational cost for retrieval is negligible compared to neural network inference. Here we provide both theoretical complexity analysis and comprehensive experimental validation.
Theoretical Analysis. The computational complexity primarily depends on two factors: the training set size (denoted as Nerain) and the number of physical field points (i.e., the resolution of physical field) (denoted as Nfict). The retrieval process consists of two main steps:
Step I: Similarity Score Calculation. For each sample in the retrieval set, computing the similarity metric (c.g.. Cosine Similarity) has linear time complexity O(Nrieta). Thus, calculating similarity scores for all retrieved samples has complexity ( Nitain).
Step 2: Similarity Score Ranking. Using QuickSort to rank samples based on similarity scores has complexity O(Nirain log Ntrain).
The overall time complexity is O(N Ntvain + Nerain log Ntrain). With typical values of Niied and Nerni ranging from 100 to 10 and 102 to 10 respectively, the computation is feasible on standard hardware.
Experimental Validation. We conducted extensive experiments to measure the actual computational overhead of retrieval:
Retrieval Time Analysis: We tested retrieval time across different training set sizes on both CPU and GPU (single RTX 3090) for Darcy Flow (85 85 resolution):
Table 15: Retrieval time (in seconds) for different training set sizes
Device
100
300
500
700
900
CPU
2.67c-3
5.57e-3
9.05e-3
1.32e-2
1.66e-2
GPU
1.89e-4
1.83e-4
1.64e-4
2.43e-4
3.06e-4
End-to-End Inference Time: We compared the total inference time per example between direct learning and residual learning across different models:
Table 16: Average inference time (in seconds) comparison
Method
FNO
FFNO
Galerkin
Direct Learning
1.20e-3
2.39e-3
5.01e-3
Residual Learning
1.67e-3
2.85e-3
5.58e-3
Increased Time
4.67c-4
4.66e-4
5.75e-4
28

---

The experimental results demonstrate that:
GPU-based retrieval is highly efficient, taking only 0.2-0.3ms across different training sizes.
The total additional time for residual learning, including retrieval and data preparation, is consis- tently around 0.5ms.
These comprehensive experiments confirm that the auxiliary sample retrieval introduces negligible computational overhead and does not significantly impact the practical deployment of our method.
C.3 Discussion on Boundary Conditions
In this work, DeltaPhi does not explicitly process boundary conditions separate input. Instead, implicitly handles them by leveraging the full auxiliary solution Uk, as a strong physical prior. This auxiliary solution, which is key imput to the model, already satisfies the boundary conditions of its corresponding system. The model's effectiveness in handling complex boundary effects is demonstrated by its strong performance irregular domain problems, such as Pipe Turbulence and Blood Flow, which feature complex inlet/outlet conditions. While our current approach is effective, future work could explore enhancing the retrieval mechanism with boundary- similarity metric for enhanced residual neural operator.
D Visualization Analysis Detail
D.1 Similarity between u(x)) Visulization Detail
We visualize the correlation between u(x) normalized distance and a(x) similarity rank (shown in Figure 4) following steps: Firstly, taking the training set T retrieval set, retrieve auxiliary sample (ak,,,, M*,,,) ) with r-th top similarity score for every test sample (atest, ufeat) in the test set. Here represents the similarity rank (higher r indicates lower similarity), and we take gradually increasing values from to 80 for Next, we calculate the normalized distance between ugeat with auxiliary sample Uke.. The normalized distance defined the same as the relative error in Equation (19).
D.2 Label Distribution Visualization Detail
We visualize the label distribution (as shown in Figure by reducing the high-dimension function fields to 2 dimensions utilizing Principle Component Analysis. Specifically, for both direct operator mapping and residual operator mapping, we first optimize a PCA model on the training labels (100 labels) and then calculate the dimension-reduce Finally, we visualize every two-dimension label point on the graph as well as the ellipses representing the stand deviation and the range along each dimension.
labels for every sample in the training and testing set.
29


Reviews:
[{"review_id": "A6Vih9U2Yy", "reviewer": "Reviewer_eyAE", "summary": "The paper introduces a training paradigm for neural operators, DeltaPhi, based on residual learning between similar PDE solutions. The method makes the hypothesis that auxiliary trajectories are available and take advantage of the information provided, by learning the residual of the solution instead of the direct solution. Finally, the method is evaluated on several datasets and studied in different settings.", "questions": "-\tWhat if no similar solution are avaiblable at inference ? How are performance evolving wrt similarity between samples ? My question here is how does a « bad » input function would worsen the output prediction ? \n-\tHow does the method behaves on chaotic PDEs ? In such cases, eq 2 can be very sensitive, ie 2 PDEs with similar dynamics could exhibits very different behaviors from slightly different ICs. \n-\tHow long does the method takes for training ? I saw the inference time comparison, but does it effect NO training ? \n-\tIn lines 142-151/190-196, Have you tried other interpolation methods ? Like linear interpolation ? Direct downsampling ? Doesn’t the 0-padding changes the Fourier deocmposition of the function ? Have you tried other padding methods ? \n-\tIn tab1, what does ‘-‘ means ? \n-\tIs the proposed method applicable to other architectures ? \n-\tIn fig3, could you detail the experimental setting ? is it evaluated on $421\\times421$ grids ? \n-\tIn fig5 : how does the training set can be expanded while the testing set in smaller ? Is there a distribution shift in the datasets ?", "limitations": "Yes a limitation section is provided.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strenght :\n-\tThe paper is well-written and easy to follow\n-\tI found the idea interesting and experiment seems to show that the method effectively helps in learning operators. \n-\tSeveral experiment illustrate the claims\n\n### Weaknesses :\n-\tThe starting hypothesis is strong, meaning that one has access to some PDE solution even at inference. \n-\tExperiment could focus on more challenging datasets to deeply study the impact of the method on several contexts (see questions).", "was_revised": true, "final_justification": "DeltaPhi presents a new training paradigm for neural operator. The method is empirically evaluated on several standard datasets. \nI agree with other reviewers and I will keep my score to 4. \n\nI think additional studies on more challenging settings would convince the reader (eg on the meaning of the similarity between 2 samples, how it can be used in practice..), since it is the core of the method.", "rebuttal": "Thanks for your thoughtful reviews and insightful questions. They have significantly improved this work. We respond point-by-point below.\n\n> **[W1] Concerns on access to some PDE solution at inference.**\n\nWe would like to clarify that in data-driven PDE solving, the assumption of having access to some PDE solutions during inference is a common and low-cost practice, rather than a strong hypothesis.\n\n(a) First, the training of any neural operator requires a dataset of PDE solutions. These existing training data can be readily stored and utilized during the inference stage. This approach, which uses training data as a non-parametric memory at test time, is a common practice in other fields [1,2]. Therefore, we believe this is a standard and reasonable assumption.\n\n(b) In addition, the costs associated with this approach are well within acceptable limits for scientific computing applications. The storage requirement for the training examples is typically a few gigabytes. The time required to retrieve examples from the training set is also negligible, usually taking only a few seconds (see Section C.2).\n\n**Reference**\n- [1] Training data is more valuable than you think: A simple and effective method by retrieving from training data\n- [2] ResMem: Learn what you can and memorize the rest\n\n> **[Q1] Performance evolving wrt similarity between samples.**\n\nThanks for your insightful question, our analysis in Appendix A.2 has directly demonstrated that **DeltaPhi is robust with respect to the similarity of auxiliary samples and performs well even on a \"bad\" input function**.\n\n(a) To evaluate performance on dissimilar samples, we systematically divided the test set into 7 non-overlapping splits based on the similarity score between each test sample and its closest training sample, as shown in the table below. Splits with lower average scores (Split1, Split2 and Split3) represent more challenging, out-of-distribution test cases, directly simulating the effect of a \"bad\" input function you mentioned.\n\n|Split|Sample Number|Min Score|Max Score|Average Score|\n|---|---|---|---|---|\n|Split1 (Most Dissimilar)|23|0.85|0.86|0.86|\n|Split2|14|0.86|0.88|0.87|\n|Split3|30|0.88|0.90|0.89|\n|Split4|37|0.90|0.91|0.90|\n|Split5|33|0.91|0.93|0.92|\n|Split6|43|0.93|0.95|0.94|\n|Split7 (Most Similar)|16|0.95|0.96|0.95|\n\n(b) As presented in the following table, the results demonstrate that DeltaPhi consistently outperforms direct operator learning across all splits, even the most challenging ones that deviate significantly from the training set:\n\n|Method|Split1 (Most Dissimilar)|Split2|Split3|\n|---|---|---|---|\n|Direct Learning|5.08e-2|5.00e-2|4.71e-2|\n|DeltaPhi ($K$=20)|4.77e-2 (**6.06%** $\\uparrow$)|4.68e-2 (**6.34%** $\\uparrow$)|4.32e-2 (**8.28%** $\\uparrow$)|\n\nMore results of different splits are shown in Appendix A.2. These results empirically demonstrate that even when the retrieved auxiliary sample is not a close match, DeltaPhi effectively predicts the solution residual.\n\n> **[W2,Q2] Effectiveness on challenging datasets involving chaotic properties.**\n\nWe understand your mentioned \"more challenging datasets\" in \"Weakness 2\" means the complex problems involving chaotic properties, which are typically challenging for general neural operators [1,2]. While DeltaPhi does not solve the inherent difficulty of direct neural operators on such complex systems, **it is an effective training strategy that enhances the performance of existing operators even in these challenging cases**.\n\nTo substantiate this, we have included the challenging Navier-Stokes equation (with low viscosity 1e-5) in the submitted manuscript.\n(a) For handling the complex problem, we follow previous works [2] and learn the system's evolution over a limited, short-term time interval where the mapping from input to output remains well-defined and predictable (i.e. similar inputs leading to similar outputs).\n(b) Table 2 present the results. For FNO, we see a 4.86% gain, and for FFNO, an 8.54% gain. This demonstrates the robustness of DeltaPhi. Even when the stability property is weaker (i.e., $C$ in Equation 2 is larger), learning residuals between similar states is still more effective than learning the direct mapping from scratch. Our framework's ability to provide gains even in this difficult setting strengthens our claim of general applicability.\n\nTherefore, compared to classical direct operator learning, DeltaPhi does not introduce excessive challenges for modeling chaotic systems.\n\n**Reference**\n\n- [1] Deeponet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators\n- [2] Fourier neural operator for parametric partial differential equations\n\n\n> **[Q3] Training time cost of residual learning.**\n\nResidual operator learning does not introduce a significant training overhead compared to a standard neural operator.\n(a) Identical Training Steps: The number of training epochs and steps are kept exactly the same as in the direct neural operator baseline. During each training step, an input function is paired with a single, randomly chosen auxiliary sample. This enhances training diversity without requiring additional optimization steps.\n(b) Minimal Per-Step Cost: The additional cost introduced in each training step—sampling the auxiliary function, concatenation, and the residual connection—are computationally inexpensive.\n\nTo provide a quantitative comparison, we report the average training time per epoch for the Darcy Flow experiment on an NVIDIA RTX 4090 GPU. The results are as follows:\n\n|Model|Training Time / Epoch (s)|\n|---|---|\n|FNO|0.265 ± 0.014|\n|FNO-DeltaPhi|0.309 ± 0.019|\n\nAs the table shows, the increase in training time is minimal (only 0.05 second per epoch), confirming that DeltaPhi is efficient and comparable to the standard approach.\n\n> **[Q4] Questions on interpolation methods.**\n\nThe employed Fourier-based interpolation is plausible and advantageous for cross-resolution solution prediction. Zero-padding in the frequency domain is equivalent to ideal sinc interpolation in the spatial domain, which effectively upsamples the signal while **exactly preserving the original low-frequency components without distortion**. This aligns with the core principles of the FNO architecture for zero-shot resolution generalization, i.e. accurately capturing the low-frequency dynamics of the physical solution.\n\nWe tested other methods including Bilinear Interpolation and Nearest Interpolation on Darcy problem (training resolution is $85 \\times 85$ and testing resolution is $421 \\times 421$), as presented in the following table.\n\n|Interpolation Methods|Relative Error|\n|---|---|\n|Fourier Interpolation|6.91e-2|\n|Nearest Interpolation|7.39e-2|\n|Bilinear Interpolation|7.36e-2|\n\nThe results show that Fourier-based interpolation performs better for zero-shot resolution generalization. We will include the analysis in our final manuscript.\n\n> **[Q5] The meaning of '-' in Table 1.**\n\nIn Table 1, the symbol ‘-’ indicates that the baseline method could not be directly applied to the problem for the following reasons: (a) For the Heat Transfer and Blood Flow problems: GraphSAGE and FNO are not applicable because the input and output functions are defined on different physical domains. (b) For the Composite problem: FNO is not applicable because the physical domain is irregular.\n\n> **[Q6] Other architectures.**\n\nThe formulated residual mapping is a general operator learning task, fundamentally independent of the specific base architecture. Consequently, **any neural operator can be employed to learn this mapping**. In the following table, we have conducted new experiments integrating our method with an additional architectures Transolver (ICML24) [1] and HPM (ICML25) [2], on Irregular Darcy problem.\n\n|Model|Relative Error|\n|---|---|\n|Transolver (ICML24)|8.54e-3|\n|Transolver-DeltaPhi|8.18e-3|\n|HPM (ICML25)|7.39e-3|\n|HPM-DeltaPhi|6.67e-3|\n\nThe results show that DeltaPhi consistently improves the base models. We will include more base models across more problems in our final manuscript.\n\n**Reference**\n- [1] Transolver: A fast transformer solver for pdes on general geometries\n- [2] Holistic Physics Solver: Learning PDEs in a Unified Spectral-Physical Space\n\n> **[Q7] Experimental setting in Figure 3.**\n\nYes, the \"High Resolution\" curves are at resolution $421 \\times 421$.\n\n> **[Q8] Questions on Distribution Augmentation.**\n\nThe expanded training set distribution and more concentrated testing set distribution observed in Figure 5 are a result of **the asymmetric sampling mechanism in the residual learning framework**. \n(a) During training, we leverage the stability property to augment the diversity of training samples. This is achieved by pairing each input with auxiliary samples from a proper range of similarities. This process effectively expands the training distribution, exposing the model to a wider variety of physically valid residuals and enhancing its ability to generalize. \n(b) During testing, however, our goal is to achieve the most accurate prediction for a given input. To do this, we could select the single most similar auxiliary sample from the training set. This focused approach leads to a more concentrated distribution of residuals for the test set, as each prediction is based on the closest available reference.\n\nThis is the key advantage of residual learning framework, as it systematically improves the model's generalization performance by diversifying training while ensuring precision at inference. We will refine the explanation about this in final version to avoid any confusion."}, {"review_id": "NAKFWv3oNV", "reviewer": "Reviewer_7n6g", "summary": "This paper proposes DeltaPhi, a residual learning framework for neural operators that attempts to transform the PDE-solving task from directly predicting solutions to predicting residuals between similar physical states. The intuition behind this approach is interesting, and the authors substantiate their method with a variety of empirical studies across multiple PDE scenarios and data conditions. However, I do have several critical concerns arise regarding the mathematical rigor, foundational assumptions, and practical applicability of the proposed method.", "questions": "I have detailed my concerns in the weakness part. Below are some suggestions based on my concerns:\n- Provide rigorous derivations or explicit constraints ensuring the Lipschitz continuity condition within their implementation.\n- I suggest explicit analyses, empirical or theoretical, of the stability constant (C) to demonstrate its impact on the practical applicability of residual learning.\n- Please establish clear mathematical conditions and proofs regarding the well-posedness and structural properties of the residual operator's codomain\n- I suggest the authors to offer a detailed error analysis or theoretical justification for the use of interpolation methods in handling irregular or mixed-resolution data.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths: The core concept of leveraging residual learning by exploiting the assumed stability of PDE solution operators is interesting. The extensive empirical experiments and benchmarks presented by the authors illustrate potential advantages of the proposed approach.\n\nWeaknesses:\n- The paper's main hypothesis relies heavily on the Lipschitz continuity of PDE solution operators. While this assumption holds for elliptic and parabolic PDEs with regular boundary conditions, its validity for hyperbolic or chaotic systems is questionable. For chaotic PDEs, initially similar states typically diverge rapidly due to exponential sensitivity. Also, the paper lacks rigorous mathematical derivations or the enforcement of hard constraints that would guarantee or at least ensure adherence to Lipschitz continuity within the model implementation.\n- I do not see any explicit investigation or even mention the magnitude or potential variability of the stability constant (C). Given that a large or varying Lipschitz constant could dramatically amplify error propagation, undermining the very motivation of residual learning, it is crucial to address this aspect.\n- The residual operator defined by the authors, $G^\\Delta : A^2 \\rightarrow \\Delta U$, maps pairs of inputs to differences in solutions. However, the codomain $\\Delta U = \\{u_i - u_j \\mid u_i, u_j \\in U\\}$ is not guaranteed to be well-defined or structured as a proper Banach space unless $ U $ itself is explicitly shown to be a linear space closed under subtraction. The authors implicitly assume $ \\Delta U \\subseteq U $ without proving or even explicitly stating the necessary conditions for this assumption to hold true.\n- The subtraction $ u_i - u_j $ presupposes that both solutions $u_i $ and $ u_j $ are defined on identical grids or domains. In practice, this assumption is nontrivial, particularly for problems involving irregular geometries or mixed-resolution training sets. The authors address this via Fourier-based interpolation methods. Yet, interpolation inherently introduces approximation errors, particularly in sparse or irregularly sampled domains, and the paper provides no rigorous error analysis or guarantees regarding the accuracy or reliability of these interpolations.\n- The proposed mthod relies on cosine similarity to select \"similar\" auxiliary inputs. However, cosine similarity is not invariant under general isometries and is only naturally meaningful within Hilbert spaces. Applying cosine similarity as a measure of proximity in more general function spaces is mathematically unjustified and potentially misleading.\n- The authors assert that implicitly augmenting data through residual learning enhances training diversity. While intuitively plausible, this \"simplification hypothesis\" is presented without rigorous theoretical or mathematical support. There is no guarantee that learning residuals inherently simplifies the approximation task; indeed, it may inadvertently introduce complexity or even meaningless noise, making learning more challenging rather than simpler.", "was_revised": true, "final_justification": "I’ve reviewed the responses and am satisfied with most of the explanations. I am good to increase my score from 3 to 4.", "rebuttal": "Thanks for your thoughtful reviews and insightful questions. They have significantly improved this work. We respond point-by-point below.\n\n> **[W1,W2,Q1,Q2] Questions on Lipschitz continuity.**\n\nWe deeply appreciate your insightful questions about the Lipschitz continuity, chaotic systems and stability constant $C$. \nIn the following, we provide a systematic response to you in three parts (I, II and III).\nWe will carefully enhance this in the final version to mitigate confusions.\n\n**Part I: Clarification: Lipschitz Continuity is a Foundation for General Nueral Operators, Not an Added Hypothesis for DeltaPhi.**\n\nRegarding your primary concern \"DeltaPhi relies heavily on the Lipschitz continuity of PDE solution operators.\", we wish to clarify a crucial point: **DeltaPhi does not introduce Lipschitz continuity as an additional or stronger assumption compared to standard direct operator learning.**\n\nIn fact, the existence of a continuous mapping from input functions to solution functions (i.e., the stability of the PDE) is a fundamental prerequisite for *any* data-driven operator learning method to succeed [1], including the baselines we compare against. If small changes in the input could lead to arbitrarily large, discontinuous changes in the output, learning a generalizable mapping would be extremely challenging for neural networks [2].\n\nOur core contribution is not to enforce this property, but to propose a novel *training framework* that **more effectively leverages this inherent, pre-existing property** for improved data efficiency. While direct methods learn the mapping $a_i \\rightarrow u_i$, we reformulate the task to learn the mapping $(a_i, a_j) \\rightarrow (u_i - u_j)$. The feasibility of learning this residual mapping is founded on the same stability property that makes the original problem learnable. Our work demonstrates that this reformulation helps mitigate overfitting, especially in data-limited settings.\n\n**Reference**\n- [1] Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems\n- [2] Some fundamental aspects about lipschitz continuity of neural networks\n\n**Part II: Effectiveness of DeltaPhi on Complex Chaotic Problems.**\n\nYou are correct that for chaotic systems, \"initially similar states typically diverge rapidly,\" which implies that the Lipschitz constant $C$ can be large or grow over time. This is a well-known challenge for all long-term prediction models in such systems.\nHowever, this challenge affects **both direct operator learning and our residual learning framework equally.** It makes the underlying mapping harder to learn in general, but it does not invalidate our approach. The key question is whether DeltaPhi still provides a relative advantage over the baseline in these difficult scenarios.\n\nOur experiments on the **Navier-Stokes equation at a low viscosity of 1e-5 (a challenging regime)** provide a direct answer. Despite this challenge, **Table 2 shows that DeltaPhi consistently improves performance across all seven base models**. For FNO, we see a 4.86% gain, and for FFNO, an 8.54% gain. This demonstrates the robustness of DeltaPhi. Even when the stability property is weaker (i.e., $C$ is larger), learning residuals between similar states is still more effective than learning the direct mapping from scratch. Our framework's ability to provide gains even in this difficult setting strengthens our claim of general applicability.\n\nIn summary, we do not claim that DeltaPhi solves the inherent difficulty of such complex systems, but rather that it is a superior training strategy that enhances the performance of existing operators even in these challenging cases.\n\n**Part III. Analysis of the Stability Constant's ($C$) Impact on Residual Learning**\n\nWe agree that a large or varying stability constant $C$ makes the learning task more challenging. This is because it can amplify the magnitude and complexity of the output residual $u_i - u_j$ even for moderately different inputs $a_i$ and $a_j$.\n\nWhile $C$ is an intrinsic PDE property and often analytically intractable, our framework involves a practical tool to balance its effect: the retrieval range $K$. There is an inverse relationship between the difficulty imposed by $C$ and the optimal size of $K$. For a system with a larger $C$ (i.e., a more chaotic or less stable system), a smaller $K$ would be necessary to restrict sampling to only the most similar pairs, thereby keeping the learning task tractable. Conversely, for a more stable system with a smaller $C$, one can afford a larger $K$ to safely learn from more diverse samples and improve generalization.\n\nOur experiments empirically support this. The poor performance of random sampling (equivalent to an unconstrained $K$) in **Appendix A.1** shows that failing to manage the impact of $C$ harms performance. The same table shows that proper values of $K$ exist, balancing diversity and learning difficulty.\n\nBased on your valuable suggestion, we will add a detailed analysis of the stability constant $C$ and its relationship with the retrieval range $K$ to the final manuscript. For this work, we used $K=20$ for experimental consistency. We have provided extensive discussion on the influence of $K$, its impact on different test distributions, and alternative sampling strategies in Appendices A.1, A.2, C.1, and A.6.\n\n\n> **[W3,Q3] Concerns on definition of codomain.**\n\nWe highlight that **the codomain for residual mapping is the same well-defined banach space used in direct operator learning**. \nThis property follows directly from the foundational axioms of function spaces in PDE theory. Neural operator learning is premised on learning a mapping between Banach spaces, $\\mathcal{G}: \\mathcal{A} \\to \\mathcal{U}$. A key property of a Banach space $\\mathcal{U}$ is that it is a complete, normed vector space. By the closure axiom of vector spaces, if any two functions $u_i$ and $u_j$ are elements of $\\mathcal{U}$, their difference $u_i - u_j$ is also guaranteed to be an element of $\\mathcal{U}$. Consequently, the set of all possible outputs of our residual operator is a subset of $\\mathcal{U}$. Therefore, our residual learning task is well-posed, as it operates within the same well-structured Banach space as conventional neural operators, rather than mapping to an ill-defined space.\n\nWe will include this in the final manuscript based on your helpful suggestion.\n\n> **[W4,Q4] Concerns on the error of interpolation methods.**\n\nWe acknowledge that the interpolation operation inevitably introduces approximation errors. However, **the introduced error of Fourier interpolation does not hurt the core capability of neural operators in the resolution generalization problem**. For cross-resolution generalization, the base model, FNO [1] relies on preserving low-frequency information for prediction. Our interpolation method also precisely preserves these low-frequency components, while confining the approximation error to the high-frequency regions as the base model. While high-frequency error is inevitably introduced for both FNO and FNO-DeltaPhi, this specific error does not seriously hurt the model's stability on unseen resolutions. As the table below shows, our experimental results verify the importance of preserving low-frequency components, with Fourier interpolation outperforming other methods.\n\n|Interpolation Methods|Relative Error|\n|---|---|\n|Fourier Interpolation|6.91e-2|\n|Nearest Interpolation|7.39e-2|\n|Bilinear Interpolation|7.36e-2|\n\nIn summary, the introduced error of Fourier interpolation is effectively constrained in the high-frequency component, thus not affecting the stable prediction in the resolution generalization problem. \n\nFor your mentioned problem where \"samples lie on non-identical geometric domains\", DeltaPhi can be employed based on Geo-FNO [2]. It maps functions from various physical domains onto a single, unified grid. The residual connection would be directly performed in the unified grid, without the need for interpolation operation.\n\n**Reference**\n- [1] Fourier neural operator for parametric partial differential equations\n- [2] Fourier neural operator with learned deformations for pdes on general geometries\n\n> **[W5] Concerns on cosine similarity.**\n\nWe agree that cosine similarity has limitations, especially in the scenario with all-zero vectors (described by Reviewer cdGa) where it is undefined.\n\nHowever, we wish to clarify that **cosine similarity is a simple but effective choice for problems studied in current experiments**. (a) The function spaces for the PDEs (e.g., Darcy Flow, Navier-Stokes, Heat Transfer) are typically Sobolev spaces, which are indeed Hilbert spaces, **making cosine similarity a mathematically sound choice, not unjustified and misleading** (concerned by Reviewer 7n6g). (b) More importantly, our empirical results confirm its practical effectiveness. The metric effectively identifies input functions that lead to similar solutions as validated in Figure 4. And the consistent performance gains across all experiments also demonstrate its suitability.\n\nFurthermore, our framework is not restricted to a single similarity function. As we show in Appendix A.5, other metrics such as Euclidean (L2 mentioned by Reviewer cdGa) and Manhattan distance also yield strong performance, demonstrating the flexibility of the DeltaPhi framework. We present the results below for convenience:\n\n| Similarity Functions | Relative Error |\n|--- |---|\n| Cosine Similarity|3.31e-2|\n| Euclidean Distance|3.31e-2|\n| Manhattan Distance|3.28e-2|\n\nWe believe that exploring more advanced, domain-specific, or even learned similarity metrics is a promising direction for future research that could unlock further performance gains of DeltaPhi.\n\n> **[W6] Questions on the learning difficulty of residual mapping.**\n\nPlease see our response in `[W3]` to Reviewer `cdGa`."}, {"review_id": "sVK0Sg5Q1S", "reviewer": "Reviewer_qC6c", "summary": "This paper presents DeltaPhi, which is a novel residual learning framework for neural operator-based PDE solvers, and targets at scenarios where training data is scarce. Instead of learning direct input-output mappings, DeltaPhi learns residual mappings between pairs of similar physical states, leveraging the stability of PDE systems, which effectively augments the training distribution without additional data by generating multiple residual pairs. DeltaPhi is agnostic and compatible with various neural operators, and evaluated across regular and irregular domains, cross-resolution generalization, and varying data sizes.", "questions": "- For temporal PDEs (e.g., Navier-Stokes), the residuals are learned against earlier frames. In this case how does the error propagate over multi-step predictions?\n- Can DeltaPhi scale to very large-scale systems where each sample is massive?", "limitations": "Yes.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "- Strengths: \n  - Theoretical innovations: This paper reformulates PDE solving as residual learning between physical states. This formulation is well-grounded in physical principles (stability of PDEs).\n  - Implicit data augmentation: The method increases the diversity of training targets without requiring new data, directly addressing data scarcity.\n- Weaknesses: \n  - Additional overhead from the retrieval process: While efficient, it still introduces extra complexity that may hinder deployment in some real-time applications.\n  - Similarity metric: The Cosine similarity metric may not always align with physical relevance, especially for chaotic or nonlinear dynamics.\n  - No Discussion on boundary conditions: DeltaPhi may behave differently for out-of-distribution samples, such as in systems with sharp boundary effects or discontinuities.", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thoughtful reviews and insightful questions. They have significantly improved this work. We respond point-by-point below.\n\n> **[W1] Concerns on efficiency of retrieval.**\n\nWe would like to clarify that **the retrieval overhead from DeltaPhi is minimal and could be further enhanced for real-time applications**.\n\nFirst, our evaluation in Appendix C.2 has shown that the current implementation is highly efficient. The retrieval process introduces a negligible overhead of only 0.2–0.3 ms on a standard GPU, which is well within the acceptable limits for many real-time systems.\n\nFurthermore, for applications with even stricter latency requirements or larger datasets, our framework's scalability can be easily enhanced. Thanks to the robust research and development efforts [1,2], advanced retrieval tools can navigate vast amounts of data without a linear increase in retrieval time. For example, our framework can seamlessly incorporate well-supported, GPU-accelerated libraries like Faiss [3]. Faiss is an open-source library proven to retrieve vectors from billion-scale datasets in just a few milliseconds. By leveraging such tools, we can ensure DeltaPhi remains highly scalable and suitable for demanding, large-scale deployments.\n\n**Reference**\n- [1] Billion-scale similarity search with GPUsc\n- [2] Fast search in hamming space with multi-index hashing\n- [3] The faiss library\n\n> **[W2] Concerns on cosine similarity.**\n\nWe'd like to clarify that **cosine similarity is effective in the systems we studied, including those with complex dynamics**. While the dynamics of such systems are complex over long durations, they exhibit sufficient stability over the shorter time intervals typically used in neural operator-based PDE solving. This short-term stability ensures that physically similar initial states lead to similar subsequent trajectories, which is the foundation for cosine similarity to be a meaningful measure. Our strong results on the Navier-Stokes equation empirically validate this.\n\nIn addition, we agree that there is significant room for future research into optimal similarity metrics, especially for more complex physical systems or different problem settings. For instance, it is significant to explore performing retrieval in a latent space to capture more complex physical relationships. However, for the scope of this work, our results conclusively show that a simple and efficient metric like cosine similarity is sufficient to unlock the substantial benefits of residual learning.\n\n> **[W3] Discussion on boundary conditions.**\n\nThank you for this valuable comment. The behavior of neural operators under varying boundary conditions, especially for out-of-distribution samples with sharp effects, is a critical aspect of PDE solving. We provide a discussion below and will include it in the final manuscript.\n\nWhile DeltaPhi does not explicitly process boundary conditions, its core mechanisms and our results demonstrate its effectiveness in handling systems with sharp boundary effects or discontinuities. (a) First, DeltaPhi explicitly leverages the full auxiliary solution as input, which provides a strong physical prior on the boundary region of complex systems. (b) In addition, our results on irregular domains provide strong evidence of DeltaPhi's robustness in handling complex geometries and their complex boundary effects. In problems like Pipe Turbulence and Blood Flow, the domains are irregular with complex inlet/outlet boundary conditions. As shown in Table 1, DeltaPhi achieves very significant relative gains of 40.99% and 11.00%, respectively. This demonstrates that the residual learning mechanism is effective even when faced with the complex boundary effects.\n\nWe agree that extreme scenarios, such as generalizing to problems with extreme effects of boundary conditions, represent a challenging frontier for all neural operators. DeltaPhi could be potentially enhanced to better address this in the future. For example, the retrieval metric could be enhanced by designing a boundary-aware similarity metric that gives higher weight to boundary regions, ensuring a more proper auxiliary sample is chosen.\n\n> **[Q1] Questions about temporal pdes.**\n\nFirst, we wish to clarify that our framework does not learn the residual against previous frames of the same prediction sequence. Instead, it learns the residual between the target solution ($u_ i$) and a complete auxiliary trajectory ($u_{k_ i}$) retrieved from the training set.\n\nRegarding how error propagates in multi-step predictions, our method, like direct neural operators [1,2], is subject to error accumulation in auto-regressive rollouts. However, DeltaPhi provides a specific advantage in mitigating this effect. Specifically, residual learning approach helps stabilize the prediction by anchoring it to a known, physically valid trajectory ($u_{k_ i}$, which could be updated based on the status of last few time steps). Instead of predicting the full state from scratch at each step, the model only needs to learn the deviation from a plausible baseline. This is potentially beneficial for long-term predictions where direct methods can become unstable.\n\n**Reference**\n- [1] Fourier neural operator for parametric partial differential equations\n- [2] Transolver: A fast transformer solver for pdes on general geometries\n\n> **[Q2] Questions about large-scale systems.**\n\nWhile scaling to massive systems is challenging for general neural operators and beyond current work's core focus, DeltaPhi is potentially effective for such problems. (a) First, for systems with extremely high-resolution samples, similarity can be efficiently computed on downsampled data or important regions (such as boundaries), as core physical characteristics are preserved in lower-frequency components, bypassing the need to process the entire sample for retrieval. (b) In addition, for scenarios with a massive number of retrieval samples, the retrieval overhead is minimal, and the framework can integrate GPU-accelerated libraries like Faiss [1] to ensure millisecond-latency retrieval from billion-scale datasets. (c) The memory requirement for storing a large dataset is a general constraint in large-scale scientific computing and not a bottleneck unique to DeltaPhi. We will explore such large-scale problems in the future.\n\n**Reference**\n- [1] The faiss library"}, {"review_id": "ixmkOZFTi6", "reviewer": "Reviewer_cdGa", "summary": "This paper presents DeltaPhi, a novel framework for enhancing neural operator learning in data-scarce PDE solving tasks. DeltaPhi redefines the learning objective from direct input-output mapping to residual prediction between similar physical states. This reformulation leverages the stability properties of physical systems to enable implicit data augmentation without additional data collection.", "questions": "1. Have you considered using the auxiliary solution function as an input and directly predicting the target solution, instead of learning the residual? This would allow the model to use the auxiliary data as a form of guidance—conceptually similar to in-context or few-shot learning in large language models, where context from similar examples is used to guide predictions.\n\n2. How does DeltaPhi perform on Navier-Stokes with higher viscosity, e.g. 1e-3 from FNO?\n\n3. In table 4, it seems that the performance gain is larger with more training data, which is the opposite of the normal setting, can you give some reasons?\n\n4. What is the training and testing resolution for Darcy flow in Table 2? As you write following FNO (line 242), I assume the training and testing resolutions are all $85\\times 85$?\n\n5. Why is DeltaPhi only applied to one model in Table 1? Is it compatible with all models? All other reported errors has no meaning at all except for NORM.\n\nMinor issues:\n\n1. Do you only refer to kernel integral operators as neural operators? Equation (1) only describes kernel integral operators, it does not describe many other operators, e.g. DeepONet you use in your work.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strengths\n1. The idea of residual learning is very interesting. \n2. DeltaPhi is a plug-and-play module, compatible with diverse operator networks, making it highly flexible.\n3. Overall, the paper is easy to follow, and necessary details are clearly explained. \n\n# Weaknesses\n\n1. Using cosine similarity to measure the similarity between two functions is potentially problematic. For example, let's say you have two functions $u_1(x)=a$ on a small disk $D_a \\subset[0,1]^2$, zero elsewhere, and $u_2(x)=b$ on a different small disk $D_b$, disjoint from $D_a$, no matter what $a$ and $b$ is, the cosine similarity will always be 0. It is definitely problematic because suppose they are initial heat sources in a heat equation, if $a=b=0$, there is no residual at all. If $a$ is very different from $b$, the residual is large. Potentially, $L_2$ or cross-correlation might even be a better choice. However, I do believe the authors need to elaborate more the choice of the similarity metric.\n\n2. All the benefits of DeltaPhi is described verbally without any theoretical foundation. \n\n3. Although the implicit data augmentation adds more training pairs, we do not get any additional information from the training dataset. Also, I think the residuals can be, under some cases, more difficult to learn than the solutions. For example, even if the solutions are very smooth, the residuals may contain sharp transitions; high frequency components can dominate. This is harmful for the learning of neural networks, as well as neural operators. This is potentially the reason why the performance gain on the Navier-Stokes equation is minimal.\n\n4. This can be very subjective, but the performance gain is not very significant, especially for the Navier-Stokes equation on Table 2. The results suggest that the method might struggle with chaotic systems (e.g., Navier-Stokes at low viscosity).", "was_revised": true, "final_justification": "Overall, my concerns have been addressed. After reading the other reviewers' comments and giving the paper careful consideration, I have decided to keep my score as is, primarily for the following two reasons:\n\n1. The use of cosine similarity is limiting (I read the responses; however, I am not fully convinced), and there is no principled way to identify suitable similarity measures.\n\n2. While I agree with the authors that chaotic systems are challenging for all neural operators, they may pose even greater difficulty for the proposed method. Regarding the results provided: 1e-3 is not chaotic at all. It represents an overly simple and smooth dataset, which favors models like FNO and its variants that are designed to operate on low-frequency signals. Also from your results, we can already see that the performance gain from less smooth dataset (1e-4) is more marginal compared to that from smooth dataset (1e-3).", "rebuttal": "Thanks for your thoughtful reviews and insightful questions. They have significantly improved this work. We respond point-by-point below.\n\n> **[W1] Concerns on Cosine Similarity.**\n\nPlease see our response in `[W5]` to Reviewer `7n6g`.\n\n> **[W2] Questions on theoretical explanation.**\n\nWe understand your concern in the presentation of core benefits of DeltaPhi. However, we wish to highlight that proving theoretical guarantees for generalization in deep learning is notoriously challenging. Our explanation in Section 3.5 for DeltaPhi's benefit is intuitive yet powerful: **it effectively mitigates the overfitting in data-driven neural operators**, particularly in data-limited settings. This is because the model is less prone to memorizing labels when learning residuals.\n\nWhile a rigorous theoretical analysis remains challenging, DeltaPhi can be understood from a geometric perspective on the solution manifold. The residual learning framework effectively **shifts the learning paradigm from approximating isolated \"points\" (absolute solutions) to learning \"tangent vectors\" (solution residuals) that describe the solution manifold's local geometry**.  Learning these \"relational vectors\" is more powerful because it provides a richer learning signal and thus enables a form of implicit data augmentation.\n\nWe will enhance the discussion about benefits of DeltaPhi in our final manuscript based on your helpful suggestion.\n\n> **[W3] Questions on the learning difficulty of residual mapping and the benefits of DeltaPhi.**\n\nFirst, we clarify that our claim is not that DeltaPhi constitutes a \"simplification hypothesis\" (mentioned by Reviewer 7n6g). We agree that learning residual mapping can be more challenging. However, we highlight that this difficulty arises from the inherent complexity of the residual mapping itself—requiring the model to synthesize information from two input functions—**rather than from the introduction of harmful or meaningless high-frequency components**. If two solution functions are smooth, their residual will naturally be smooth, not introducing unnecessary sharpness (concerned by Reviewer cdGa). In addition, every training pair strictly adheres to the residual mapping, representing the exact difference between the solutions given two input functions.\n\nDespite the increased learning difficulty, our results demonstrate that **existing neural operators are fully capable of learning residual mapping** under original parameter and training configurations. Notably, the model even learns to implicitly respect the symmetry of the residual mapping without any explicit constraint, as detailed in Appendix A.7.\n\nThe core benefit of residual learning is its effectiveness in mitigating overfitting in data-limited scenarios. This is achieved in two aspects: (a) Because the predicted residual must adapt to different auxiliary samples, the model is discouraged from simply memorizing the limited training samples. (b) By randomly selecting auxiliary samples, we significantly increase the density and diversity of the training distribution. This directly addresses the problem of sparse training data in direct learning, making it more likely that the residual required for a given test sample (relative to its closest auxiliary sample) is within a space the model can generalize to.\n\nIn summary, the increased difficulty is not beyond the capability of existing neural operators, but alleviating overfitting by discouraging memorization, and the implicit data augmentation creates a denser and more diverse training distribution that improves generalization. This is precisely why DeltaPhi delivers performance gains in data-limited settings, as the model learns a more robust and generalizable representation of the underlying system. The consistent improvements, even on the challenging Navier-Stokes problem, validate such benefits.\n\n> **[W4,Q2] Concerns on performance of DeltaPhi on chaotic systems.**\n\nWe agree that extremely chaotic systems, where small input variations lead to huge output changes, present a difficult learning task. However, we would like to emphasize that this is a fundamental challenge for all data-driven neural operators, not a unique limitation introduced by DeltaPhi. \n\nOur core argument is that **within the general problems where direct neural operators are effective, DeltaPhi's residual learning framework, combined with its similarity-based retrieval mechanism, consistently improves generalization capability**. Our results empirically show that DeltaPhi achieves consistent gains on irregular/regular domain problems, various training numbers and cross-resolution scenarios over extensive base models.\n\n**Even for relatively complex systems like the Navier-Stokes equation, DeltaPhi remains its effectiveness.** \n(a) First, DeltaPhi has been equipped with explicit mechanism for eliminating the negative effects of chaotic properties in complex dynamics. By explicitly constraining the retrieval range $K$, the introduced similarity-based retrieval mechanism essentially limits the maximum difference between input pairs ($a_i,a_k$). This ensures the network learns from residuals ($u_i−u_k$) that are  manageable and less chaotic, making the residual learning feasible even when the system's stability is weaker (i.e. $C$ in Equation 2 is large). \n(b) In addition, our experiments show that even without problem-specific tuning of $K$ and using only a simple, single-step retrieval based on the input state, DeltaPhi still delivers performance gains across seven different backbone models (Table 2). This validates that learning residuals is a more effective strategy for generalization. Though beyond the core of current work, we believe performance could be further enhanced with more problem-specific designs, such as designing specific retrieval metric, re-retrieving auxiliary samples based on the most recent time step, etc.\n\nBased on your valuable suggestion, we include an additional comparison on the Navier-Stokes with relatively higher viscosity of 1e-3 and 1e-4. The output steps are 40 and 20 steps for viscosity 1e-3 and 1e-4, respectively.\n\n|Model|v=1e-4|v=1e-3|\n|---|---|---|\n|FNO|3.04e-1|1.21e-1|\n|FNO-DeltaPhi|2.43e-1|6.88e-2|\n\nThe results validate the consistent gains on Navier-Stokes problem. We will include more base models in our final manuscript.\n\n> **[Q1] Performance without residual connection.**\n\nThank you for the insightful question. While direct prediction seems intuitive, it has two main drawbacks compared to residual learning: (a) Failure to Utilize Auxiliary Input: The model is not effectively compelled to use the auxiliary information. Since the target output remains fixed for a given input, the network can simply learn to ignore the changing auxiliary sample, as it isn't necessary for producing the correct answer. (b) Lack of Label Diversity and Overfitting Risk: This approach fails to enhance training label diversity (the training labels are the same as in direct learning). It can lead the model to memorize the limited set of solutions, increasing the risk of overfitting. We provide the results in the table below:\n\n|Model|Relative Error|\n|---|---|\n|FNO|3.70e-2|\n|FNO-DeltaPhi (Residual Prediction)|3.31e-2|\n|FNO-DeltaPhi (Direct Solution Prediction)|4.38e-2|\n\n> **[Q3] Question on Table 4**\n\nWe understand your insightful confusion regarding the trend of relative gains. However, the relative performance gain is influenced by several interacting factors, not solely the amount of training data. Key factors include the resolution of the training data, the specific generalization task, and the representational capacity of the base model. \n\nFor instance, in the resolution generalization problem you referenced (Table 4), the model is trained on very low-resolution data but test on high-resolution data. In such a challenging scenario, the quality of the retrieved auxiliary sample becomes more critical for accurate residual prediction. Therefore, having a larger pool of data (900 samples vs. 100) significantly increases the likelihood of finding a suitable auxiliary sample, which in turn allows the residual learning framework to achieve a greater performance gain. \n\nIn summary, the trend in relative gains can indeed vary depending on the specific experimental configuration. This does not alter our core conclusion: residual learning consistently alleviates the performance decline of neural operators in data-limited scenarios.\n\n\n> **[Q4] Experiment setting in Table 2**\n\nYes, the training and testing resolution for Darcy flow in Table 2 are all $85 \\times 85$.\n\n> **[Q5] Question on applicability of DeltaPhi.**\n\nYes, DeltaPhi is a general framework that can be applied to all these neural operators. The reason we only reported the results of DeltaPhi with NORM in Table 1 is that the other baseline results were directly cited from the NORM [1]. Unfortunately, the official code for these baselines has not been open-sourced, which prevents us from conducting a fair and direct comparison by applying DeltaPhi to them.\n\nTo demonstrate its broader compatibility, we further test DeltaPhi on additional models for the irregular darcy problem, including Transolver (ICML 2024) [2] and HPM (ICML 2025) [3]. The results are presented below. We plan to include more baseline comparisons on irregular problems in the final version of our paper.\n\n|Model|Relative Error|\n|---|---|\n|Transolver|8.54e-3|\n|Transolver-DeltaPhi|8.18e-3|\n|HPM|7.39e-3|\n|HPM-DeltaPhi|6.67e-3|\n\n**Reference**\n- [1] Learning neural operators on riemannian manifolds\n- [2] Transolver: A fast transformer solver for pdes on general geometries\n- [3] Holistic Physics Solver: Learning PDEs in a Unified Spectral-Physical Space\n\n> **[Minor Issue] Definition of neural operator.**\n\nThank you for this valuable suggestion. We will revise Equation (1) in the final version to provide a more general formulation that is applicable to a broader class of neural operators like DeepONet series."}]

Author rebuttal:

Meta-review: The paper addresses an important challenge in data-driven PDE solving: neural operators need large, high-quality datasets, which are often expensive or unavailable. Direct input–output mapping in such regimes limits generalization. The authors build on the observation that many physical systems are stable, with nearby initial states leading to nearby solutions. They reformulate the task so that the model learns residuals between a target solution and an auxiliary one. This reframing reduces overfitting, increases training diversity without new data, and better exploits limited information.

The main contribution is the DeltaPhi framework, an architecture-agnostic residual learning strategy that can be combined with standard operators such as FNO, FFNO, or more recent solvers. Experiments show that DeltaPhi improves generalization in data-limited settings, across both regular and irregular domains, and in cross-resolution transfer, with only minimal computational overhead. Gains are consistent though not always large, including on Navier–Stokes. The contribution lies in providing a general training paradigm rather than a new operator design.

In conclusion, the paper has some weaknesses, such as the simple similarity metric and limited gains on chaotic systems, but the contribution is clear. The framework is original, addresses an important problem, and is supported by solid experiments and clarifications during rebuttal. Reviewers converged on the view that the strengths outweigh the weaknesses. I recommend acceptance.
Decision: Accept (poster)