Title: DeltaPhi: Physical States Residual Learning for Neural Operators in Data-Limited PDE Solving
Venue: NeurIPS 2025
Accepted: yes
Keywords: PDE

Operator Learning
Abstract: The limited availability of high-quality training data poses a major obstacle in data-driven PDE solving, where expensive data collection and resolution constraints severely impact the ability of neural operator networks to learn and generalize the underlying physical system. To address this challenge, we propose DeltaPhi, a novel learning framework that transforms the PDE solving task from learning direct input-output mappings to learning the residuals between similar physical states, a fundamentally different approach to neural operator learning. This reformulation provides implicit data augmentation by exploiting the inherent stability of physical systems where closer initial states lead to closer evolution trajectories. DeltaPhi is architecture-agnostic and can be seamlessly integrated with existing neural operators to enhance their performance. Extensive experiments demonstrate consistent and significant improvements across diverse physical systems including regular and irregular domains, different neural architectures, multiple training data amount, and cross-resolution scenarios, confirming its effectiveness as a general enhancement for neural operators in data-limited PDE solving.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents DeltaPhi, a novel framework for enhancing neural operator learning in data-scarce PDE solving tasks. DeltaPhi redefines the learning objective from direct input-output mapping to residual prediction between similar physical states. This reformulation leverages the stability properties of physical systems to enable implicit data augmentation without additional data collection.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n1. The idea of residual learning is very interesting. \n2. DeltaPhi is a plug-and-play module, compatible with diverse operator networks, making it highly flexible.\n3. Overall, the paper is easy to follow, and necessary details are clearly explained. \n\n# Weaknesses\n\n1. Using cosine similarity to measure the similarity between two functions is potentially problematic. For example, let's say you have two functions $u_1(x)=a$ on a small disk $D_a \\subset[0,1]^2$, zero elsewhere, and $u_2(x)=b$ on a different small disk $D_b$, disjoint from $D_a$, no matter what $a$ and $b$ is, the cosine similarity will always be 0. It is definitely problematic because suppose they are initial heat sources in a heat equation, if $a=b=0$, there is no residual at all. If $a$ is very different from $b$, the residual is large. Potentially, $L_2$ or cross-correlation might even be a better choice. However, I do believe the authors need to elaborate more the choice of the similarity metric.\n\n2. All the benefits of DeltaPhi is described verbally without any theoretical foundation. \n\n3. Although the implicit data augmentation adds more training pairs, we do not get any additional information from the training dataset. Also, I think the residuals can be, under some cases, more difficult to learn than the solutions. For example, even if the solutions are very smooth, the residuals may contain sharp transitions; high frequency components can dominate. This is harmful for the learning of neural networks, as well as neural operators. This is potentially the reason why the performance gain on the Navier-Stokes equation is minimal.\n\n4. This can be very subjective, but the performance gain is not very significant, especially for the Navier-Stokes equation on Table 2. The results suggest that the method might struggle with chaotic systems (e.g., Navier-Stokes at low viscosity).\n\nQUESTIONS:\n1. Have you considered using the auxiliary solution function as an input and directly predicting the target solution, instead of learning the residual? This would allow the model to use the auxiliary data as a form of guidance—conceptually similar to in-context or few-shot learning in large language models, where context from similar examples is used to guide predictions.\n\n2. How does DeltaPhi perform on Navier-Stokes with higher viscosity, e.g. 1e-3 from FNO?\n\n3. In table 4, it seems that the performance gain is larger with more training data, which is the opposite of the normal setting, can you give some reasons?\n\n4. What is the training and testing resolution for Darcy flow in Table 2? As you write following FNO (line 242), I assume the training and testing resolutions are all $85\\times 85$?\n\n5. Why is DeltaPhi only applied to one model in Table 1? Is it compatible with all models? All other reported errors has no meaning at all except for NORM.\n\nMinor issues:\n\n1. Do you only refer to kernel integral operators as neural operators? Equation (1) only describes kernel integral operators, it does not describe many other operators, e.g. DeepONet you use in your work.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-04"}

comments: SUMMARY:
This paper presents DeltaPhi, which is a novel residual learning framework for neural operator-based PDE solvers, and targets at scenarios where training data is scarce. Instead of learning direct input-output mappings, DeltaPhi learns residual mappings between pairs of similar physical states, leveraging the stability of PDE systems, which effectively augments the training distribution without additional data by generating multiple residual pairs. DeltaPhi is agnostic and compatible with various neural operators, and evaluated across regular and irregular domains, cross-resolution generalization, and varying data sizes.

STRENGTHS AND WEAKNESSES:
- Strengths: 
  - Theoretical innovations: This paper reformulates PDE solving as residual learning between physical states. This formulation is well-grounded in physical principles (stability of PDEs).
  - Implicit data augmentation: The method increases the diversity of training targets without requiring new data, directly addressing data scarcity.
- Weaknesses: 
  - Additional overhead from the retrieval process: While efficient, it still introduces extra complexity that may hinder deployment in some real-time applications.
  - Similarity metric: The Cosine similarity metric may not always align with physical relevance, especially for chaotic or nonlinear dynamics.
  - No Discussion on boundary conditions: DeltaPhi may behave differently for out-of-distribution samples, such as in systems with sharp boundary effects or discontinuities.

QUESTIONS:
- For temporal PDEs (e.g., Navier-Stokes), the residuals are learned against earlier frames. In this case how does the error propagate over multi-step predictions?
- Can DeltaPhi scale to very large-scale systems where each sample is massive?

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-20

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes DeltaPhi, a residual learning framework for neural operators that attempts to transform the PDE-solving task from directly predicting solutions to predicting residuals between similar physical states. The intuition behind this approach is interesting, and the authors substantiate their method with a variety of empirical studies across multiple PDE scenarios and data conditions. However, I do have several critical concerns arise regarding the mathematical rigor, foundational assumptions, and practical applicability of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: The core concept of leveraging residual learning by exploiting the assumed stability of PDE solution operators is interesting. The extensive empirical experiments and benchmarks presented by the authors illustrate potential advantages of the proposed approach.\n\nWeaknesses:\n- The paper's main hypothesis relies heavily on the Lipschitz continuity of PDE solution operators. While this assumption holds for elliptic and parabolic PDEs with regular boundary conditions, its validity for hyperbolic or chaotic systems is questionable. For chaotic PDEs, initially similar states typically diverge rapidly due to exponential sensitivity. Also, the paper lacks rigorous mathematical derivations or the enforcement of hard constraints that would guarantee or at least ensure adherence to Lipschitz continuity within the model implementation.\n- I do not see any explicit investigation or even mention the magnitude or potential variability of the stability constant (C). Given that a large or varying Lipschitz constant could dramatically amplify error propagation, undermining the very motivation of residual learning, it is crucial to address this aspect.\n- The residual operator defined by the authors, $G^\\Delta : A^2 \\rightarrow \\Delta U$, maps pairs of inputs to differences in solutions. However, the codomain $\\Delta U = \\{u_i - u_j \\mid u_i, u_j \\in U\\}$ is not guaranteed to be well-defined or structured as a proper Banach space unless $ U $ itself is explicitly shown to be a linear space closed under subtraction. The authors implicitly assume $ \\Delta U \\subseteq U $ without proving or even explicitly stating the necessary conditions for this assumption to hold true.\n- The subtraction $ u_i - u_j $ presupposes that both solutions $u_i $ and $ u_j $ are defined on identical grids or domains. In practice, this assumption is nontrivial, particularly for problems involving irregular geometries or mixed-resolution training sets. The authors address this via Fourier-based interpolation methods. Yet, interpolation inherently introduces approximation errors, particularly in sparse or irregularly sampled domains, and the paper provides no rigorous error analysis or guarantees regarding the accuracy or reliability of these interpolations.\n- The proposed mthod relies on cosine similarity to select \"similar\" auxiliary inputs. However, cosine similarity is not invariant under general isometries and is only naturally meaningful within Hilbert spaces. Applying cosine similarity as a measure of proximity in more general function spaces is mathematically unjustified and potentially misleading.\n- The authors assert that implicitly augmenting data through residual learning enhances training diversity. While intuitively plausible, this \"simplification hypothesis\" is presented without rigorous theoretical or mathematical support. There is no guarantee that learning residuals inherently simplifies the approximation task; indeed, it may inadvertently introduce complexity or even meaningless noise, making learning more challenging rather than simpler.\n\nQUESTIONS:\nI have detailed my concerns in the weakness part. Below are some suggestions based on my concerns:\n- Provide rigorous derivations or explicit constraints ensuring the Lipschitz continuity condition within their implementation.\n- I suggest explicit analyses, empirical or theoretical, of the stability constant (C) to demonstrate its impact on the practical applicability of residual learning.\n- Please establish clear mathematical conditions and proofs regarding the well-posedness and structural properties of the residual operator's codomain\n- I suggest the authors to offer a detailed error analysis or theoretical justification for the use of interpolation methods in handling irregular or mixed-resolution data.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a training paradigm for neural operators, DeltaPhi, based on residual learning between similar PDE solutions. The method makes the hypothesis that auxiliary trajectories are available and take advantage of the information provided, by learning the residual of the solution instead of the direct solution. Finally, the method is evaluated on several datasets and studied in different settings.\n\nSTRENGTHS AND WEAKNESSES:\n### Strenght :\n-\tThe paper is well-written and easy to follow\n-\tI found the idea interesting and experiment seems to show that the method effectively helps in learning operators. \n-\tSeveral experiment illustrate the claims\n\n### Weaknesses :\n-\tThe starting hypothesis is strong, meaning that one has access to some PDE solution even at inference. \n-\tExperiment could focus on more challenging datasets to deeply study the impact of the method on several contexts (see questions).\n\nQUESTIONS:\n-\tWhat if no similar solution are avaiblable at inference ? How are performance evolving wrt similarity between samples ? My question here is how does a « bad » input function would worsen the output prediction ? \n-\tHow does the method behaves on chaotic PDEs ? In such cases, eq 2 can be very sensitive, ie 2 PDEs with similar dynamics could exhibits very different behaviors from slightly different ICs. \n-\tHow long does the method takes for training ? I saw the inference time comparison, but does it effect NO training ? \n-\tIn lines 142-151/190-196, Have you tried other interpolation methods ? Like linear interpolation ? Direct downsampling ? Doesn’t the 0-padding changes the Fourier deocmposition of the function ? Have you tried other padding methods ? \n-\tIn tab1, what does ‘-‘ means ? \n-\tIs the proposed method applicable to other architectures ? \n-\tIn fig3, could you detail the experimental setting ? is it evaluated on $421\\times421$ grids ? \n-\tIn fig5 : how does the training set can be expanded while the testing set in smaller ? Is there a distribution shift in the datasets ?\n\nLIMITATIONS:\nYes a limitation section is provided.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-04"}