Sensitivity-Constrained Fourier Neural Operators for Forward and Inverse Problems in Parametric Differential Equations

Authors: Abdolmehdi Behroozi, Chaopeng Shen, Daniel Kifer
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: fourier neural operator, sensitivity analysis, parametric differential equations, surrogate modes, differentiable numerical solvers, inverse problems

Abstract:
Parametric differential equations of the form  $\frac{\partial u}{\partial t} = f(u, x, t, p)$ are fundamental in science and engineering. While deep learning frameworks like the Fourier Neural Operator (FNO) efficiently approximate differential equation solutions, they struggle with inverse problems, sensitivity calculations  $\frac{\partial u}{\partial p}$, and concept drift.  We address these challenges by introducing a novel sensitivity loss regularizer, demonstrated through Sensitivity-Constrained Fourier Neural Operators (SC-FNO). Our approach maintains high accuracy for solution paths and outperforms both standard FNO and FNO with Physics-Informed Neural Network regularization. SC-FNO exhibits superior performance in parameter inversion tasks, accommodates more complex parameter spaces (tested with up to 82 parameters), reduces training data requirements, and decreases training time while maintaining accuracy.  These improvements apply across various differential equations and neural operators, enhancing their reliability without significant computational overhead (30%–130% extra training time per epoch). Models and selected experiment code are available at:  [https://github.com/AMBehroozi/SC_Neural_Operators](https://github.com/AMBehroozi/SC_Neural_Operators).

Decision: accept (poster)
Reviewer scores: 8, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1729393420886

thread_type:
official_review

<thread_object>
creator: reviewer_wfxg

<content>
summary:
This paper proposes a sensitivity-constrained loss in the original FNO model, making it suitable for ODE/PDE inversion problems. Definitely this loss can make the prediction more stable with respect to the physical system parameters. This loss works with a wide range of neural operators and other supervising sensitivity tasks. The paper is well-written. The method is effective compared to the original FNO and FNO-PINN. The experiments on five ODE/PDE problems are impressive.

strengths:
1.The paper focus on the sensitivity prediction to physical ODE/PDE systems, which is important. 

2.The paper’s contribution is clear, and the paper is easy to follow.

3.The experiments are diverse.

4.The paper’s improvements help enhancing the reliability and applicability of neural operators in complex physical systems modeling and engineering analysis.

weaknesses:
1.The idea of using gradient loss to guarantee sensitivity is not new, such as the gradient PINN and others. 

2.The sensitivity loss is supervised, which means additional labeled data {\partial u}/{\partial p} are needed compared to original FNO. On the other hand , if we have additional labeled data, it is directly to add the supervised loss in the total loss, which is SC-FNO exactly, so I think the novelty is rather limited. Although the {\partial u}/{\partial p}  can also be approximately computed from the original data, the numerical error influence is not analyzed. On the contrary, the PINN loss is unsupervised.

3.The SC-FNO loss is hybrid, the training process should be reflected in the main text or appendix. SC-FNO should be hard to train compared to FNO, but the author didn’t mention the difference.

4.Since the PINN-FNO also add the PINN loss as the regularizer to guarantee the prediction of physical parameters, I wonder why it behaves bad to SC-FNO. Maybe the training difficulty is different. The author use a learnable coefficients combination of the hybrid loss Liebel & Körner (2018) in Algorithm 3, but FNO-PINN may be trained well for adaptive hybrid loss as in Wang S, Wang H, Perdikaris P. Improved architectures and training algorithms for deep operator networks[J]. Journal of Scientific Computing, 2022, 92(2): 35.

5. SC-FNO architecture,  as well as the Algorithm, should be exhibited in the main text,  not the appendix.

questions:
See in the weaknesses 2-4.

Why use R^2 as metric, why not the relatively L^2 norm as in the original FNO paper？The metric should be comparable to existing works.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730607092715

thread_type:
official_review

<thread_object>
creator: reviewer_kldr

<content>
summary:
This paper introduces a sensitivity loss regularizer for neural operators, specifically applying it to the Fourier Neural Operator (FNO) framework, resulting in Sensitivity-Constrained Fourier Neural Operators (SC-FNO). SC-FNO improves accuracy in solution paths, inverse problem-solving, and sensitivity calculations, even under sparse data and concept drift. It outperforms both standard FNO and FNO with PINN regularization in parameter inversion tasks, achieving high accuracy without large computational or memory costs. These enhancements extend the applicability and reliability of neural operators in modeling complex physical systems.

strengths:
1. The paper's proposed Sensitivity-Constrained approach is applicable to various neural operators, offering high research value.
2. The paper is clearly written, with no redundant mathematical derivations, and offers excellent readability.
3. In modeling PDEs, this approach not only enables accurate approximation of the target variable but also effectively models its derivatives.

weaknesses:
1. There is no enough discussion on the memory overhead introduced by AD (Automatic Differentiation) and FD (Finite Differences).
2. The experiments are limited. It is not sure whether this method remains effective in complicated and high-resolution PDE scenarios.

questions:
In above part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730715887107

thread_type:
official_review

<thread_object>
creator: reviewer_dfuf

<content>
summary:
In this paper, the authors propose a regularization technique that examines the gradient of the solution function with respect to parameters in differential equations. To the best of my knowledge, this approach to sensitivity regularizer, aimed at enhancing the generalization ability of neural operators, is unique. The paper is well structured, with clear motivation and objectives.

strengths:
The study addresses the limitations observed when Fourier neural operators, combined with a physics-informed loss function, may yield suboptimal performance. The proposed regularization method appears to mitigate these challenges effectively.

weaknesses:
The proposed approach requires additional information and computational resources, as it leverages direct solution-based information, such as a precomputed Jacobian in the differentiable numerical solver (Eq, (6)). 

Another concern is the regularizer’s potential weakness to noise in practical settings. Derivative-based techniques are typically vulnerable to data noise, which could affect prediction accuracy, especially in applications beyond simulation, such as parameter inversion. In real-world applications, obtaining accurate derivative information can be challenging. While the finite difference methods employed in this paper are an option, they may perform poorly under noisy conditions.

questions:
1. Could the authors elaborate on the approach to ensure fair comparisons between the standard Fourier neural operator and the proposed method in Tables 1, 2, and 3? Including details on training times, model complexity, and the number of learnable parameters would enhance the rigor of the paper. 

2. Higher-order derivatives might be utilized in constructing the loss function (as in Equation (6)). It would be beneficial if the authors discussed the potential advantages or limitations of incorporating these higher-order terms. 

3. Although Fourier neural operators are generally applicable to rectangular domains, other neural operators are available even for non-rectangular domains. Given that the authors provided results using DeepONet in D.6, could they consider including experiments across various domain shapes or grid resolutions to further validate the robustness of the proposed approach?
 
4. Has the proposed method been tested in scenarios involving bifurcation, where small parameter changes induce rapid solution changes (e.g. the exponential model y’=py with p near zero)? 

5. The computational cost associated with this model may increase significantly as the number of parameters grows, or if parameters are represented as functions rather than scalars. While results are presented for a four-parameter case in Table 1, could authors consider examining more complex cases – such as those with over ten parameters, function-based parameters, or parameters with probability density function across a range?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730836438832

thread_type:
official_review

<thread_object>
creator: reviewer_a6gh

<content>
summary:
In this paper, the author proposes a sensitivity loss as a regularizer for Fourier neural operator, called SC-FNO. Specifially, the sensitivity is measured by the derivative (jacobian) du/dp, where p is the physical parameter. The jacobian du/dp is computed via auto differentiation. Experiments are conducted on 2 ODEs and 2 PDEs. It shows that SC-FNO outperforms previous FNO and Pinn-FNO.

strengths:
- This paper examines a notable case involving the system parameter $P$, where $ \frac{du}{dp} $ is computed as a sensitivity loss to regularize the model.
- The results indicate that SC-FNO achieves greater accuracy in $ \frac{du}{dp} $ and demonstrates enhanced robustness against perturbations in $p$.
- This approach aids inverse modeling in identifying the system parameters more effectively.

weaknesses:
### Writing
The overall writing quality could be enhanced, with several important details suggested to be moved from the appendix into the main text.
- The physical parameter $P$ is central to the study and should be introduced more thoroughly. Currently, it is presented in a general form, but it’s important to clarify whether $P$ is a scalar in $\mathbb{R}$, a vector in $\mathbb{R}^d$, or a function in $L(\mathbb{R})$.
- Starting with a concrete example in the introduction would improve clarity and reader engagement.
- On Page 5, while introducing the ODEs and PDEs, the nature of the parameter $P$ remains ambiguous. Explicitly writing out the equation would help clarify its role.

### Methods
- It is unclear how $P$ is incorporated into the FNO model. If $P$ is a scalar, is it treated as a constant function and added as an additional channel?
- Computing the derivative requires additional runtime and memory, which should be discussed in detail.

### Experiments
- SC-FNO appears to achieve higher accuracy primarily on $\frac{du}{dp}$ but not significantly on the solution $u$ itself.
- Runtime and memory usage should be addressed, especially in comparison to alternative approaches.

questions:
- What are the runtime and memory consumption, and which is the primary factor to consider?
- Can the model handle functional input such as $ \frac{du}{du_0} $? In this case, the Jacobian could be huge.
- Can the model account for parameters like the Reynolds number or viscosity as $P$?
- Can the model work on chaotic system such as the lorenz system, where the underlying parameter is very sensitive?

---
updates: the rebuttal addressed most of my questions. I am happy to raise my score to 8.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733312061628

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of the discussion and revision

comment:
We sincerely thank all the reviewers for their thoughtful and constructive feedback, which has significantly improved the quality of our manuscript. Throughout the discussion period, we carefully addressed all identified concerns and incorporated additional experiments and clarifications into the revised version. Below is a summary of our key updates:

### 1. Improved Metrics and Enhanced Clarity
- The revised manuscript added L2 errors as a metric, clearly showing SC-FNO’s substantial advantage, particularly during inversion tasks, where it achieves errors as low as 1/5 or 1/6 of FNO’s inversion errors in even low-dimensional problems.
- We provided a detailed explanation of how sensitivity supervision (\(\partial u / \partial p\)) is integrated into the SC-FNO framework and clarified its distinction from existing approaches like FNO and FNO-PINN. Our selective sampling strategy and efficient training procedures are now thoroughly detailed.
- We clarified that both FNO and SC-FNO use identical neural network architectures, differing only in loss configurations. This ensures a fair comparison. Additionally, we included a section detailing how parameters are processed through the lifting layer alongside spatial coordinates and initial conditions.

### 2. Additional Experiments
- New experiments on bifurcation-rich systems (e.g., the Allen-Cahn equation in the revised manuscript and the Ginzburg-Landau equation in the rebuttal) validate SC-FNO's robustness.
- An 82-parameter case (Burger’s equation) was added to the revised manuscript, demonstrating SC-FNO’s significantly smaller errors for both the state variable and sensitivities. Even with only 100 training samples and less training time, SC-FNO achieves less than 1/3 of the error of FNO trained with 500 samples.
- We also conducted experiments on Navier-Stokes equations, using varying Reynolds numbers to demonstrate SC-FNO's strong performance even with limited training data.
- Collectively, these results highlight SC-FNO’s ability to maintain superior performance, especially under limited data, high-dimensional parameter and perturbed parameter scenarios.

### 3. Computational Feasibility
- We conducted detailed runtime and memory analyses, showing that SC-FNO introduces only modest computational overhead while significantly improving accuracy and robustness.
- SC-FNO can reduce data requirements (and thus training time), making it a practical and efficient tool for sensitivity supervision and generalization.

### 4. Broader Context
- We emphasized SC-FNO's novelty as a sensitivity-supervised variant of neural operators, showcasing its unique strengths in addressing parameter-sensitive tasks.

We emphasize that SC-FNO's accuracy during inversion and robustness under perturbations directly address critical limitations of existing neural operators and overcome longstanding challenges in their practical applications. SC-FNO's demonstrated ability to capture sensitivities for \(10^2\) parameters represents a noteworthy advancement, improving both state variable predictions and parameter inversion tasks. Additionally, we are actively investigating its application to very high-dimensional inputs and are highly confident in its potential based on preliminary findings.

We are deeply grateful for the reviewers' engagement, which has helped refine our work and articulate its broader impact. We believe our contribution offers a unique perspective and a noteworthy advancement in neural operator methodologies, with far-reaching implications for parameter-sensitive modeling tasks.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The work proposes imposing a new loss when learning physical systems which is based on the derivative of the system with respect to the parameter considered. Numerical experiments show greater stability of the learned model, making it more performant especially for applications to inverse problems.