Sensitivity-Constrained Fourier Neural Operators for Forward and Inverse Problems in Parametric Differential Equations
Authors: Abdolmehdi Behroozi
Chaopeng Shen
Daniel Kifer
Venue: ICLR 2025 Poster (2025)
Keywords: Fourier Neural Operator
Sensitivity Analysis
Parametric Differential Equations
Surrogate Modes
Differentiable Numerical Solvers
Inverse Problems
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: Introducing a novel Sensitivity-Constrained Fourier Neural Operator (SC-FNO) to improve the accuracy of parametric differential equation solutions and their sensitivities using differentiable numerical solvers.

Abstract: Parametric differential equations of the form  $\frac{\partial u}{\partial t} = f(u, x, t, p)$ are fundamental in science and engineering. While deep learning frameworks like the Fourier Neural Operator (FNO) efficiently approximate differential equation solutions, they struggle with inverse problems, sensitivity calculations  $\frac{\partial u}{\partial p}$, and concept drift.  We address these challenges by introducing a novel sensitivity loss regularizer, demonstrated through Sensitivity-Constrained Fourier Neural Operators (SC-FNO). Our approach maintains high accuracy for solution paths and outperforms both standard FNO and FNO with Physics-Informed Neural Network regularization. SC-FNO exhibits superior performance in parameter inversion tasks, accommodates more complex parameter spaces (tested with up to 82 parameters), reduces training data requirements, and decreases training time while maintaining accuracy.  These improvements apply across various differential equations and neural operators, enhancing their reliability without significant computational overhead (30%–130% extra training time per epoch). Models and selected experiment code are available at:  [https://github.com/AMBehroozi/SC_Neural_Operators](https://github.com/AMBehroozi/SC_Neural_Operators).

Published as a conference paper at ICLR 2025
SENSITIVITY-CONSTRANNE FOURIER NEURAL OPER- ATORS FOR FORWARD AND INVERSE PROBLEMS IN PARAMETRIC DIFFERENTIAL EQUATIONS
Abdolmehdi Behroozi & Chaopeng Shen
Daniel Kifer
Department of Civil and Environmental Engineering
School of Electrical Engincering and Computer Science
Penn State University
Penn State University
University Park, PA 16802-1408, USA (amb10399, cxs1024 ) @psu. edu
University Park, PA 16802-1408, USA
duk17@psu.edu
ABSTRACT
Parametric differential equations of the form Ou = = fu,, p) are fundamental in science and engineering. While deep learning frameworks like the Fourier Neural Operator (FNO) efficiently approximate differential equation solutions, they struggle with inverse problems, sensitivity calculations 8p and concept drift.
Dt
We address these challenges by introducing a novel sensitivity loss regularizer, demonstrated through Sensitivity-CCnstraimn
Fourier Neural Operators (SC-FNO). Our approach maintains high accuracy for solution paths and outperforms both standard FNO and FNO with Physics-Infommet Neural Network regularization. SC- FNO exhibits superior performance parameter inversion tasks, accommodates more complex parameter spaces (tested with up to 82 parameters), reduces training data requirements, and decreases training time while maintaining accuracy. These improvements apply across various differential equations and neural operators, enhancing their reliability without significant computational overhead (30%-130% extra training time per epoch). Models and selected experiment code are available at:
com/ http///Meeeeoi//// /SC SC___s______ /eeraa/
INTRODUCTION
Ordinary and Partial Differential Equations (ODEs and PDEs), which contain physically meaningful parameters p. form the foundation of most modern engineering systems across diverse fields such fluid mechanics, medicine design, molecular dynamics, relativistic physics, climate, and environ- mental sciences (Palais & Palais 2009). These physical parameters p. whether describing fixed or evolving system properties (Feng et 2022), physiological constants (Reichert & Omlin (997). environmental characteristics (Taatakkvsky et al. 2020), or serving as simplifications for subgrid-scale processes Yuval O Gorman 22200] Meyt-Meyerree al. 2024), significantly influence the behavior of the equations. Neural Operators are neural networks whose inputs are initial conditions and physical parameters, and whose output function u, also known as a solution path, that approximates the solution to the ODE/PDE. When the values of are known, neural operators can be used for simulations (predicting system state u(t) at time t). However, in many cases, there is uncertainty about the values of p, they must be estimated from observations of u. This is known as (parameter) inversion. The estimated p can then used to simulate u beyond existing observations, or they may themselves be quantities of interest to scientists. Other potential uses of neural operators include sensitivity analysis (i.e., estimating Ou/ōp) and applying them situations not covered by training data. Even state-of-the-ar neural operators struggle with these applications and this paper presents method to address this problem.
al. 2021 2024) leverage the linear nature of differential operators in Fourier space, delivering orders of magnitude of computa- tional efficiencies compared to traditional numerical solvers. They are currently the most popular type of neural operator. FNOs are known for their meshless or discretizationninvaate charaatrrrtaaatttisiiiicc..
Fourier Neural Operators (FNOs). Introduced in 2021, FNOs (L FNOs
"Corresponding author

---

Published as a conference paper at ICLR 2025
and some versions eliminate the need for time-stepping, allowing them to output solutions at any spatiotemporal resolution. FNOs achieve these improvements over prior surrogate models (Audet) et al. 2000 Alizadeh et al. (020) by utilizing representations in the frequency domain efficiently learn time trajectories of the evolution of initial conditions. Learning the trajectories in time domain would necessitate large amounts of weights and extensive training data.
While this method offers significant efficiency, prediction accuracy may decline for predictions far into the future (Grady et al. 2023), prompting the use of additional constraints in some models (Jiang et al. 2023 Bonev et al. (203). FNO is so far mostly used in forward simulations and research; the critical issues of parameter inversion have received much less attention by neural operator research and estimating the parameter sensitivity 0u/Op has not been studied for neural operators (to the best of our knowledge). Li et al. (202T) evaluate the use of FNO to run the differential equation backward in time to recover initial conditions; although they name it Bayesian inversion, they did not recover physical parameters. In follow-up work, (Li et al. 2023) also observed the problems of using FNO for parameter inversion and proposed to add physics- informed neural network (PINN) regularization to training. While it improves over vanilla FNO, we show that our approach can outperform FNO+PINN by up 30%. Furthermore, FNO+PINN still produces inaccurate sensitivity estimates. An alternative to neural operators is learning direct inverse mapping (Vadeboncoeur et al. 2023 Li et al. 2023), but then one still needs separate techniques for estimating solution paths and sensitivities. In contrast, SC-FNO addresses all of these issues in one (convenient) framework.
Sensitivity awareness allows model adapt its predictions to different scenarios and respond to input and parameter changes, thus reducing overfitting and improving robustness. In practical applications, the ability to predict these Jacobians of the outputs with respect to physical parameters can also have significant value. In engineering design and control systems, knowing the sensitivity of performance outputs relative to design parameters helps in optimizing design and improving performance. Similarly, in financial modeling or resource management, sensitivity information allows for better risk assessment and resource allocation decisions. Gradient- based optimization or data assimilation (Barker et al., 2004), of course, is critically guided by gradients. In scenario analysis, parameters or inputs are perturbed, often beyond the observed ranges, to assess the responses. Without correctly capturing the sensitivity, the responses could be erroneous (Saltelli al.. 2019).
Gradient Computation for PDE Solvers: Gradient, or sensitivity computations, are widely used in the analysis of partial differential equations (PDEs). Traditionally, methods like finite differences can approximate gradients using existing numerical solvers and they remain relevant. Now we also have the option of using automatic differentiation (AD) offered by differentiable diffrrentibeepporaamnnig platforms such as Py Torch, TensorFlow, Julia, or JAX, which enable efficient computation of gradients of outputs with respect to inputs. While the main purpose of implementing solvers on these platforms is often to train process-based equations together ("end-to-end") with NNs so that missing relationships can be learned along with partial knowledge (Innes et al., 2019: Shen et al. 2023) Zhu et al. 2023 Schoenholz & Cubuk 2020: Tsai et al. 2021). these solvers indeed provide rapid calculation of gradients as a byproduct. Gradients can be also computed by adjoint methods, either "discretize-the then- poptimize-then-disee
(Chen et al.) 2018). Differentiable PDE solvers are rapidly increasing across numerous domains, demonstrating highly competitive performance, especially in data-scarce (Feng al. 2023) or unseen extreme (Song et al. 2024b) scenarios. Our work is compatible with either finite differences or AD. In this context, our work introduces a novel methodology that seamlessly integrates sensitivity analysis directly into the Fourier Neural Operator framework, significantly enhancing the operator's 's capability to tackle both forward and inverse problems. Notable studies such as Gradient enhanced neural networks Liu & Batill (2000) and Sobolev training Czarnecki et al. (2017) have explored the utilization of derivative information to improve neural network training, but they focused on low-dimensional ap- proximation of derivatives. Our proposed Sensitivity-Constrainee Fourier Neural Operator (SC-FNO) overcomes the problems with the estimation of parametric sensitivity, allowing better inversion while maintaining FNO-Ievel computational efficiency for solving differential equations. The SC-FNO can provide highly accurate parameter sensitivities with less training data, ensuring that solution accuracy is maintained even under input perturbations and even concept drift (physical parameters in testing exceed ranges encountered during training). Although we focus attention on FNOs here, our experiments show the approach generalizes many other neural operators (see Appendix D.I
& Ruthotto 2020; Song al.) (024a)
2

---

Published as a conference paper at ICLR 2025
2 METHODOLOGY
Training data for Neural Operators are obtained by running physics- based model simulations multiple times with different initial conditions and physical parameters. We note that for differentiable physics- based models, the sensitivities 0u/Öp can also be computed. For non-differentiable models, they can be approximated with finite differences. The training of SC-FNO uses loss function Lu, over the solution paths (as with FNO) and also adds a loss L. over the parameter sensitivities. One can optionally add PINN equation loss Lg (resulting in SC-FNO-PINN). This framework can be used with other neural operators as well (see Appendix D.I.
2.1 SC-FNO: ENIIANCING FNO WITH SENSITIVITY
Original Fourier Neural Operators (Li et al., 2021 FNOs represent a class of learning architec- tures designed to model mappings between infiniteddimensinne function spaces, thereby providing efficient solutions to complex parametric PDEs. Mathematically, FNOs operate within a defined domain D C R The function spaces, denoted as (D; Rd for inputs and (D; Rd)) for outputs, encapsulate the functional domains of the inputs and outputs respectively. The goal of FNO is to approximate the operator
G A x θ + U, which ideally maps each input function aj in A to an output function u, U. Formally, the FNO framework is expressed as:
A θ - UU, U,
where θ represents the space of parameters that the FNO optimizes to reduce the discrepancy between predicted and true outputs. The Fourier transformation, essential in the operation of FNOs, is applied transform function data into the frequency domain, facilitating the application of linear operators:
/ (()k))2 dk.
(Ff), = - (x(x) 2mi(x,.) dx
and
(F (F-'))(())
This representation utilizes the convolution theorem to model complex nonlinear behaviors efficiently: K (x) = F- (R... (F),))
where R.g. represents a learnable parameterized function within the Fourier domain, allowing FNOs to encode and decode information effectively. While FNOs offer significant computational advantages.
they necessitate extensive training data (Kovachki et al. 2021 [.i et al. 2021). Moreover, generalizing beyond the training datasets remains a challenge.
We introduce sensitivity- loss terms to Neural Operators, resulting in a novel Neural Operators (SC-NO) (Figure A.7 in Appendix A. For clarity, we demonstrate this concept primarily through its application to Fourier Neural Operators (FNO), yielding SC-FNO. However, we also show that the framework is generalizable to various types of neural operators, such as Wavelet Neural Operators (Tripura & Chakraborty 2023). Multiwavelet Neural Operators (Gupta et al. 2021), ) and DeepONets (Wang et al.] (021) (descriptions and results are in Appendix(D.I] information (such as Jacobians) and never explicitly guaranteed how the inputs should be used the model. However, our approach leverages either finite difference these sensitivities. This development improves the robustness of prediction of both u and 0u by ensuring that input variables are utilized correctly by the FNO we dictate the sensitivity of the predicted outcomes with respect to input parameters and initial conditions, while in the future boundary conditions can also be incorporated. The model expressed concisely as:
Our Contribution:
framework we call
The current neural operator training never harnessed sensitivity
differentiable solver compute
u(x,t) = FSo x, t, p),
where FSC-ENO denotes the SC-FNO mapping, un are the initial conditions, represents spatial coordinates, is time. The output u is governed by the differential equation:
Ou
fu, x,
(5)
Øt
with of defining the dynamics of the system. In this formulation, encapsulates parameters that critically modulate, influence, characterize the system, including physical constants, material
3

---

Published as a conference paper at ICLR 2025
properties, scale-dependent parameterizations. SC-FNO incorporates sensitivity loss to enforce the accuracy the predicted sensitivities (Jacobians) against those computed from the differentiable numerical solver. The SC-FNO model computes the solution across all time and space in single execution. The mathematical definition of sensitivity loss L,g is as follows:
M Σ M
aū(x;, (:P) au(x;.
T4
(6)
Öp
Öp
where xj and by represent the spatial and temporal coordinates of the sampled points which the Jacobians are evaluated. 0ú/0p is the Jacobian of the predicted outputs with respect to the input parameters, obtained through AD applied to the SC-FNO. 0u/@p @p is the true Jacobian derived from precise differentiable numerical solvers or known analytical solutions. M denotes the number of evaluation points across the domain, including points used to impose boundary and initial conditions.
2.2 THE PINN-LOST EQUATION AS AN OPTIONAL REGULARIZER
Physics-Ine Informe Neural Networks (PINNs) regularize the gradients of the neural networks using the governing differential equations at collocation points and boundary conditions Raissi et al. (2019); Kamiadakis et al. (2021) I): He et al. (2022). The core PDE constraint in PINNs is represented as:
Mu[(x t): pl 0
The PINN loss function, Lay incorporates the discrepancy from the PDE with terms for initial and boundary conditions (Lic and LBC):
N
1
LEq = Lede + a(Lic + LBC) = W[u(x; aLLic + LBC):
where a is weighting factor, and N denotes the number of collocation points in the computational domain. PINNs trained for specific conditions often cannot predict general evolutionary trajectories or guarantee accurate parameter sensitivity Jii et al. 2222)) Ren et al. (022).. Additionally, PINNs can be computationally slower than traditional numerical solvers. In this work, we evaluate the ability of PINN-type loss to improve prediction accuracy along with sensitivities. In a broadened sense, SC-
FNO can be considered novel type of PINN as also regularizes the gradient of the neural networks, but their procedures are fundamentally different. PINNs do not have access to this sensitivity, usual PDEs do not contain du/Op Op. SC-FNO prepares gradient data to supervise ôu/0p but do not require optimization during forward simulation, whereas PINNs rely collocation points for equation-base loss optimization to supervise (0//xx Ou/ōt), but not Ōu/ /ôp. SC-FNO designed to serve as an efficient forward simulator that can handle various changes in high dimensional input, which can also help with inversion tasks.
2.3 GRADIENT COMPUTATION METHODS FOR DIFFERENTIAL EQUATIONS
To prepare training and validation datasets containing true solution paths along with their sensitivity (gradients with respect to parameters), we developed and implemented two distinct approaches:
1. A differentiable numerical solver based on the torchdi f ffeq package. We extended this ODE-oriented framework to handle PDEs by reformulating them in the form di de (x). where RHS(x) encapsulates terms related spatial derivatives. This solver leverages PyTorch's is AD capability, although adjoint methods can also be used. The computation time for preparing datasets with and without Jacobian computation is presented
Table(D112
2. Finite difference methods for gradient computation with traditional solver. The method approximates gradients by solving the PDE multiple times with slightly perturbed parameter values and computing the differences between the solutions. This process offers non- intrusive means of gradient estimation for any existing numerical solvers.
We implemented efficient functions to compute the sensitivities. These solution paths and sensitivities are computed once and archived for later use, though they could also be generated on the fly desired. This represents one-time cost per equation, and since input impacts are resolved upfront, no repeated preparation needed for different parameters. We assessed the effectiveness of both gradient computation methods and compared their accuracies.
4

---

Published as a conference paper at ICLR 2025
2.4 IMPLEMENTATION DETAILS
The SC-FNO architecture processes parameters (p) alongside spatial coordinates and initial conditions through the lifting layer as function inputs. This layer reshapes and repeats parameters to match the problem's spatial-temporal dimensions, then concatenates them with other inputs before neural network processing. Notably. both FNO and SC-FNO share identical neural network architectures and inputs, differing only in their loss configurations. After computing true gradients (Qu/0p) once during dataset preparation (Section 2.3), our training procedure contains several efficiency- focused steps. Instead of computing gradients at all points, we randomly select subset of spatial-temporal points each epoch (n < N spatial points x t < T time points) where π < N and < T. The neural operator's 's predicted gradients at these points are computed using AD and compared with the pre-computed true gradients. This sampling varies between epochs to eventually cover the full solution space. This procedure eliminates the need for additional solver runs during training while maintaining effective sensitivity supervision. Each minibatch requires only one forward pass before applying AD, adding minimal computational overhead. The pre-computed gradients are stored and reused throughout training, making the approach efficient as parameter dimensionality increases.
3 EXPERIMENTS
We evaluated four different configurations of the FNO models, each defined by a unique combination of loss functions. The first configuration uses only the data loss L, (FNO), which matches predicted state variables with the solution paths: The second, L. + LEg (FNO-PINN), incorporates the equation loss; The third, Lu + LEq + L, (SC-FNO-PINN) combining both sensitivity and equation losses; and finally, Lu + L, (SC-FNO), which adds only sensitivity loss to data loss. Pseudocodes for these models can be found in the Appendix A
We evaluated the competing methods on the following
well-known differential equations.
Duffing Oscillator Equation each operate within a temporal domain of f E [0, 1), where was discretized into = 100 equal time steps. Our goal was learn the operator that maps the first M time steps of solutions u, alongside parameters p. to the solutions at the next N - M subsequent time steps, formulated as u : [0 MM - : [M : N].
ODE1: Composite Harmonic Oscillator and ODE2:
PDEI: Generalized Nonlinear Damped Wave Equation is explored within both temporal domain t E [0, 1] and a spatial domain x E [0, 1). The temporal domain was discretized into N = 30 equal time steps and the spatial domain was discretized into S₂ = 20. We aim to learn the operator that maps from the first M initial time steps of u, alongside parameters p. to the next N - M subsequent time steps, expressed as u [0, [0, M] Up - u : [0, S.] x NJ.
PDE2: Forced Burgers' Equation is analyzed across time interval & E [0, and a spatial domain I E [0, 1]. The temporal domain is segmented into N 30 equal parts, and the spa- tial domain into S₂- = 40 parts. The aim is to identify the operator that maps the first M time steps of u. in conjunction with parameters p. to the remaining N M time steps, formulated as u: S,] x [0, M] Up - S.) x [M, NN.
PDE3: Stream Vunction-Vorticityy Formulation of the Navier-Stokes Equations spans the tem- poral domain L E [0, 3) and spatial domains x. y E [0, 1]. The spatial domain and the temporal domain are [0, 1] and [0, 3). respectively. The 2D spatial domain was discretized into S₂ = S, = 64 equal spatial divisions. Here, our goal learn an operator that takes initial conditions of along with parameters and directly maps them to the solution of vorticity at the final time step & = 3s, formulated as u [0, Sz) x [0. Syl x [t 0()) Up u [0, S.] x [0, Sy] [t 3(s)].
PDE4: Allen-Cahn equation is analyzed within temporal domain t E [0, 1.0) and spatial domain E [0. 1), discretized into N 30 and 5~ = 40 parts respectively. We aim learn the operator that maps from first M time steps of and parameters p to the next N M time steps, formulated as 8: [0, S. 1 x (0, M] Up -> u [0, S_ x [M, N).
Detailed specifications of the differential equations and the ranges of their parameters are provided in Table B.6 in Appendix B The architectural details, hyperparameters and training time information comprehensively presented in Tables C.7 and C.8 in Appendix
In the following. we first demonstrate the superior performance of SC-FNO in inversion (or op- timization) tasks compared to FNO alone. Then explain such outperformance using several experiments: we show how trained FNOs alone captured the gradients poorly and thus fared worse than SC-FNOs when inputs are perturbed, which often occurs during inversion. Then, we show
5

---

Published as a conference paper at ICLR 2025
that SC-FNO requires fewer training samples reach higher quality, particularly when the input dimension higher. Finally, we show that finite difference can also functional. Experiments were with various neural operators and PDEs to ensure the generality of the conclusions.
3.1 PARAMETER INVERSION FROM SOLUTION PATHS
Parameter inversion with PDEs plays a pivotal role in system identification, calibration, and optimiza- tion, and serves primary area benefiting from the efficiency of surrogate models. Our inversion experiments utilized FNO and SC-FNO surrogate models, which were trained and rigorously evalu- ated synthetic datasets as described in Section 3.2 for PDE1-PDE3. These datasets, containing 2 x 10s training samples, were generated using the differentiable numerical solver. They feature randomly generated initial conditions and parameter values, alongside their corresponding solutions multiple time points (Table B.6 Appendix B) and Jacobians. We used 70% of the data for training, 15% for validation, and 15% for testing, and ensured that the validation and test sets contained parameter values not encountered during training. The first experiment inverted the models to infer the a parameter alone while treating others as known. We then used backpropagation to optimize the parameter by minimizing the discrepancy between the synthetic data and PDE solutions. The second task simultaneously inverted all parameters of either PDEI or PDE2. SC-FNO demonstrates notably superior performance over FNO and FNO-PINN in simple inversion tasks, achieving nearly perfect inversion while the latter two showed significantly more scattering (Figure In the single-paramett inversion, SC-FNO (R² 0.998) achieves less than 1/5 and 1/4 the relative L2 inversion errors of FNO (R² = 0.905) and FNO-PINN, respectively (Figure 1b). Multi-parameter inversion incurs larger uncertainty and greater contrasts SC-FNO has 1/6 and T/2.8 the relative 1,2 inversion errors of FNO and FNO INO-PINN, respectively (Figures Tb and 2}. Retrieving a in the multi-parameter case results in large heteroscedastic scattering with FNO (R² 0.635), whereas SC-FNO remains highly accurate (Figure For PDE2 with four parameters, SC-FNO's R² values are above 0.96, while those of FNO hover around 0.85. For PDEI with five parameters, SC-FNO maintains R² above 0.94 for all parameters, while those of FNO drop below 0.64, suggesting overfitting and breakdown of the surrogate model. Similar contrasts are found with PDE3 (Navier Stokes) (Figure [D. in Appendix D) This performance gap may widen further with increased parameter dimensions. FNO-PINN's relative L is half of that FNO (Figures | and but still 3x-5x that of SC-FNO. Additional scatter plots for parameter inversion of the sensitivity loss lead to consistent enhancements across other neural operators like WNO, MWNO, and DeepONet, with uniform conclusions reflected in Table D.11 in Appendix(] These enhancements are larger than the differences between different neural operators. By training on both solution data and their gradients, SC-FNO builds more robust internal representation of the PDE dynamics, particularly regarding the roles of inputs. The next sections explore in more depth the advantages of gradient-awant surrogate models.
available Figures D.8 and D.9 Appendix D The introduction
0.112
0.842
C.222 CLIDI
CHNG-MMNN U1M0 DUUsI
0.01- PYNG-RINN
SEL-ND
0.672
0.102
0.06
Prodicted
0.05
3.04
0.04
1.5%
0.02
+5% Error
-5% Error
-5% Emror
3.00
5.02
0.04
0.06
0.08
0.30
0.000
0.32
0.34
0.95
0.88
0.30
(b)
True
Figure I: Inversion of the parameter PDEI using FNO SC-FNO models (a) single parameter inversion, (b) simultancous multi-parameter inversion.
Figure 2: Simultanenus multi- parameter inversion accuracy for PDEs using
FNO and SC-ENO.

---

Published as a conference paper at ICLR 2025
3.2 SURROGATE MODEL QUALITY AND ROBUSTNESS TO INPUT PERTURBATION
Surrogate quality assessments indicate that the superior inversion performance of SC-FNO, even with so few parameters, can be attributed to the model's unique ability to accurately capture parameter sensitivities and its robustness to input perturbations. As discussed earlier, the surrogate models were trained on identical input and output datasets uniformly prepared for each differential equation, but only SC-FNO SC-FNO-PINN used parameter sensitivities. We first evaluated the models test datasets with parameters ranging from [a. b], the same as the training data, as detailed in Table T.6 in Appendix [. Subsequently, to assess the models' generalizability, we perturbed the parameters beyond their original ranges by applying various perturbation percentages, X, resulting in a parameter range of (lb, (I+))b]). By systematically increasing A, we evaluated the models' performance various levels of extrapolation beyond the training dataset.
Table I: Frmor metrics for PDEI (5 parameters) and PDE2 (4 parameters) with 2 10" training samples. Borth have low dimensional parameters. (6) PDEI
Value
Metric
Original raage of parameters
Pesturbed mage (A : 0..4)
PNO
SC-PNO
SC-ENO-PINN
ENO-PINN
FNO
SCANO
SC-PNO-PINN
PNO-PINN
(<(E)
Relative L?
R
0.0146
0.966
0.0175
0.983
0.01122
0.989
0,0220
0.978
0:4716
0.529
0,0852
0.912
0.0721
0.928
0.3605
0.620
R?
0.723
0.924
0.943
0.801
0.515
0
0.915
0.605
Rolative 1.5
0.2772
0.0761
0.8577
0.1992
0.4855
0
0.0851
0,3951
0.41
0,925
0.945
0,805
0,522
0,908
0.924
0.615
Relative
0.2590
0,0150
0.0552
0.1955
0,4780
0,0824
0.0767
0.3852
R!
0,763
0,930
0.956
0.817
0,530
0.914
0,930
0.622
Relative
0.2370
0.0700
0.8441
0:1854
0.4708
0.0862
0:0732
0.3781
0.772
0.955
0,530
0
0.630
Relative
0.2283
0.0696
0.0451
0.1858
0,4610
0,0801
0.0684
0.3706
OI Cha
R?
0.781
0.932
0.963
0.825
0.545
0.924
0.937
0.632
Relative 1.3
0.2190
0.0682
0.0.755
0.1752
0.4557
0.0761
0.0633
0.3687
(b) PDE2
Value
Metric
Original raage of parameters
Perturted mage (A: 014)
PNO
SC-FNO
SC-FNO-PINN
FNO-PINN
SC-FNO
SC-FNO-PINN
PNO-PINN
MC(E
Relative L.
0.0029
0.997
0.0016
0,997
0.0065
0.995
0,0073
0.995
0.0325
0.734
0,0112
0.933
0.0124
0.923
0.0287
0,902
D
0.206
0.907
0.152
0.113
Relative 1.3
0.2992
0.0135
0.0813
0.8555
1,1256
0
0.0987
0.9865
0.423
0.986
0.991
0.519
0.311
0.903
0.904
0.429
Relative LX
0.8542
0.0540
0.0523
0,7366
0.9875
0,0755
0.8244
R
0.821
0.912
0.957
0,871
0.604
0.835
0.876
0.719
03
Relative L2
0.1245
0.0756
0.8567
0,1023
0.2544
0.1023
0.0858
0.1567
R?
0.321
0,982
0.912
0.427
0.236
0.912
0.834
0.353
Relative
0.0567
0.0789
0.7899
0.0722
0.0944
0.5878
For PDEs, when the test data originate from the same parameter ranges as the training data (Table B.6 in Appendix BB) the solution paths are of high quality, but FNO leams sensitivities significantly poorer than those SC-FNO (Table Ileft half). While the metrics for u are similarly high among all models, the R² values for FNO gradients range only from 0.72 to 0.78 (relative L2: 0.28 to 0.22) for PDEI and from 0.21 1 to 0.82 (relative L 0.21 to 0.12) for PDE2. In contrast, SC-FNO and SC-FNO- PINN consistently achieve R22 values of 0.92-0.93 (relative L22 0.08-0.07) for PDE1 and 0.91-0.99 (relative L2: 0.09-0.1) for PDE2. The inclusion of PINN-type equation loss (Lex) in FNO-PINN provides only minor benefits for ap. most gradients in PDF2. These improvements are not comparable to those achieved by SC-FNO and SC-FNO-PINN, highlighting the predominant impact of I., terms. ODEs show similar pattern where SC-FNO has much better sensitivity accuracy for both parameters (Appendix Table D.14 and initial conditions (y in Table 1.h and ( in Table D.14b). In some sample test predictions (Figure Ba-d), surprisingly large and unphysical oscillations
du with R² values remaining below 0.52 (relative L >0.48) for
observed in FNO preedccddt sensitivities. These simple cases were designed to show that a surrogate model accurately predicting u may fail to capture the dependence of solution paths on the inputs, leading large inversion errors.
This pattern is reliably repeated for more challenging PDEs with a small number of parameters, e.g., for PDE3 (Navier Stokes, 2 parameters), where models lacking the sensitivity loss well predicts
7

---

Published as a conference paper at ICLR 2025
Table Error metries for PDES with 1 x 10" training Samples. Both have
Bw/Ba
Ow/03
Method
Relaiive
n
Relative L2
RRa
Relative L7
x2²
FNO
0/0312
0.997
0,7230
0.016
0.9642 0.0132
0.036
SC-FNO
0.0345
0.986
vorticity but falter in capturing the sensitivities (Table missing both fine-scale features and large- scale patterns (Figure 6). In contrast, SC-FNO accurately recreates patterns and even fine-scale details. In the case of PDE4 (Allen-Cahn), a challenging test case due to its bifurcation nature where small parameter changes can cause abrupt phase transitions in solutions, SC-FNO exhibits mildly better u solution accuracy than ENO (Table 3). More noticeably, SC-FNO generates 1/25 the Jacobian enror as FNO, even with reduced samples changes sharply. Given the diverse dynamics in the equations tested, the general incapability of FNO to capture sensitivity is evident. Note that SC-FNO does not entail excessive additional training cost (Table C.8). We also tested broad range of other neural operators, all of which exhibited markedly improved sensitivities upon introducing the sensitivity loss (Appendix Table [D9]. The difficulty FNO has in capturing parameter gradients leads to prominently degraded performance under input perturbations. though SC-FNO remains robust and reliable. The R² for the solution at 40% perturbation decreases precipitously to 0.529 (relative 1,2: 0.471) for PDEI and 0.734 (relative 2.2: 0.266) for PDE2, compared to 0.912 (relative I.2: 0.088, 1/5 that of FNO's) and 0.933 (relative L 0.067, 1/4 that of FNO's) s) for SC-FNO, respectively (Table I As the perturbation ratio increases for PDEI, a stark decline in R is observed for FNO, in contrast to the relatively stable SC-ENO (Figure 5). We argue that the large poor performance in inversion tasks, during which the search algorithm ventures into under-sampled regions of the parameter space, consistent with the arguments in Li al. (2023). At an accuracy level of 1t2 = 0.529 (relative L 0.471) (as illustrated in Figure Db by the scatter plot), the surrogate model loses its ability guide the inversion. Even without straying beyond the training parameter range, mere change in the pattern of inter matam-pa correlations could cause a departure from the training conditions (concept drift). While increasing training data can help. it becomes exponentially more difficult to adequately cover the parameter space for FNO as the input dimension increases. This challenge is highlighted by the contrasts observed between Figures Th and [b. as well as between Figures 2 and 2b. Additionally, SC-FNO-PINN only shows a slight benefit in 2 over SC-FNO in all of these tests TTblee] The minimal impact of the equation loss is both surprising and previously unexamined. This ineffectiveness likely stems from the absence of terms related to (time-integrated) 0u/Op in the differential equations, which leaves this sensitivity unconstrained even though p is in the equation. For coupled optimization tasks, it is imperative that surrogate models not only reflect the physical phenomena accurately but also adapt to new, unobserved conditions. SC-FNO thus provides effective alternative solution.
critical parameter values where solution behavior
cnttturbationiinduue error is primary reason for FNO's
15 DIFF
1.0 Pe)
1.6 te) POE2
1.5
10
0.5
1.0
3.2
1.0
3.5
2.0
00E1
(40 00E2
1.00 (9) POE3
Thi PDE2
0.75
278
0/2 0.50
0.25
0.00
1.3
2.0
2.0
1.3
True Value
PNO
FNO-PINN
SC-FNO-PINN
SC-FNO
Figure 3: Sample prediction of models for ODEs and PDEI and PDE2.
3.3 MODEL. PERFORMANCE ACROSS VARYING TRAINING DATA VOLUMES
We explored how different models respond to varying amounts of training data. Specifically, we aimed to investigate how the integration of different loss terms affects both the accuracy and the
8

---

Published as a conference paper at ICLR 2025
1.0
1.00
1.0
(e)
1.0
0.9
0.95
2
0.9
0.0
0.7
0.6
0.05
0.7
0.5
2000
1500
1000
500
0.00
2030
1:00
1000
500
20
40
100
0.8
100
Training Size
Training Size
Perturbation(55
Perturbatio(53)
MRO
HI PMN
9C FM
SO+
FND
PNO PPN
FMD PMN
SC FNO
Figure 4: Models' performance on PDET across training sample sizes, (1) 2u (b) (t)..
Figure S: Performance of models for PDEI for perturbed datasets, (A)
Standardized áe and (b) u(t). Up
W
Vorticity (w)
dw /aa
0es/Oß
True
FNO
SC-FNO
True
FNO
SC-FNO
True
FNO
SC-FNO
Figure Sample prodiction of models for PDE
generalization capabilities of the models under limited training data scenarios. The remaining portion of the dataset, not used for training, served test and measure the models' performance.
Table Performance comparison for PDE4
Table 4: Performance comparison for zoned PDE2
N=500
N=100
N=500
N=100
Menrics
Metrics
FNO
SCANNO
PNO
SC-FNO
ENO
SCAENO
PNO
SCANN
State Value Metrics
State Value Merrics
0.559
0.999
0.997
0.998
R
0:960
0.997
0.927
0.996
Relative L
0,0110
0.0151
Relative L
0,0282
0.0073
00087
0,0087
Meon Jacobian Merrics
Meon Jucobiio Metrics
-3.11
0.993
-14.012
0.927
Relative L2
0.5212
0.0207
0.5830
0,0136
Relative L
1.9627
0,1770
2.4623
0.2134
The experiment demonstrates that FNO performance can rapidly degrade as the volume of training data decreases, whereas SC-FNO maintain higher accuracy and continue function effectively as surrogate model (Figure 4h. While all models exhibit a decrease in R² scores for both state values dü (t)) and gradients äp training sizes diminish, SC-FNO and SC-FNO-PINN show notably slower rate of decline. With only 500 training samples, FNO's R² dropped to 0.8, displaying an acceleration in degradation, while SC-FNO maintained an R around 0.9, comparable to FNO's performance with 1,000 training samples. This advantage is attributed to the use of higher-order (gradient) information, which, similar to traditional numerical methods such as spline interpolation second-order finite differences, typically results in a better rate of егтоr convergence. This sensitivity to the volume of training data contributed to FNO's 's poor performance in inversion tasks (Section B.I). indicating that dense training data is necessary throughout the parameter search space to ensure accuracy. In practical applications where comprehensive datasets are unattainable, the ability to maintain high model performance with fewer examples is invaluable.
3.4 HANDLING FUNCTIONAL AND HIGH- LIGH-DIMENSIONA PARAMETER SPACE
All previous examples have small numbers of scalar parameters. To evaluate the models' effectiveness modified PDE2 (Burger's equation) with zoned parameters. We divided the spatial domain into S = 40 segments with different advection a and forcing amplitude in each zone, along with global parameters 7 and w, resulting 25 Maintaining the settings from Section 3.2, models were trained with different sample sizes. SC-FNO's advantages become immediately prominent from this higher dimensional case, even for the solution path itself. At 500 samples, SC-FNO has only 1/4 the relative L error (0.0073) as FNO (0.0282, Table 4 while requiring moderately more training time (Appendix Table C.8 With N = 100 samples, SC-FNO maintains the low error (relative L 0.0087) while FNO degrades significantly (relative L = 0.039). In fact, SC-FNO with 100 samples has less than 1/3 of the relative L error of FNO with 500 samples, and also requires less time train. SC-FNO's level of accuracy seems
with higher parameter dimensions,
total parameters.

---

Published as a conference paper at ICLR 2025
elusive for FNO, which sees L error decreasing by only 28% N increases from 100 to 500, and thus we argue SC-FNO lifts the performance ceiling. The contrast is more dramatic for Jacobian predictions (SC-FNO: relative L 0.213; FNO: relative L = 2.46), which, as discussed earlier, has strong implications for the success of inversion and generalizability. These results highlight SC-FNO's capability handle high-dimensional parameter spaces efficiently, maintaining accuracy even with limited training data. We can reasonably hypothesize that drastically increasing the number of parameters will further highlight SC-FNO's unique capability, which we reserve for future work.
3.5 DIFFERENT GRADIENT CALCULATION METHODS
We tested gradients derived from both a differentiable solver with automatic differentiation (AD) and a fourth-order finite difference solver (FD) using a traditional solver against the analytical solution for gradients of ODE1. The results of this verification presented in Table D.13 in Appendix D.3 Following this validation, we used these methods to generate solution paths and sensitivities for training and testing surrogate models. We trained SC-FNOs using 70% of the produced data, dividing the remainder into 15% each for validation and testing. The validation and testing datasets included parameter values not featured in the training data, testing the models' generalization capabilities. SC-FNOs trained with either AD- or FD-generated gradients proved effective, producing accurate predictions for both solutions and their gradients, as shown in Table [] These models both achieved R > 0.95 (relative 2 < 0.05) for solution paths and R > 0.9 (relative I <.1)) for sensitivities. While AD provides higher accuracy and efficiency (Table D.13). FD remains effective and applicable to any existing model code. This makes SC-FNO versatile while maintaining computational efficiency comparable to traditional methods.
Table S: Performance comparison SC-FNO and SC.FNO.PINC using AD and ID solvers.
ODEI
PDEI
SC-FNO (AD)
SC-FNO
SC-FNO- PLNNADD
SC-FNO- PINNGED)
SC-FNO (AD)
SC-FND (FD)
SC-FNO- PINNIAD)
SC-FNO- PINN(FD)
Value
Metric
(FD)
R²
0.591
0:968
0.994
0
0.983
0.933
0.989
0,978
Relative
0.009
0.006
0.017
0.011
Avg. 2+
R²
0.996
0.995
0.983
0.928
0,952
0,932
Relative
0.004
0.005
0.017
0,072
0.048
3.6 FURTHER DISCUSSION AND CONCLUSIONN
This work has demonstrated the powerful regularizing effect of sensitivities. The effectiveness of SC-FNO stems from the explicit governance of input influence through inteeintereteed parameter sensitivities (Qu/Op). Unlike PINNs, which supervise spatial-tempon derivatives (Qu/de /dx, iu/0t) through equation-baset loss optimization, SC-FNO directly constrains parameter sensitivities typ- ically absent in PDE formulations, using forward numerical models with differentiable solvers to prepare sensitivity data. This fundamental difference enhances model interpretability and reliability, making SC-FNO more suitable for coupled inversion or optimization tasks. Especially when input parameters have a higher dimension, SC-FNO can reduce training data demand, maintain robustness and generalizability, elevate the performance ceiling of neural operators, and even reduce training time. Our innovations include employing differentiable numerical solvers-and alternatively, finite differences- for computing (almost) exact gradients. Although differentiable programming is gaining traction in various domains (Shen et al., 2023; Song et al.. 2024a Aboelyazeed et al.. (023), few studies have leveraged computed gradients beyond backpropagation. We have used these gradients to supervise FNOs, calculating second-order gradients during training. The additional computational cost remains affordable- tereeeettinnngggni the FNO on PDEI used 722 MB while training SC-FNO required 764 MB. We argue that sensitivity regularization and time-step-free methods like FNO complement each other exceptionally well. SC-FNO's programmatic differentiability allows its seamless integra- tion with neural networks (NNs), speeding up hybrid NN-based learning and optimization. Accurate gradients are crucial for the effective training of coupled NNs, thus opening up new avenues in the field of Al-enhanced solutions to differential equations.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work was primarily supported by subaward A23-0249-S001 from the Cooperative Institute for Research to Operations in Hydrology (CIROH) through the National Oceanic and Atmospheric Administration (NOAA) Cooperative Agreement (grant no. SA22NWS43200033, The statements, findings, conclusions, and recommendations are those of the authors and do not necessarily reflect the view of NOAA. It was also partially supported by the U.S. Department of Energy. Office of Science, under award no. DE-SC0021979. Chaopeng Shen has financial interests in HydroSapient. Inc.. company that could potentially benefit from the results of this research. This interest has been reviewed by The Pennsylvania State University in accordance with its conflict of interest policy to ensure the objectivity and integrity of the research. The other authors have no competing interests to declare.
REFERENCES
Doaa Aboelyazeed. Chonggang Xu, Forrest Hoffman, Jiangtao Liu, Alex Jones, Chris Rackauckas, Kathryn Lawson, and Chaopeng Shen. A differentiable, physics-informed ecosystem modeling and learning framework for large- inverse problems: demonstration with photosynthesis doi: 10.5194/bg- 20t/bb--0--67--20000 URL https: http://hwwppppaaoooooooooorrarrr
simulations. Biogeosciences, 20(13):2671 2023.
Reza Alizadeh, Janet K Allen, and Farrokh Mistree. Managing computationaa complexity using surrogate models: a critical review. Research in Engineering Design, 31(3):775-29, 2020.
Charles Audet, J Denni, Douglas Moore, Andrew Booker, and Paul Frank. A surrogate-model- based method for constrained optimization. In 8th symposium multidisciplinary analysis and optimization, pp. 4891, 2000.
D. M. Barker, W. Huang, Y-R. Guo, A. J. Bourgeois, and Q. N. Xiao. A three-dimensional variational data assimilation system for mm5: Implementation and initial results. Monthly Weather Review, 132(4))897 - 2004. doi: 10. 10.175///52000000000411222277770
2.0.CO:2. URL https://// journal ame.soc......................o.. www.oorgvvv...... journa wtwpenaaasmwwwe/eemmmmmmmmm h/tps///wwwm/////////// 1520-0493 2004 132 0897 atvdas 2.0.co 2. xml
Boris Bonev, Thorsten Kurth, Christian Hundt, Jaideep Pathak, Maximilian Baust, Karthik Kashinath, and Anima Anandkumar. Spherical Fourier neural operators: Learning stable dynamics on the sphere. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of ICML'23, '3, pp. 2806-2823 Honolulu, Hawaïi, USA, July 2023. JMLR.org. doi: 10.5555/3618408.. 3618525.
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations. In Proceedings of the 32nd International Conference on Neural Informa- tion Processing Systems, NIPS' 18, pp. 6572-6583, Montreal, Canada, December 2018. Curran Associates Inc.
Wojeiech M Czamecki, Simon Osindero, Max Jaderberg. Grzegorz Swirsze7, and Razvan Pascanu. Sobolev training for neural networks. Advances in neural information processing systems, 30, 2017.
Dapeng Feng, Jiangtao Liu, Kathryn Lawson, and Chaopeng Shen. Differentiable, learnable, region- alized process-bar based models with multiphysical outputs can approach -fate-of-the-aat hydrologic prediction accuracy. Water Resources Research, 58(10)::2022RR032443 2022. ISSN 1944-7973. doi: 10. 00.1029/2022WR03240044
Dapeng Feng, Hylke Beck, Kathryn Lawson, and Chaopeng Shen. The suitability differentiable, physics-informed machine learning hydrologic models for ungauged regions and climate change impact assessment, Hydrology and Earth System Sciences, 27(12):23572237, June 2023. ISSN 1027-5606. doi: 1..1199/esss22-2355-200
Thomas Grady, Ravi Khan, Mathias Louboutin, Ziyi Yin, Philipp A Witte. Ravi Chandra, et al. Model-paraliel fourier neural operators as learned surrogates for large-scale parametric pdes. Computational Geosciences, 178: 10540,, 2023. doi: 10. 200.:.ccaae..2221555450000
11

---

Published as a conference paper at ICLR 2025
Gauray Gupta, Xiongye Xiao, and Paul Bogdan. Multiwavelet- based Operator Learning for Differen- tial Equations. In Advances in Neural Information Processing Systems, November 2021.
QiZhi He, Panos Stinis, and Alexandre M. Tartakovsky. Physics-constrained deep neural network method for estimating parameters in redox flow battery. Journal of Power Sources, 528:231147, April 2022. ISSN 0378-7753 doi: (o..000f:ppo.ssu::2222222233
Mike Innes, Alan Edelman, Keno Fischer, Chris Rackauckas, Elliot Saba, Viral B. Shah, and Will Tebbutt. A Differentiable Programming System to Bridge Machine Learning and Scientific Computing, July 2019.
Peishi Jiang, Zhao Yang, Jiali Wang, Chenfu Huang, Pengfei Xue, T. C. Chakraborty, Xingyuan Chen, and Yun Qian. Efficient Super-Resolution of Surf-Se Climate Modeling Using the Fourier Neural Operator. Journal of Advances in Modeling Earth Systems, 15()):2003350038000 2023. ISSN 1942-2466. doi: 10. 00.1029/0023MS0038000
Xiaowei Jin, Shengze Cai, Hui Li, and George Em Karniadakis. NSFnets (Navier-Stokes flow nets): Physics-ifformed neural networks for the incompressible Navier-Stoke equations. Journal of Computational Physics, 426:109951 February 2021. ISSN 0021-9991. 109951.
doi: 10.1016/1.jcp.202..
George Em Karniadakis, Ioamnis G. Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang. Physics-informed machine learning. Nature Reviews Physics, 3(6):422-440, June 2021. ISSN 2522-5820. doi: 1211038/44254--02-00033
Nikola B Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, et al. Neural operator: Learning maps between function spaces. ArXiv Preprint ArXiv:210808481, 2021.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier Neural Operator for Parametric Partial Differential Equations, May 2021.
Zongyi Li, Hongkai Zheng, Nikola Kovachki, David Jin, Haoxuan Chen, Burigede Liu, Kamyar Azizzadenesheli, and Anima Anandkumar. differential equations, 2023.
informed neural operator for learning partial
Zongyi Li, Hongkai Zheng, Nikola Kovachki, David Jin, Haoxuan Chen, Burigede Liu, Kamyar Azizzadenesheli, and Anima Anandkumar. Physics-Informet Neural Operator for Learning Partial Differential Equations. ACM IMS Journal of Data Science, February 2024. doi: 10.1145/3648566
Lukas Liebel and Marco Körner. arXiv:1805.06334 2018
Auxiliary tasks in multi-task learning.
arXiv preprint
Weiyu Liu and S Batill. Gradient-enhanced neural network response surface approximations. In 8th Symposium on Multidisciplinary Analysis and Optimization, pp. 4923, 2000.
Derek Onken and Lars Ruthotto. Distretize-Optimize vs. Optimize--Discetiie for Time-Series Regression and Continuous Normalizing Flows, July 2020.
Richard S Palais and Robert Andrew Palais. Differential equations, mechanics, and computation,
volume 51. American Mathematical Soc., 2009.
M. Raissi, P. Perdikaris, and G. E. Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-7077 2019. ISSN 0021-9991. doi: 10/gfzbvx.
Peter Reichert and Martin Omlin. On the usefulness of overparameterrieed ecological models. Ecolog- ical Modelling, 95(2):289-29) 2999 February 1997. ISSN 0304-3800. doi: 10.1016/50034443800996 00043-9.
12

---

Published as a conference paper at ICLR 2025
Pu Ren, Chengping Rao, Yang Liu, Jian-Xum Wang, and Hao Sun. PhyCRNet: Physics-informed cnvvouutional-recrreee network for solving spatiotemporal PDEs. Computer Methods in Applied Mechanics and Engineering, 489:114399) February 2022. ISSN 0045-7825. doi: 10.1016/j.cmm
2021. 11339..
Andrea Saltelli, Ksenia Aleksankina, William Becker, Pamela Fennell, Federico Ferretti, Niels Holst, Sushan Li, and Qiongli Wu. Why many published sensitivity analyses are false: A systematic review of sensitivity analysis practices. Environmental modelling & software, 114:29-399 39, 2019.
Samuel Schoenholz and Ekin Dogus Cubuk. JAX MD: A Framework for Differentiable Physics. In Advances in Neural Information Processing Systems, volume 33, pp. 11428-11441. Curran Associates, Inc., 2020.
Chaopeng Shen, Alison P. Appling, Pierre Gentine, Toshiyuki Bandai, Hoshin Gupta, Alexandre Tartakovsky, Marco Baity-Jesi, Fabrizio Fenicia, Daniel Kifer, Li Li, Xiaofeng Liu, Wei Ren, Yi Zheng, Ciaran Harman, Martyn Clark, Matthew Farthing, Dapeng Feng, Praveen Kumar, Doaa Aboelyazeed, Farshid Rahmani, Yalan Song. Hylke E. Beck, Tadd Bindas, Dipankar Dwivedi, Kuai Fang, Marvin Höge, Chris Rackauckas, Binayak Mohanty, Tirthankar Roy, Chonggang Xu, and Kathryn Lawson. Differentiable modelling to unify machine learning and physical models for geosciences. Nature Reviews Earth & Environment, 4(8):552-567, August 2023. ISSN 2662-138X. doi: 10. 101103//43011-003--0455544
Yalan Song. Wouter Knoben, Martyn Clark, Dapeng Feng, Kathryn Lawson, Kamlesh Sawadekar, and Chaopeng Shen. When ancient numerical demons meet physics-informed machine learn- ing: adjoint-based gradients for implicit differentiable modeling. Hydrology and Earth Sys-
lem Sciences, 28(13):301 2024a. doi:
-8-14//////s-2055---220 wwwpoo/gggrrre.... arti clee /2////////00/222
URL https
chess.copernicuse.e wwg
Yalan Song, Kamlesh Sawadekar, Jonathan M Frame, Ming Pan, Martyn Clark, Wouter JM Knoben, Andrew W Wood, Trupesh Patel, and Chaopeng Shen. Improving physics-informed differen- tiable hydrologic models for capturing unseen extreme events. ESS Open Archive, 2024b. doi: 10.22541/essoonf 1 072204444882777777577777 012334422.82707777
A. M. Tartakovsky, C. Ortiz Marrero, Paris Perdikaris, G. D. Tartakovsky, and D. Barajas-Solano. Physics-informis deep neural networks for learning parameters and constitutive relationships in subsurface flow problems. Water Resources Research, 5665):20119R0266331 2020. ISSN
1944-7973. doi: 10. 00.1099/01991022731..
Tapas Tripura and Souvik Chakraborty. Wavelet Neural Operator for solving parametric partial differential equations in computational mechanies problems. Computer Methods in Applied Mechanics and Engineering, 404:1 15783, February 2023. ISSN 0045-7825. doi: 10.1016/j.cma 2022.115783.
Wen- Ping Tsai, Dapeng Feng, Ming Pan, Hylke Beck, Kathryn Lawson, Yuan Yang, Jiangtao Liu, and Chaopeng Shen. From calibration to parameter learning: Harnessing the scaling effects of big data in geoscientific modeling. Nature Communications, 12(1):5988, October 2021. ISSN
2041-1723. doi: 411033//4144---222-6667777
Arnaud Vadeboncoeur, Ömer Deniz Akyildiz, Ieva Kazlauskaite, Mark Girolami, and Fehmi Cirak. Fully probabilistic deep models for forward and inverse problems in parametric pdes. Journal of Computational Physics, 491:112366 2023. ISSN 0921-9991. doi: http:///do..or/110000////
2023.1 12369. URL https:/ www. www.nceeiiicccccccocooocoo rect. com/ sc htt.e//ea//ttttee...i..... 90021999123004641
Sifan Wang, Hanwen Wang, and Paris Perdikaris. Learning the solution operator of parametric partial differential equations with physics-informed deeponets. Science advances, :(40))eab8 2021.
Oliver Watt-Meyer, Noah D. Brenowitz, Spencer K. Clark, Brian Henn, Anna Kwa, Jeremy McGib- bon, W. Andre Perkins, Lucas Harris, and Christopher S. Bretherton. Neural Network Parameteri- zation of Subgrid-Scal Scale Physics From Realistic Geography Global Sooum--eoolving Simulation. Journal of Advances in Modeling Earth Systems, 1622))2203MS000668, 2024. ISSN 1942-2466. doi: 10. 00100220023M0003668..
13

---

Published as a conference paper at ICLR 2025
Janni Yuval and Paul A. O'Gorman Stable machine-leaming parameterization of subgrid processes for climate modeling at a range of resolutions. Nature Communications, 11(1):3295, July 2020.
ISSN 2041-1723. doi: 14038///114670000-1112222
Shang Zhu, Bharath Ramsundar, Emil Annevelink, Hongyi Lin, Adarsh Dave, Pin-Wen Guan, Kevin Gering, and Venkatasubramanian Viswanathan. Differentiable Modeling and Optimization of Battery Electrolyte Mixtures Using Geometrie Deep Learning, November 2023,
14

---

Published as a conference paper at ICLR 2025
APPENDIX
A SC-FNO ARCHITECTURE AND SENSITIVITY INTEGRATION
Figure illustrates the schematic the SC-FNO model architecture. In this framework, this variation of the Fourier Neural Operator (FNO) integrates various inputs: parameters (P) that influence the differential equation, spatial and temporal coordinates (X [z. y, E)) and the function a(x), which may represent different initial conditions. This setup enables comprehensive learning and adaptation across different scenarios by leveraging both automatic differentiation for optimization and multiple loss components tailored to the specific dynamics and constraints of the system. Furthermore, in our framework, the FNO can be regularized by both a PINN differential equation (LE,) well the sensitivities (gradients of the solution path of differential equation with respect to parameters contributing to the diff equation). These regularization terms ensure that the model adheres to the underlying physical laws that reflect how the solution path is determined by parameter and how it changes when a parameter changes.
Trppor Liking
Fourko Layers
Projetion Layer
ALECOI
Loss Companett
States
P
Bay
Cooddi pies
Figure A.7: Sebematic SC-FNO architecture.
Pseudocode for different FNO models, each with various loss configuration settings, are presented as follows. This section aims highlight how different loss functions be integrated and optimized within these models to enhance their predictive accuracy and performance.
Algorithm FNO Training Loop with L,, Loss Over Epochs
1: Initialize FNO model
2: for epoch
max_epocch do Shuffle training data
4: for each batch P, une in training data do
5: Predict state values using FNO: ú + FNO(P)
6: Calculate loss: Lu loss Uque)
Backpropagate loss and update FNO model
8:
end for
9:
Evaluate on validation set
10: Record training and validation loss
11:
end for
15

---

Published as a conference paper at ICLR 2025
Algorithm 2 SC-FNO Training Loop with Lu and Lg Loss Over Epochs
1: Initialize FNO model
2: for epoch = 1 to max_ max__pochs do
3: Shuffle training data
4: for each batch Umme oF
Bums in training data do
Predict state values using FNO: <- FNO(P)
Calculate primary loss: Lu loss(ú, Unus)
Predict Jacobian of state values using Auto Diff (AD): J
or
Calculate sensitivity loss: L,, = loss oF
9:
Calculate total loss: Loai = C1 La + C2 Ls
10:
Backpropagate total loss and update FNO model
11:
end for
Eevaluate on validation
13:
Record training and validation loss
14:
end for
C1 and C2 are learnable coefficients
Algorithm 3 SC-FNO-PINN Training Loop with Ly, L,, and Leg Loss Over Epochs
1: Initialize FNO model
2:
for epoch to max max__epochs do
Shuffle training data
each batch P, Unee, 2P
dumne training data do
5:
Predict state values using FNO: f- FNO(P)
6:
Calculate primary loss: L., loss(ū. Mnns)
7:
Predict Jacobian of state values using Auto Diff (AD): +-
2á
DP
8:
Calculate sensitivity loss: L, = loss(j, 0wnx)
9:
Calculate equation loss: Leg residual(ú)
10:
Caleulate total loss: Loout C1 L.. + C2 La + Leg
Backpropagate total
update FNO model
12:
end for
13:
Evaluate on validation set
14:
Record training and validation loss
15:
end for
C1, Cz, and es are learnable coefficients Liebel & Körner 2018
16

---

Published as a conference paper at ICLR 2025
B DIFFERENTIAL EQUATION DETAILS
In this section, the differential equations investigated in the work are detailed. This includes their mathematical formulations, initial conditions, parameter setups.
ODE1: Composite Harmonic Oscillator We chose this ODE because it has an analytical solution, allowing us validate our differential equation solvers and sensitivity computations. The ODE is defined as:
du
= sin(art) + 3 cos(3Tt)
(7)
dt
with the initial condition u(0) sin()>) This oscillator's behavior is modulated by the parameters o, 8, and 7, affecting the frequency and amplitude of oscillations within the temporal domain / E [o, 1]. The analytical solution for u(t) is:
1 u(t cos(a + sin(Brt) + sin(yT)) +
(8)
The sensitivities of u with respect to each parameter are:
Tu
I ssnn(att)
(9)
8a
Ou cos(Brtt)
(10)
03
du
cos(TT))
(11)
These analytical solutions provide benchmark against which we can compare the accuracy of our numerical solvers and sensitivity computations.
ODE2: Duffing Oscillator Equation
I + 8á + al + 33t = cos(w()),
(12)
with initial conditions z²0) = e, (0) = Ç. This equation describes a non-linear oscillator where damping 8, stiffness a, non-linear stiffness 8, driving amplitude y, and frequency w play crucial roles.
PDE1: Generalized Nonlinear Damped Wave Equation
Su 012
2 02
+ a du
=
dx2
Ø1
+ +Bu + sin(ww))
(13)
with initial conditions u( 0) no. गt
Ou (x, 0) ug- This PDE extends spatial domain E 1] and temporal domain t E [0, exploring wave propagation influenced by damping a,
stiffness 8, and external forcing y and
PDE2: Forced Burgers' Equation
Ou Tu
Pu = 0x
+ ar
+ sinn(w)),
(14)
at
Or
This equation is within spatial domain [0, 1.0] and temporal domain E [0, models fluid dynamics phenomena such as velocity u(æ; l), incorporating effects of advection a, viscosity and external periodic forcing characterized by amplitude ö and frequency The initial state of the system is defined as follows:
u(z, 0) = uo(x) = (e-
(r-rg)² 3nd
+ sin (0.55TT)
(15)
where the Gaussian pulse is centered at .g = 0.5 with width o = 0.3, combined with a sinusoidal component. The model employs periodic boundary conditions, ensuring that u(0, t) m(1.0,
17

---

Published as a conference paper at ICLR 2025
throughout the simulation, facilitating the study of continuous and cyclie phenomena in finite spatial interval.
PDE3: Stream Function-Voriie
Vorticity Formulation of the Navier-Stokes Equations
Ow
dw
dw
1
@²w
+
vu
2x
+
(16)
dt
8x
dy
Re
dx²
dy²
Ww
(17)
2x
Oy²
with initial condition wxx, y, 0) = f(x y: a, (3 where:
(x a,, - sin(ar)) cos((0)) (By) cos(ag) sin(Bx) + sin By) cos(ay Bxx,,
(18)
covering the spatial domain y E [0, 1] and temporal domain tin 3). The Reynolds Number Re for the Navier-Stokes equations is set to 1000 to simulate realistic fluid dynamics. This equation captures the dynamics of fluid flow, with initial vorticity distribution determined by parameters and B.
PDE4: Allen-Cahn equation
0u
8a. u 0t Ox
=
+ au Buu
(19)
with initial condition uæz, ctanh(wx) and periodic boundary conditions. This PDE, known for cxhibiting rich bifurcation behavior, explores phase transition phenomena influenced by diffusion coefficient c, linear term o, cubic term 8, and initial condition parameters and The equation's solutions can undergo sharp qualitative changes with small parameter variations, making it an excellent test case for sensitivity analysis.
The parameter ranges used in our simulations are detailed in Table B.6 Parameters for these simu-
Table B.6: Parameter values for different ODE and PDE cases.
Case
8
C
M
ODE
[l. 3]
[0.
ODE 2
[0:02, 0.06]
[0.01, 0.03)
[20, 60)
[0.5. 1.5)
[0.2, 0.6]
[0.0. 0.2)
[0.0. 0.2)
10
0.25)
[0.0, 0.11
0.25]
[0.0, 0.25]
PDE
[0.1, 1.0]
[0.025, 0.25]
[0.1,0.5]
[0.01, 0.1]
[x]]
[s.5a]
PDE
[0.1,099]
[0.01.100]
[0.01.10]
[5.0. 10.0]
[0.01.100]
lations were randomly generated using uniform distribution. The uniform distribution is denoted by M(a b), where a is the lower bound and is the upper bound of the distribution. This means that any value within the range (a, b] has equal probability of being selected. The uniform distribution was chosen to ensure a balanced representation of parameter values across the entire specified range, without favoring any particular subset of values. This approach allows for comprehensive explo- ration of the parameter space, providing robust test of our models across a wide range of potential input conditions.
18

---

Published as a conference paper at ICLR 2025
C FNOS HYPERPARAMETERS
Table C.7 presents the hyperparameters used for training the FNO models for each case study. In the network architecture, "Mode" refers to the Fourier modes used in the neural network's layers for each dimension (t, x, and y). while "Width" denotes the number of channels features in the hidden layers of the neural network. The learning rate and number of epochs used for training are also provided for each case. Additionally, Table C.8 compares training times per epoch for different model configurations and batch sizes.
Table C.7: Hyperparameters for FNOs
Mode for I
Mode for x
Mode
Number Fourier Layers
Learning Rate
Number of Epochs
Number of Learnalle Parameters
Case
Width
for
ODE
20
0.001
500
17921
ODE
20
0,001
500
17921
PDE
20
0.001
500
107897
PDE
20
0.001
500
107897
20
PDE 4
20
0.001
500
107897
Table C.8: Comparison model configurations training time
Number physical parameters (p)
Number training samples
Average training time epach (6)
Case
Batch size
FNO
SC-FNO
FNO-PINN
SC-FNO-PINN
ODEI
16
2000
1.10
1.94
1.53
2.46
ODE2
16
2000
1.58
2.13
1.76
2.86
PDEI
2000
35.24
53,32
$2.13
PDE2
4
2000
32.66
44,92
39.11
73.06
PDE2 (Zoned)
100
5.37
PDE2 (Zoned)
82
500
8.09
11.23
PDE3
1000
47.16
109.43
PDE4
100
11.54
19.12
19

---

Published as a conference paper at ICLR 2025
D ADDITIONAL RESULTS
D.1 PERFORMANCE COMPARISON OF NEURAL OPERATORS
This appendix section presents comprehensive comparison of the performance of various neural operators, both in their original form and with our proposed sensitivitycconstrinedd framework. We evaluate these operators on four different systems, PDE1 and PDE2. The following tables provide quantitative results for Fourier Neural Operators (FNO), Wavelet Neural Operators (WNO), Multiwavelet Neural Operators (MWNO), and DeepONets, along with their sensitivity counterparts.
Table D.9: nt for PDEI with 2 x training samples.
Value
FNO
SC-FNO
WNO
SC-WNO
MWNO
SC-MWNO
DeepONet
SC-DeepONet
a(()
0.986
0.983
0
0.989
0.978
0,952
0.974
0.954
g5
0.723
0.924
0.661
0.919
0.596
0.939
0.117
0.508
0.925
0.763
0,930
0.921
0.914
0.863
0.915
0.778
0.827
0.572
2b
Table D.10: for PDE2 with 10 training samples.
SC-FNO
SC-WNO
DeepONet
SC-DespONet
t(())
0.997
0.997
0.981
0.979
0.989
0.971
0,990
0.985
Zhu
0.986
0.562
0.905
0,511
0.873
0.912
0.894
0.445
0,958
0,321
0,982
0.425
0,542
0.214
0,952
0,363
0.806
D.2 ADDITIONAL RESULTS FOR THE INVERSION EXPERIMENTS
Figures D.8 D.9 and D.10 showcase parameter inversion results for parameters for PDE1, PDE2, and PDE3, respectively. SC-FNO works significantly better than either FNO or FNO-PINN. Table D.IT shows the comparison between various neural operators and their sensitivity versions in simultaneous parameter inversion. Note the uniform pattern that the sensitivit--constrineed versions have much higher inversion accuracy. Furthermore, the differences between different neural operators are smaller than the difference between the versions with and without the sensitivity constraint.
(SC-) constrained
Table D.I:: Multi-parameter inversion accuracy for PDEs and 2 with and without gradicat supervision.
(a) PDE1
Fourier Neural Operators
Other Neural Operators
FNO
SC-ENO
Without gradient supervision.
With gradient supervision.
Parameter
R²
Relative L
R²
Relative L
WNO
MWNO
DeepONet
SC-WNO
SC-MWNO
SC-DeepONet
0.657
0.212
0.987
0.035
0.636
0.614
0.538
0.984
0.981
0.977
0.642
0.222
0.986
0.036
0.621
0.598
0.519
0.984
0.980
0.977
0.974
0.668
0.969
0.804
0,165
0.985
0.037
0.792
0,780
0.736
0.982
0.978
0,975
W
0.750
0.186
0.916
0.075
0.735
0.719
0.663
0.901
0.879
0.859
(b) PDE2
Fourier Neural Operators
Other Neural Operatons
FNO
SC-ENO
Without Grad. Sup.
With Grad. Sup.
Parameter
R²
Relative 1
R²
Relative
WNO
MWNO
DeepONet
SC-WNO
SC-MWNO
SC-DeepONet
0.954
0.078
0.982
0.045
0.951
0.949
0.938
0.979
0.974
0.970
?
0.925
0.082
0.967
0.022
0.921
0.916
0.899
0.961
0.953
0.945
0.895
0.118
0.986
0.042
0.889
0,888
0.859
0.983
0.973
0.976
20

---

Published as a conference paper at ICLR 2025
0.25
9.29
190-1NV
(1
0722
0.20-
Poddede
0.30
6.8
0.23
5% Errer
+5% Erorr
0.15
105
(a) parameter
(b)
parameter
9.25
2004
0.25
-TRO
RT 0.20
Preditted
0.55
Wed Wtdd
9.20
0.25
2.15
0.35
Tuu
Thus
(c) 7 parameter
(d) parameter
Figure Individual parameter results
parameter recovery.
comparing FNO and SC-FNO performance. has beon presented in Figure Tb)
AT 6454
0.40
-SCHD
2952
-ND-F
12 0.280
0000
Prodictod
Freduced
0.5
0.4
0.5
55% Erto
0.7
-5% Emror
0.1
02
0.4
0.6
3.8
1.0
(a) a parameter
(b) parameter
-1NO
0.45
FNC-FRN
-5C-PR
0.00
P
0.35
Wedd
0.00
0.00
0.20
-5% Error
0.3
2.03
Frue
Thue
(c) 5 parameter
(d) parameter
Figure D.9; Individual parameter results simultaneous multi mullii-prrmmterr recovery.
comparing FNO and SC-FNO performance.
D.3 VERIFICATION DIFFERENTIABLE AND FINITE DIFFERENCE SOLVERS
Table D.13 presents comparison of R² values for both the solution path u(t) and the gradients with respect to different parameters obtained from both differentiable solver (AD) and 4th order finite difference solver (FD). The table includes runtime measurements for generating x 10 samples using both solvers, providing insight into their computational efficiency. These measurements were obtained using a machine equipped with a V100 GPU and four Intel Xeon processors, offering standardized comparison of computational cost (The Wall Clock Times) between the two solvers.
21

---

Published as a conference paper at ICLR 2025
R
45% Emrr
Thom
Figure D.10: the paramelerr using und SC-FNO
Computation for preparing datasets using with and without Jacobian
Table D:13: Envor metrics and runtime comparison for training data ODE!
preparation AD
Number of input parameter (P)
solver
Computation (s)
Value
R²
Runtime
R²
Runtime
With jacobian
Without jucobian
0,9872
0.9981
PDEI
5
1,852
0.932
0.9712
1907.32
0.9906
252.54
PDE2
1.387
0.796
0.9885
PDE3
2
6.205
2.762
0.9618
0.9890
D.4 SURROGATE MODEL QUALITY FOR THE ORDINARY DIFFERENTIAL EQUATIONS (ODES).
Table D.14: Test R²? values and Relative L? for the solution paths u and sensitivities "p
of the surrogate models for ODE1 and ODE2 with 2 10a training samples. The and parameters the range. ODFs simpler capture suroogate models than PDFs.
(a) ODEI
Value
SC-FNO
SC-FNO-PINN
FNO-PINN
R
0.996
0.991
0.994
0.991
Relative
0.009
0.006
0.327
0.994
0.992
0.318
Relative 1.2
0.673
0.006
0.008
0.682
0u 0.415
0.462
0,
Relative L
0.585
0.005
0.005
0.538
8u 0.028
0.998
0,998
0.131
Relative L*
0.972
0.002
0.002
0.869
ODE2
Value
Metric
FNO
SC-FNO
SC.FNO-PINN
FNO-PINN
0.995
0.997
0.997
Relative L*
0.002
0.005
0.003
0.003
Ou D0
0.998
0.152
Relative 1.2
0.838
0.003
0.002
0.848
du
0.956
0.997
0.998
0.951
Relative
0.044
0.003
0.002
0.049
Ra
0.135
0.996
0.997
0.147
Relative
0.865
0.004
0.003
0.853
22


Reviews:
[{"review_id": "veiqBpDh8L", "reviewer": "Reviewer_a6GH", "summary": "In this paper, the author proposes a sensitivity loss as a regularizer for Fourier neural operator, called SC-FNO. Specifially, the sensitivity is measured by the derivative (jacobian) du/dp, where p is the physical parameter. The jacobian du/dp is computed via auto differentiation. Experiments are conducted on 2 ODEs and 2 PDEs. It shows that SC-FNO outperforms previous FNO and Pinn-FNO.", "questions": "- What are the runtime and memory consumption, and which is the primary factor to consider?\n- Can the model handle functional input such as $ \\frac{du}{du_0} $? In this case, the Jacobian could be huge.\n- Can the model account for parameters like the Reynolds number or viscosity as $P$?\n- Can the model work on chaotic system such as the lorenz system, where the underlying parameter is very sensitive?\n\n---\nupdates: the rebuttal addressed most of my questions. I am happy to raise my score to 8.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- This paper examines a notable case involving the system parameter $P$, where $ \\frac{du}{dp} $ is computed as a sensitivity loss to regularize the model.\n- The results indicate that SC-FNO achieves greater accuracy in $ \\frac{du}{dp} $ and demonstrates enhanced robustness against perturbations in $p$.\n- This approach aids inverse modeling in identifying the system parameters more effectively.", "weaknesses": "### Writing\nThe overall writing quality could be enhanced, with several important details suggested to be moved from the appendix into the main text.\n- The physical parameter $P$ is central to the study and should be introduced more thoroughly. Currently, it is presented in a general form, but it’s important to clarify whether $P$ is a scalar in $\\mathbb{R}$, a vector in $\\mathbb{R}^d$, or a function in $L(\\mathbb{R})$.\n- Starting with a concrete example in the introduction would improve clarity and reader engagement.\n- On Page 5, while introducing the ODEs and PDEs, the nature of the parameter $P$ remains ambiguous. Explicitly writing out the equation would help clarify its role.\n\n### Methods\n- It is unclear how $P$ is incorporated into the FNO model. If $P$ is a scalar, is it treated as a constant function and added as an additional channel?\n- Computing the derivative requires additional runtime and memory, which should be discussed in detail.\n\n### Experiments\n- SC-FNO appears to achieve higher accuracy primarily on $\\frac{du}{dp}$ but not significantly on the solution $u$ itself.\n- Runtime and memory usage should be addressed, especially in comparison to alternative approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VrGv5H5JEM", "reviewer": "Reviewer_DfuF", "summary": "In this paper, the authors propose a regularization technique that examines the gradient of the solution function with respect to parameters in differential equations. To the best of my knowledge, this approach to sensitivity regularizer, aimed at enhancing the generalization ability of neural operators, is unique. The paper is well structured, with clear motivation and objectives.", "questions": "1. Could the authors elaborate on the approach to ensure fair comparisons between the standard Fourier neural operator and the proposed method in Tables 1, 2, and 3? Including details on training times, model complexity, and the number of learnable parameters would enhance the rigor of the paper. \n\n2. Higher-order derivatives might be utilized in constructing the loss function (as in Equation (6)). It would be beneficial if the authors discussed the potential advantages or limitations of incorporating these higher-order terms. \n\n3. Although Fourier neural operators are generally applicable to rectangular domains, other neural operators are available even for non-rectangular domains. Given that the authors provided results using DeepONet in D.6, could they consider including experiments across various domain shapes or grid resolutions to further validate the robustness of the proposed approach?\n \n4. Has the proposed method been tested in scenarios involving bifurcation, where small parameter changes induce rapid solution changes (e.g. the exponential model y’=py with p near zero)? \n\n5. The computational cost associated with this model may increase significantly as the number of parameters grows, or if parameters are represented as functions rather than scalars. While results are presented for a four-parameter case in Table 1, could authors consider examining more complex cases – such as those with over ten parameters, function-based parameters, or parameters with probability density function across a range?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The study addresses the limitations observed when Fourier neural operators, combined with a physics-informed loss function, may yield suboptimal performance. The proposed regularization method appears to mitigate these challenges effectively.", "weaknesses": "The proposed approach requires additional information and computational resources, as it leverages direct solution-based information, such as a precomputed Jacobian in the differentiable numerical solver (Eq, (6)). \n\nAnother concern is the regularizer’s potential weakness to noise in practical settings. Derivative-based techniques are typically vulnerable to data noise, which could affect prediction accuracy, especially in applications beyond simulation, such as parameter inversion. In real-world applications, obtaining accurate derivative information can be challenging. While the finite difference methods employed in this paper are an option, they may perform poorly under noisy conditions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xdAN611wrY", "reviewer": "Reviewer_KLdr", "summary": "This paper introduces a sensitivity loss regularizer for neural operators, specifically applying it to the Fourier Neural Operator (FNO) framework, resulting in Sensitivity-Constrained Fourier Neural Operators (SC-FNO). SC-FNO improves accuracy in solution paths, inverse problem-solving, and sensitivity calculations, even under sparse data and concept drift. It outperforms both standard FNO and FNO with PINN regularization in parameter inversion tasks, achieving high accuracy without large computational or memory costs. These enhancements extend the applicability and reliability of neural operators in modeling complex physical systems.", "questions": "In above part.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper's proposed Sensitivity-Constrained approach is applicable to various neural operators, offering high research value.\n2. The paper is clearly written, with no redundant mathematical derivations, and offers excellent readability.\n3. In modeling PDEs, this approach not only enables accurate approximation of the target variable but also effectively models its derivatives.", "weaknesses": "1. There is no enough discussion on the memory overhead introduced by AD (Automatic Differentiation) and FD (Finite Differences).\n2. The experiments are limited. It is not sure whether this method remains effective in complicated and high-resolution PDE scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x3aBeTMdRL", "reviewer": "Reviewer_WFXg", "summary": "This paper proposes a sensitivity-constrained loss in the original FNO model, making it suitable for ODE/PDE inversion problems. Definitely this loss can make the prediction more stable with respect to the physical system parameters. This loss works with a wide range of neural operators and other supervising sensitivity tasks. The paper is well-written. The method is effective compared to the original FNO and FNO-PINN. The experiments on five ODE/PDE problems are impressive.", "questions": "See in the weaknesses 2-4.\n\nWhy use R^2 as metric, why not the relatively L^2 norm as in the original FNO paper？The metric should be comparable to existing works.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1.The paper focus on the sensitivity prediction to physical ODE/PDE systems, which is important. \n\n2.The paper’s contribution is clear, and the paper is easy to follow.\n\n3.The experiments are diverse.\n\n4.The paper’s improvements help enhancing the reliability and applicability of neural operators in complex physical systems modeling and engineering analysis.", "weaknesses": "1.The idea of using gradient loss to guarantee sensitivity is not new, such as the gradient PINN and others. \n\n2.The sensitivity loss is supervised, which means additional labeled data {\\partial u}/{\\partial p} are needed compared to original FNO. On the other hand , if we have additional labeled data, it is directly to add the supervised loss in the total loss, which is SC-FNO exactly, so I think the novelty is rather limited. Although the {\\partial u}/{\\partial p}  can also be approximately computed from the original data, the numerical error influence is not analyzed. On the contrary, the PINN loss is unsupervised.\n\n3.The SC-FNO loss is hybrid, the training process should be reflected in the main text or appendix. SC-FNO should be hard to train compared to FNO, but the author didn’t mention the difference.\n\n4.Since the PINN-FNO also add the PINN loss as the regularizer to guarantee the prediction of physical parameters, I wonder why it behaves bad to SC-FNO. Maybe the training difficulty is different. The author use a learnable coefficients combination of the hybrid loss Liebel & Körner (2018) in Algorithm 3, but FNO-PINN may be trained well for adaptive hybrid loss as in Wang S, Wang H, Perdikaris P. Improved architectures and training algorithms for deep operator networks[J]. Journal of Scientific Computing, 2022, 92(2): 35.\n\n5. SC-FNO architecture,  as well as the Algorithm, should be exhibited in the main text,  not the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)