HyperFNO: Improving the Generalization Behavior of
Fourier Neural Operators

Francesco Alesiani∗
NEC Labs Europe

Makoto Takamoto,
NEC Labs Europe

Mathias Niepert
University of Stuttgart

Abstract
Physic-informed machine learning aims to build surrogate models for real-world
physical systems governed by partial differentiable equations (PDEs). One of themore popular recently proposed approaches is the Fourier Neural Operator (FNO),
which learns Green’s function operator for PDEs based only on observational data.
These operators are able to model PDEs for a variety of initial conditions andshow the ability of multi-scale prediction. However, as we will show, this model
class is not able to generalize to changes in the parameters of the PDEs, such as
the viscosity coefficient or forcing term. We propose H YPER FNO, an approachcombining FNOs with hypernetworks so as to improve the models’ extrapolation
behavior to a wider range of PDE parameters using a single model. HyperFNO
learns to generate the parameters of functions operating in both the original and thefrequency domain. The proposed architecture is evaluated using various simulation
problems.

1

Introduction

Physical systems governed by differential equations are pervasive in the sciences and engineering.
Indeed, partial differential equations (PDEs) are the most popular modeling method in numerous
disciplines such as the material and climate sciences. On the other hand, the recent success ofdata-driven modeling with machine learning (ML) has started a trend to use ML methods, which areboth differentiable and efficient at prediction time, for approximating simulations of physical systems
and solving inverse problems. For instance, neural networks are now commonly used to approximate
the solution of a partial differential equation or its Green’s function (Avrutskiy, 2020; Karniadakis
et al., 2021; Li et al., 2021; Raissi et al., 2019; Chen et al., 2018; Raissi, 2018; Raissi et al., 2018;
Pfaff et al., 2020; Wang et al., 2020; Khoo et al., 2021).
Neural Operators (NOs) Li et al. (2020) and in particular, Fourier Neural Operators (FNOs) (Guibas
et al., 2022; Li et al., 2021) have shown impressive results as surrogate models and have been
applied in challenging scenarios such as weather forecasting (Pathak et al., 2022). Neural operators,
however, work under the assumption that the governing PDE is fixed, that is, its parameters are
static while the initial condition is what changes. If this assumption is not met, the accuracy ofthese approaches deteriorates (Mischaikow & Mrozek, 1995), also visible in Figure 3. Thus, when
we are in a situation where we want to generalize to multiple physical model parametrizations, wewould need to either (1) re-train the NO for each of the parameter configurations or (2) include
the parameter values as input to the neural operator (Arthurs & King, 2021). Training over a large
number of possible parametrizations is computationally demanding. On the other hand, increasing
the number of parameters of the network increases the inference time which limits the efficiency
advantage surrogate models have over numerical solvers.
∗

E-mail:Francesco.Alesiani@neclab.eu

Machine Learning and the Physical Sciences workshop, NeurIPS 2022.

Figure 1: A H YPER FNO that implements Mθλ consists of a base neural operator and a hypernetworkthat learns to generate parameters of subnetworks of the base model. H YPER FNO generates, for eachlayer of the base network, and depending on the configuration, the frequency and/or spatial weightmatrices.
With this work, we formulate a meta-learning problem where each task corresponds to a parametervalue of a given PDE. At inference time, we use the learned meta-model to adapt to the current task,that is, the given parameters of the PDE. To this end, we propose H YPER FNO (section 2), a method toadapt the Neural Operator over a wide range of parameter configurations, which uses Hypernetworks(Ha et al., 2016). We also propose variations of the H YPER FNO framework (section 2) withlow memory requirements; while we show experimentally (section 3) why modeling the change
in parameters of a PDE in the frequency domain is beneficial; Finally, since the H YPER FNO isdifferentiable with respect to the PDE parameters, we show how to solve inverse problems throughstochastic gradient descent using H YPER FNO in section 2.

2

H YPER FNO: The Hyper Fourier Neural Operator

Problem definition A solution to a PDE is vector-valued function u : T × X × Λ → R on
some temporal domain T , spatial domain X , and parameterized over Λ. For example in the heatdiffusion equation, u could represent the temperature at a location x ∈ X at a time t ∈ T , where
the conductivity field is defined by λ : X → R+ . The forward operator maps the solution at one
instant of time to a future time step F : u(t, x, λ) → u(t + 1, x, λ), which is used to compute thesolution of the PDE at any time, given the initial conditions. We consider the general problem oflearning a class of operators, which includes the forward operator F . M λ : A × Λ → U between two
infinite dimensional spaces of functions A : Rd → Rp and U : Rd → Rq , on the space of parametersΛ, from a finite collection of observed data {λj , aj , uj }N
i=1 , λj ∈ Λ, aj ∈ A, uj ∈ U, composed ofparameter-input-output triples, where aj ∼ µ and λj ∼ ρ are drawn from two known probability
distributions µ over A and ρ over Λ; using a family of operators Mθλ : A × Λ × Θ → U parameterizedby θ ∈ Θ. We then aim to minimize the expected loss minθ Ea∼µ,λ∼ρ L(Mθλ (a), M λ (a)), withL(u′ , u) measuring the difference between the true and predicted output.
Fourier Neural Operators Fourier Neural Operators (FNO) (Guibas et al., 2022; Li et al., 2021) areamong the most successful Neural Operators since they model spatial and frequency domains. FNO
implements a discrete version of Mθ : A × Θ → U, and is composed of initial and final projection
networks parameterized by P , Q, and Q′ , and of a sequence of Fourier layers, parameterized by atensor W and is implemented using a 1-d convolutional network for the spatial component, whileparameterized by a tensor R for the frequency component. The FNO can be expressed asz l+1 = σ(W l z l + F −1 (Rl F (z l ))), z 0 = P x, u = Q′ σ(Qz L−1 ),(1)with z l the latent tensor at l-layer of proper dimension depending on the domain of the PDE, whilex, u are the input and output of the operator. The output projection is implemented using two
consecutive fully connected (FC) layers, the input projection with one FC, and F is Fast Fourier
Transform (FFT).
2

H YPER FNO: Hyper Networks for FNO Hypernetworks Ha et al. (2016) have been proposed asa meta-learning method composed of two networks: the main network (f (ψ, x)), with parameters
ψ, and the hypernetwork2 (h(θ, λ)),with parameters θ. The H YPER FNO network consists of a
hypernetwork that generates a subset of the parameters of the FNO network. If we write a FNO as
the function f (ψ, x), then the H YPER FNO can be written as ψλ = h(θ, λ), û = f (ψλ , x), whereû is the predicted solution given the PDE of parameters λ and initial condition x, and ψ are theparameters of the FNO network generated by the hypernetwork. We train the hypernetwork endto-end by minimizing the following expectation of a loss function minθ Eλ∼p(λ) Ltrλ (θ, λ), whereLλ (θ, λ) = E(x,u)∼Dλtr ||u − f (ψλ , x)||2 , and ψλ = h(θ, λ). p(λ) is some given distribution overtr
the PDE’s parameters λ, while Dλis the training dataset, conditioned by the PDE’s parameter. Fig.1
shows an implementation of the H YPER FNO.
H YPER FNO hypernetwork architecture Hypernetworks are used to generate the parameters of amain neural network conditioned on the current task. Typically, the hypernetwork is a large neuralnetwork that generates the parameters of a smaller network. This approach increases the efficiencyof the model at inference time. An alternative approach aims at using a hypernetwork that only
marginally increases the size of the main network but still allows it to easily adapt to new tasks. We
focus on the second approach and propose a special class of layers of the hypernetwork that can beused to assemble the main network. In the H YPER FNO, each layer of the FNO and its initial andfinal projection operators (Equation 1)are generated by the corresponding hypernetworksl
W l = hw (θw, λ), Rl = hr (θrl , λ), P = hp (θp , λ), Q′ = hq′ (θq′ , λ), Q = hq (θq , λ) (2)In the following sections, we considered different implementations of the hypernetworks h∗ (θ, λ) of
Equation 2.
Addition and Taylor Model While we could use a generic multi-layer perceptron (MLP), we
present more memory-efficient architectures. From Equation 1, we scale the row and columns of theW l tensor or only the rows of the Rl matrix. The intuition is to allow to learn a fixed direction term
R1l and then be able to scale independently in each dimension based on the parameter λk . We callthis version the Addition version.
lhr (θrl , λ) = R0l + (V0l diag(λ)V1lT ) ⊙row,col R1l , hw (θw, λ) = W0l + (U0l λ) ⊙row W1l ,(3)where ⊙row and ⊙row,col represent the Hadamard product only along the specified dimensions (i.e.
along the rows and along both rows and columns). We can modify the previous model to reduce thenumber of parameterslhr (θrl , λ) = R0l ⊙ (I + V0l diag(λ)V1lT ), hw (θw, λ) = W0l ⊙ (I + U0l λ),(4)where ⊙ refers to the Hadamard operator along the relevant dimensions. We call this version Taylor.
The parameters λ can be encoded using additional neural networks of minimal size, λ′ = g(θg , λ),with θg additional H YPER FNO parameters.
Inverse Problem over PDE parameters Since H YPER FNO is able to adapt to the change inthe parameters λ of the PDE, H YPER FNO is used for the inverse problem, where, after training
H YPER FNO network over multiple PDE’s parameters, we can recover the parameters of new samples
by solving λ∗ = minλ E(x,u)∼Dfs ||u − f (ψλ , x)||2 , ψλ = h(θ, λ), where λ∗ is the optimal
parameter given the trained hypernetwork h(θ, λ) and the new dataset Dfs . The previous problem
can be solved using stochastic gradient descent λt+1 = λt − µ∇λ E(x,u)∼Dfs ||u − f (ψλ , x)||2

3

Experiments

In order to evaluate the performance of H YPER FNO, we considered the following problems (Liet al., 2021; Takamoto et al., 2022): 1) one-dimensional Burgers’ equation, 2) one-dimensional
reaction-diffusion equation, 3) two-dimensional Decaying Flow problem and 4) two-dimensional
Compressible Fluid Dynamic (Compressible Navier Stokes - CNS). Contrary to (Li et al., 2021), we
prepare datasets allowing various parameter values for instance for the diffusion coefficient.
2

Hypernetwork term refers to the combination of the two networks or only to the generative network, themeaning is given by the context.

3

U-NetFNO

HyperFNO

2

1.5

MSE (log)

1

Compressible Fluid Dynamic

Reaction-diffusion

2.0

MSE

MSE (log)

20

2D Darcy Flow

1e 5

8
6
4
2
0

MSE (log)

Burgers

3

1.0

1

0.5

U-Net

FNO

0.0

HyperFNO

U-Net

FNO

0

HyperFNO

U-Net

FNO

HyperFNO

Figure 2: MSE at test time for U-Net, FNO, and H YPER FNO models; tested on (left to right) Burgers,
2D Darcy Flow, Reaction-Diffusion, and 2d+time CNS datasets; we use log scale for visualization.

u0(x)u(1, x)u(1, x)1.0

u0(x)u(1, x)u(1, x)

1.0

1.0

0.8

0.8

0.6

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0.0

0.0

0.0

0.0

0.8

0.0

0.2

0.4

0.6

(a) FNO

0.8

1.0

u0(x)
u(1, x)u(1, x)
0.0

0.2

0.4

0.6

0.8

1.0

0.0

(b) FNO

0.2

1.0

0.8

0.4

0.6

0.8

(c) H YPER FNO

1.0

u0(x)
u(1, x)u(1, x)
0.0

0.2

0.4

0.6

0.8

1.0

(d) H YPER FNO

Figure 3: Visual comparison of FNO and H YPER FNO for different initial conditions and PDEparameters, test on the left (a,c), while training on the right (b,d). The x-axis is the spatial domain,
while the y-axis is the solution, either predicted û(t = 1, x)or true u(t = 1, x), while we also report
the initial condition u0 (x) in a different color.
Zero-Shot learning In Zero-Shot learning, at training time, we have access to solutions of a PDE
over different initial conditions and for a set of PDE parameters. At inference time, we use the PDE
parameters of the new environment as input to the H YPER FNO to generate the parameters of themain FNO network. We then use this network to predict the solutions for new initial conditions.
To evaluate the performance of H YPER FNO, we compared it in various numerical computational
problems against the original FNO (Li et al., 2020) and the U-Net (Ronneberger et al., 2015).
Results Figure 2 shows the MSE error at test time for the Burgers, 2-d Darcy Flow, ReactionDiffusion, and 2d+time compressible Navier-Stokes simulations. We trained over 10 tasks and 100
epochs, where we split the data between training and testing 60%/40% over 1′ 000 samples (500 for
CNS). We use a 128 dimensional grid for 1-d equations, while 64 × 64 for the 2-d problems. We
use GeForce RTX 2080 GPU for 1D simulation, while GeForce GTX 3090 for 2D Navier Stocks
simulations. In visualizing the results, we use a log-scale plot when the scale is too large. The CNS
equation is trained auto-regressively, while the others are trained to predict a specific time step. In
all cases, FNO has better performance with respect to U-Net, while H YPER FNO presents greater
flexibility to adapt to the new environment. Figure 3 shows qualitatively that FNO is not able topredict the solution of the Burgers equation even if the task was seen during training.

4

Conclusion

We propose H YPER FNO, a method to adapt the FNO architecture over a wide range of parametersof the PDE. We show the improvement gained over different physics systems, such as the Burgers
equation, the reaction-diffusion, the Darcy flow, and the compressible fluid dynamic. Meta-learning
for Physics Informed Machine Learning is an important direction of research and we proposed amethod in this direction that allows us to adapt NOs to new environments. The current method islimited to PDE of the hydro dynamic family. In its mathematical form, the proposed method has
direct foreseen no negative societal impact.

4

References
Christopher J. Arthurs and Andrew P. King. Active training of physics-informed neural networks
to aggregate and interpolate parametric solutions to the navier-stokes equations. Journal ofComputational Physics, 438:110364, Aug 2021. ISSN 00219991. doi: 10.1016/j.jcp.2021.110364.
V. I. Avrutskiy. Neural networks catching up with finite differences in solving partial differential
equations in higher dimensions. Neural Computing and Applications, 32(17):13425–13440, Sep
2020. ISSN 0941-0643, 1433-3058. doi: 10.1007/s00521-020-04743-8. arXiv: 1712.05067.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary
differential equations. Advances in NeurIPS, 31, 2018.
John Guibas, Morteza Mardani, Zongyi Li, Andrew Tao, Anima Anandkumar, and Bryan Catanzaro.
Adaptive fourier neural operators: Efficient token mixers for transformers. ICLR, 2022.
David Ha, Andrew Dai, and Quoc V Le. Hypernetworks. arXiv preprint arXiv:1609.09106, 2016.
George Em Karniadakis, Ioannis G. Kevrekidis, Lu Lu, Paris Perdikaris, Sifan Wang, and Liu Yang.
Physics-informed machine learning. Nature Reviews Physics, 3(6):422–440, Jun 2021. ISSN
2522-5820. doi: 10.1038/s42254-021-00314-5.
Yuehaw Khoo, Jianfeng Lu, and Lexing Ying. Solving parametric pde problems with artificial neuralnetworks. European Journal of Applied Mathematics, 32(3):421–435, 2021.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew
Stuart, and Anima Anandkumar. Neural operator: Graph kernel network for partial differential
equations. arXiv:2003.03485, Mar 2020. URL http://arxiv.org/abs/2003.03485.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew
Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations.
arXiv:2010.08895, May 2021. URL http://arxiv.org/abs/2010.08895.
Konstantin Mischaikow and Marian Mrozek. Chaos in the lorenz equations: a computer-assisted
proof. Bulletin of the American Mathematical Society, 32(1):66–72, 1995.
Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay,
Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Azizzadenesheli, Pedram
Hassanzadeh, Karthik Kashinath, and Animashree Anandkumar. Fourcastnet: A global data-driven
high-resolution weather model using adaptive fourier neural operators. arXiv:2202.11214, Feb
2022. URL http://arxiv.org/abs/2202.11214.
Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzalez, and Peter W Battaglia. Learning meshbased simulation with graph networks. arXiv preprint arXiv:2010.03409, 2020.
M. Raissi, P. Perdikaris, and G. E. Karniadakis. Physics-informed neural networks. Journal ofComputational Physics, 378:686–707, Feb 2019. ISSN 0021-9991. doi: 10.1016/j.jcp.2018.10.045.
Maziar Raissi. Deep hidden physics models: Deep learning of nonlinear partial differential equations.
arXiv:1801.06637, Jan 2018. URL http://arxiv.org/abs/1801.06637.
Maziar Raissi, Alireza Yazdani, and George Em Karniadakis. Hidden fluid mechanics: A navier-stokes
informed deep learning framework for assimilating flow visualization data. arXiv:1808.04327,
Aug 2018. URL http://arxiv.org/abs/1808.04327.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical
image segmentation. In Int. Conf. on Medical image computing and computer-assisted intervention,
pp. 234–241. Springer, 2015.
Makoto Takamoto, Timothy Pradita, Raphael Leiteritz, Dan MacKinlay, Francesco Alesiani, Dirk
Pflüger, and Mathias Niepert. PDEBench: A diverse and comprehensive benchmark for scientific machine learning, 2022. URL https://darus.uni-stuttgart.de/privateurl.xhtml?
token=1be27526-348a-40ed-9fd0-c62f588efc01.
Rui Wang, Karthik Kashinath, Mustafa Mustafa, Adrian Albert, and Rose Yu. Towards physicsinformed deep learning for turbulent flow prediction. In Proceedings of the 26th ACM SIGKDDInternational Conference on Knowledge Discovery & Data Mining, pp. 1457–1466, 2020.
5

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope? [Yes](b) Did you describe the limitations of your work? [Yes](c) Did you discuss any potential negative societal impacts of your work? [Yes](d) Have you read the ethics review guidelines and ensured that your paper conforms tothem? [Yes]2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A](b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] Code will be
provided; experimental detail could not be included in the main paper.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they
were chosen)? [Yes] We presented the main setup of the experiments, but not all
experimental detail is included.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes] We reported only partial error information; we run
experiments multiple time.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)? [Yes] We reported some information on
the execution environment.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Yes, we cited theauthors of the work we based our work on.
(b) Did you mention the license of the assets? [N/A] we will provide the license of thecode.
(c) Did you include any new assets either in the supplemental material or as a URL? [No](d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating? [N/A](e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable? [N/A](b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable? [N/A](c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation? [N/A]

6