FNOPE: Simulation-based inference on function spaces with Fourier Neural Operators
Authors: Guy Moss
Leah Sophie Muhle
Reinhard Drews
Jakob H. Macke
Cornelius Schröder
Venue: NeurIPS 2025 poster ()
Keywords: simulation-based inference
Fourier Neural Operators
flow matching
applications
Primary area: probabilistic_methods
TL;DR: We propose a method for performing simulation-based inference (SBI) of function-valued parameters, and apply it to spatial inference problems in the geosciences.

Abstract: Simulation-based inference (SBI) is an established approach for performing Bayesian inference on scientific simulators. SBI so far works best on low-dimensional parametric models. However, it is difficult to infer function-valued parameters, which frequently occur in disciplines that model spatiotemporal processes such as the climate and earth sciences. Here, we introduce an approach for efficient posterior estimation, using a Fourier Neural Operator (FNO) architecture with a flow matching objective. We show that our approach, FNOPE, can perform inference of function-valued parameters at a fraction of the simulation budget of state of the art methods. In addition, FNOPE supports posterior evaluation at arbitrary discretizations of the domain, as well as simultaneous estimation of vector-valued parameters. We demonstrate the effectiveness of our approach on several benchmark tasks and a challenging spatial inference task from glaciology. FNOPE extends the applicability of SBI methods to new scientific domains by enabling the inference of function-valued parameters.

FNOPE: Simulation-based inference on function spaces with Fourier Neural Operators
Guy Moss
Leah Sophie Muhle
Reinhard Drews?
Jakob H. Mackel241
Cornelius Schröder12.t
Machine Learning in Science, University of Tübingen, Tübingen, Germany
Tübingen AI Center, Tübingen, Germany
'Department of Geosciences, University of Tübingen, Tübingen, Germany *Department Empirical Inference, Max Planck Institute for Intelligent Systems, Tübingen, Germany *Jont supervision
Abstract
Simulation-based inference (SBI) is an established approach for performing Bayesian inference on scientific simulators. SBI so far works best on low- dimensional parametric models. However, it difficult to infer function-valued parameters, which frequently occur in disciplines that model spatiotemporal pro- cesses such as the climate and earth sciences. Here, we introduce an approach for efficient posterior estimation, using Fourier Neural Operator (FNO) architecture with a flow matching objective. We show that our approach, FNOPE, can perform inference of function-valued parameters at fraction of the simulation budget state of the art methods. addition, FNOPE supports posterior evaluation at arbitrary discretizations of the domain, well as simultaneous estimation of vector-valued parameters. We demonstrate the effectiveness of our approach on several benchmark tasks and challenging spatial inference task from glaciology- FNOPE extends the applicability of SBI methods to new scientific domains by enabling the inference of function-valued parameters.
1 Introduction
Probabilistic inference of mechanistic parameters in numerical models is ubiquitous task across many scientific and engineering disciplines. Among methods for Bayesian inference, simulation- based inference (SBI, (1+6) has emerged as powerful approach for performing inference without requiring explicit formulation or evaluation of the likelihood. Instead, SBI only requires a simulator model which can sample from the likelihood. By training generative model pairs of parameters and simulation outputs, SBI can directly estimate probability distributions such as the posterior distribution.
However, existing SBI methods are designed infer a limited number of vector-valued parameters, which strongly limits their use for inferring spatially and/or temporarily varying, function-valuee parameters. In these cases, parameters are commonly inferred fixed discretizations of the domain. Despite some recent advances leveraging generative models to infer digher-dimensional posterior
"ff rffirstaame.o "(firstname. secooddaaann))mmmlmleuueieeee
Code available
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

mechanistic
block
model 20381 PVAS1 of vf
Speitall ai-or
Darcy flw (a(s,a/Vi) f(9)
functional parameterss
simulations
FNOPE
posterior
Figure 1: Overview. FNOPE approximates the posterior over function-valued parameters a mechanistic model conditioned on function-valued observations. We FNO architecture with a flow matching objective to efficiently represent the function-valued parameters, enabling us to estimate extremely high dimensional posterior distributions arbitrary discretizations of the domain.
distributions challenge.
7-11 the high-dimensional inference problems that arise from such approaches remain
Furthermore, current models need to be retrained for new discretizations the parameters or the observations. This is particularly challenging in fields like the geosciences, where observations cannot always be made at the same locations. An alternative to using fixed discretizations is to represent the functions using fixed set of basis functions, where the inference problem becomes inferring the basis function coefficients, as used, e.g., in [2]. However, these approaches require good selection of basis functions and suffer from trade-off between choosing sufficiently expressive basis sets, while maintaining a tractable number of parameters to infer.
To overcome these limitations, we require methods that are capable of modeling and inferring function- valued data. Here, we propose to make use of the Fourier Neural Operator (FNO, (33 architecture, which operates on fincctio-vvluee data, for performing SBI on function-valued parameters. Neural operators [4 combine operations on global features of function-valued data with local (typically pointwise) operations, thus capturing both global and local structures. In particular, FNOs use Fourier features to model the global structure. For smoothly varying data, the spectral power is concentrated in the lower frequency components of the spectral decomposition. This allows for compact representation of the global structure the data, and hence for the inference of function-valuee data on high resolution discretizations.
We present FNOPE (Fig.[) [ an inference method for function-valued parameters: trains FNOs for Posterior Estimation using flow-matching objective 19, [7]. FNOPE is capable of solving inference problems for spatially and temporally varying parameters, and can generalize to posterior evaluations non-uniform, previously unseen discretizations of the parameter and observation domains. Furthermore, FNOPE can estimate additional, vector-valued parameters of the simulator. We demonstrate these features collection of benchmark tasks, as well challenging real world task from glaciology. We compare the performance of FNOPE to SBI approaches that use fixed- discretization Pasis-function representation of the parameters, and show that FNOPE outperforms these methods, especially for low simulation budgets. Thus, FNOPE enables efficient inference for high dimensional inference problems that were previously challenging even intractable.
2 Preliminaries
2.1 Simulation-based inference
SBI designed solve stochastic inverse problems: Given simulator parametrized by a known prior distribution p(0) and observation N= the goal is to infer the posterior p(0 æ) for

---

(typically) vector-valued parameters 0 E R. The simulator implicitly defines the model likelihood p(z 8) by allowing us sample z ~ p(2' 8). We can construct training dataset by sampling from the joint p(0)p(( 0) to construct a dataset of simulations S = {(0,. xx)) ) for number of
simulations K. Standard approaches in neural posterior estimation (NPE) approximate the posterior
(0(0 x) with normalizing flow, which is trained by minimizing the negative log-likelihood
- log ( 2)) T3. In contrast to this, flow-matching posterior estimation (FMPE) [9]
learns a conditional velocity field vf..e (0x) to iteratively denoise samples from a base distribution
(typically Gaussian distribution) to the posterior distribution p(0 x)) The velocity v is trained via the flow matching objective
LYMPE =1422//
(ze 10]1
(1)
where (z 0) are the sample-conditional flow paths for Σt, and ue(ze 0) are the true velocity fields. The sample-conditional paths are chosen so that pt and ue are analytically tractable.
2.2 Fourier Neural Operators
We use FNOs 13
to efficiently learn the posterior distribution of function- valued parameters. FNOs class of neural operators using the Fourier basis an intermediate representation of functional data to learn mappings between function spaces. We assume to have a bounded domain D C Rd on which we define function spaces A(D; R(): and ((D:R%) The goal of neural operators is to approximate some given operator g A B by a learnable operator g A B. In practice, the function-valued data is represented as discretizations of sample functions as E A, b; E B on the domain D. A single-layer FNO, G A B, is defined by
b(x) a (WWaaaa + (K*a) Va D,
(2)
where We is learnable linear operator, is (pointwise) non-linearity, and
(x) = F (R* E ED
Here, F and F-1 refer to the Fourier and inverse Fourier transformation, and Re refers to some operator acting the Fourier modes of a. Typically, R is linear transformation, and therefore corresponds to a convolution in real space with KC. But typically Ro only acts on the lower Fourier modes, discarding higher ones, and therefore gives rise compact representation of high-resolution data. However, as Eq. 2 includes the linear operator W (z) FNOs are still able to capture local structures.
Method
To extend the standard SBI setting to inferring function-valued parameters, develop FNOPE by extending FMPE with FNOs as backbone (Figs. (F))..[[[). FNOPE takes the function- valued parameters and observations x as input, and estimates the FMPE flow-field o for function-valued parameters
using a combination of several FNO blocks.
We assume that 0 well are evaluated on discretizations specified by positions 1 and 1 which means we choose it from some set of (!,,") E D9 x D.. Here, the parameter positions & are independent of the observation positions ( and can additionally vary between samples i. To adapt the parameter prior to function- valued parameters, we define a prior draw as an evaluation of an underlying measure ya (e.g., a Gaussian Process) at specific locations 19: O ~ pro. The simulator then returns observations x at locations 1 following the likelihood Pt- (x 0, 19). Many such simulations create a dataset = {(L 0,, 1f ); for a number of simulations K. The explicit usage of the positions 1e and 1 allows for flexible conditioning of the posterior.
3.1 Function-valued FMPE objective
To learn the velocity field vf we adapt the FMPE objective function 9 for the function- setting. Given a discretized observation (Ia, 18), and a desired parameter discretization 19, we want to sample 0 ~ Pie (0 x., 1.). This is done by first sampling from a base distribution { ~ pie(E) and
3

---

then learning the velocity field oi (t, Eo, ( for Çt. In the following, we omit the arguments of V/ for clarity. The learned velocity field allows us to iteratively denoise ε into sample & from the target posterior distribution. Note that the noise distribution is discretized on the same positions 1 as the parameter 0. Similarly to Eq. the velocity field ofe is optimized via the loss function
L = 210 (1 (S.10]] us 0)]1"
(3)
Here, Pe.ro (Et 0e) describes known noising process such that { B (approximately) drawn from the base distribution (E). Furthermore, (z. 0) is the true vector field of the path defined by Pt. (&t 0z). We use the rectified flows formulation [18), such that uE(E. 0t) = (Ee 0).
The noise distribution, Ε (E), com- monly defined be independent Gaussian white noise, & ~ (0, I). Such distributions give rise to samples with a uniform power spectrum of E. As FNOs typically operate on the lower frequency modes of their inputs, independent Gaussian white noise would not be suitable base distribution choice for our application. In-
(0"11,
(ar,
FNO block
frequency limitd
special decompostion
Doo mone 10
stead, we sample noise from a Gaussian Pro- cess E ~ GP(O, k(--)) 19 20,, where .) is the square exponential kernel with lengthscale 1. Using Bochner's 's theorem (Appendix [S3.I] [2 22], the spectral density of samples
Inquurncy-mmied spectrll cecompostton
contaxtenation
Incar transformation
transforation original domain
P(f) (2 exp (-2
where dg is the domain dimension of (and therefore of (). We choose to be dependent on the highest Fourier mode M used by the FNO. This ensures that the majority of the signal power in the noise samples ε is con- served by the FNO block. We use the heuristic 7(M/2+15. which in expectation assures that > 99% of the spectral density of samples & is in the lower M frequency modes (derivation in Appendix S3.2). The covariance kernel k of the Gaussian Process is scaled to have unit marginal variance and defines the FMPE noise sampling during training via
spec.
Enear sanstormation
I 2
Figure 2: FNOPE architecture. FNOPE is based on several FNO blocks (gray): A FNO block re- ceives the diseretivation-deeendeee spectral fea- tures of the function-vall -eccuen-valled parameters and obser- vations as an input and processes them in linear layer before transforming it back to the original do- main via the approximate inverse transformation. A pointwise linear operation on the input is added, along with embeddings of the flow time, point po- sitions, and vector-valued parameters. We expand this setup several parallel channels and stack layers. The vector-valued velocities are separately estimated via a MI..
Pro (Ee 0) N(1 - (k(,,
3.2 Adapting non-uniform, unseen discretizations of the domain
To operate on non-uniform discretizations, we adopt the work of Lingsch et al. 23 and use
FNO with a type non-uniform fast Fourier transform (NUDFT, 24 ). The NUDFT allows us to approximate the first M spectral modes of data or parameters discretized on any N points in the domain. We additionally add the positions 1 and 1" to the input of the FNO blocks through multilayer
perceptrons (MLPs, omitted in Fig. 2 for clarity) [25 26
In addition to non-uniform discretizations, FNOPE is also able deal with distinct data and pa- rameter discretizations at training and evaluation time. If we can query the simulator for arbitrary discretizations 1 1s, we can generate the training data with mixed discretizations. However, this is not the case for all simulators. To mitigate errors when evaluating the posterior at non-uniform discretizations unseen during training, we perform data augmentation during training. First, we independently mask parts of the parameters and observations by randomly removing entries of 0 and J; and the corresponding positions 1 and If. Second, we add small, independent Gaussian noise to the remaining positions (Appendix $3.3).

---

In addition to this flexible implementation. we provide and evaluate FNOPE (fix), variant of FNOPE which uses the fast Fourier transform (FFT) the FNO blocks and can be used for applica- tions which exclusively consider parameters and observations discretized on uniform grids. The FFT computes spectral components in O(N log N) in the number of points in the discretization N, com- pared to O(N MD) for the NUDFT, where D is the domain dimension. Therefore, FNOPE scales favorably with the parameter dimensionality FNOPE (fix) scales favorably with the domain dimensionality D (details in Appendix S3.4
(the number points in the discretization), but
3.3 Inferring additional parameters
As most real world simulators have additional vector-valued parameters 7), we extend the FNOPE ar- chitecture to also infer their posterior distribution. Vector-valued parameters are drawn from known prior distribution p(7) and the model likelihood becomes Pty (z; 0, (..1,))) 1). The resulting inference problem is estimate the posterior distribution (i= Zo, 1.) Hence, we now condition the velocity field vio additionally on the vector-valued parameters rj by embedding them into the channel-dimensee and subsequently adding them to the input of the FNO blocks (Fig. 2
To estimate the velocity field the vector-valued parameters Dy (Fig. 2). we use a multilayer perceptron (MLP) to process the spectral features of the output of the final FNO block together with the vector-valued parameters and the spectral features of the observation. This approach results in
network which targets the combined velocity e = [uge: u The combined network can be trained by an extension of the loss in Eq.
where the noise p(z 1)) of the vector-valued parameters is given by normal distribution N(2;; (1 t)n, I)), and [Er 0), (2 1)] The vector field ux n) for the vector- valued parameters is analogously defined as us 77) = (22 - n). In practice, we separately normal- ize the loss for efo and e by the number of parameters (Appendix S3.5
4 Experiments
We apply FNOPE to four simulators: a Gaussian lincar toy example, the SIRD model from epi- demiology, the Darcy flow inverse problem and real world application from glaciology (details in Appendix S5).
For the linear Gaussian simulator, we can analytically compute the posterior distribution, allowing us to compare the estimated posterior distributions this ground truth using the Sliced- Wasserstein Distance (SWD) (27. However, as is common in SBI applications, we do not have access to the ground truth posterior for the other simulators. Instead, we use combination of two metrics to measure the quality of our posteriors: First, we report the predictive mean square error (Pred. MSE) between ground truth observations and predictive simulations from posteriors conditioned on those observations. We complement this metric with simulation-base posterior marginal distributions. We quantify posterior calibration using the Error of Diagonal (EoD), measuring the average distance of the calibration curve of the estimated posterior from perfectly calibrated posterior. Good performance on both of these metrics is not a sufficient condition to indicate a correctly estimated posterior, but healthy posteriors typically achieve good performance these metrics. All evaluation metrics are averaged over three runs and we report mean + standard error. We provide more details on all evaluation metries in Appendix [S2 We provide an overview of training and sampling times, as well as network sizes and computational resources used for all tasks, in Appendix ST
calibration (SBC) 28 the
4.1 Baseline methods
We compare FNOPE to three baseline methods: NPE (with normalizing flows) B and FMPE g on the coefficients of the spectral basis functions of the parameters (NPE/FMPE (spectral) respectively, details in Appendix (S4). We also compare to FMPE with a fixed parameter discretization (FMPE (raw)).
For all baseline methods, we use the shi toolbox 29 For the SIRD simulator we compare to Simformer [30), transformer-basee amortized inference approach that is also capable of flexible
5

---

discretization of function-valee valued parameters. The other baselines cannot be applied in their basic version to this task because they do not support onn-ummfomm discretizations of both parameters and observations.
4.2 Linear Gaussian
We first show the ability of FNOPE approximate the true posterior of a linear Gaussian, as commonly done in SBI benchmarks [31]. To illustrate FNOPE's ability to infer a large number of parameters, we increase the dimensionality to 1000. We also replace the independent Gaussian prior in this task with a Gaussian process to model smoothly-varying function-values parameters.
SWD
FNOPE clearly outperforms all benchmark methods on this problem (Fig. 3). With training dataset of 10 sim- ulations the SWD is close to zero for both FNOPE and FNOPE (fix). In contrast, both NPE and FMPE based on spectral features need as many as 10 simulations to achieve similar performance. Furthermore, this exam- ple shows that the data augmentation applied in training FNOPE, results in a small difference between FNOPE and FNOPE (fix), which is an effect of the introduced posi- tional noise. FNOPE learns a posterior under slightly broader likelihood than what is defined by the model and for very constrained posteriors the posterior quality slightly poorer. However, we will see that for more challenging tasks, this is an acceptable trade off, as we gain flexibility on evaluation points.
10
10
10
10"
# simulations
FNOPE
NPE (spectral)
FNOPE (fix)
(spectral)
FMPE (raw)
Figure 3: Linear Gaussian simulator. Sliced-Wasseres Wasserstein distance (SWD) to ground truth posterior.
We perform ablation experiments (Appendix SS7.1) and observe that the performance of FNOPE is dependent on using sufficiently many Fourier modes in the FNO blocks (Fig, ST..b). However, other hyperparameters show less influence on the performance (Fig. --e-)).
4.3 SIRD: Inference on unseen, non-uniform discretizations
Next, we consider the Secceptible-Iffectad -nf-eeee---ecooeeeeeddddd aecovered-Deceased (SIRD) model [32 to demonstrate the ability of FNOPE to solve inference problems on non-uniform discretizations of the parameters and observations that were not seen in the training data. In addition, we also show its ability to simultancously infer veluer-valued parameters. The model has three parameters: recovery rate, death We use the same setup as in Glocckler et al. [30], where we assume time, but recovery and death rates constant in time. We sample training simulations on a dense uniform grid for both parameters and observations. For evaluation we sample 100 observations, each discretized
rate, and contact rate [33 3
that the contact rate varies
different set of 40 randomly sampled time points in [0, 50). using contact rates defined on a distinct set of 40 randomly sampled times.
FNOPE, well as Simformer, can reliably infer the posterior distribution (h) and the observa- tions lie close to the mean of the posterior predictive (Fig. 4p). Both methods are comparable terms of MSE of posterior predictive samples the observations, as well producing well-calibrated pos- teriors (Fig. (b). When we use only 20 timepoints to condition on, the performance of FNOPE slightly decreases (Fig.
This highlights the necessity of the FNO block to have enough observation points to perform a reliable (approx.) Fourier transformation. We also observe that FNOPE performs robustly across the base distribution lengthscale (Fig. [S3). This experiment shows that FNOPE is par with the state of the art on this low dimensional problem: successfully infers function-valuee parameters together with vector-valued parameters and be conditioned on arbitrary discretizations of the observations. However, FNOPE can also be applied to very high dimensional problems, as shown in the following experiment.
4.4 Darcy flow: Scalable inference in high dimensions
The Darey flow is defined by second order elliptic PDE and has been used to model many processes including the deformation of linearly elastic materials, or the electric potential in conductive materials.

---

Contact rate
1.0
0.4
Death
Figure 4: SIRD model. (a) Posterior conditioned on 40 time points. left: Posterior (mean = std.) of the time-varying param- eter and ground truth parameters (dashed). right: Two dimensional posterior of vector-valued param- eters and ground truth parame- ters (dot). (b) Posterior predic- tive (mean + std.) of infected, re- covered and deceased populations with observations marked. (c) up- per: MSE of posterior predictive samples to observations. lower: Simulation-banl calibration error of diagonal (SBC EoD). 'Lower bound' refers to the SBC FoD for uniformly sampled posterior ranks (details in Appendix (2).
0.0
0.2
40
0.2
Time [days]
Recovery rate 3
x10
0.1
FNOPE
Infected
Simformer
Pred. MSE
8
Data
2004
Recovered
0.2
0.0
0.2
lower bound
SBC EoD
Deceased
0.2
0.0
0.0
20
40
10
10"
10"
Time [days]
# simulations
In the geosciences, the Darcy equation is used to describe the distribution of groundwater function of the spatially variable hydraulic permeability, which can be inferred from point observations in wells [35] The Darcy flow is common benchmark model for FNO applications, especially in the context of training PDE emulator models [13 36 37].
We consider the steady-state of the two dimensional Darcy flow equation unit square:
-V (a(x) Vu(x)) =
E (0, 1)²
u(x)
000,
where azx) 2 is the permeability we want to infer and u(x) is the hydraulic potential. We adapt implementation from (38], which provides a GPU-optimized solver. We use a log-normal prior distribution for the permeability, similar to Lim et al. 37 b log (a) is the Laplacian operator and - 9. We sample the prior on a 129 x 129 grid which results in 16k parameters. For FNOPE, spectral NPE/FMPE we used the first 16 modes, resulting in 2 162 512 parameter dimensions. For all methods, we infer the tog-permeability and evaluate in the original space (as in (3)).
N(0, (-A - where
the first 32 Fourier modes in both spatial dimensions and for
Samples from the posterior inferred with FNOPE closely resemble the ground truth (Fig. 5 and Fig.SO). S6). Both FNOPE and FNOPE (fix) correctly capture the fine-structure of the posterior samples and reproduce parameters at much higher fidelity than all baseline methods. While the spectral methods learn oversmoothed posteriors that do not capture local structures, the posterior samples from FMPE (raw) are much noisier and only capture the rough global structure. The posterior means show similar trend (Fig. (Fig.(7)). and the standard deviations of the baseline methods are higher compared to FNOPE (Fig. S8).
The MSEs between posterior predictive samples and ground truth observations of FNOPE and FNOPE (fix) are consistently better compared the spectral baseline methods, especially at lower simulation budgets, and are in the same range FMPE (raw) (Fig. 5b). (Fig.5p). While all methods are reasonably well-calibrated (Fig. (5b), the visual appearance is vastly different. We additionally measure the posterior quality in terms of posterior log- probabbbbyyl (normalized by the number of pixels) the associated ground truth parameter [31]. FNOPE has much higher log probability (Fig. 5l) compared to FMPE (raw). FNOPE (fix) also achieves strong performance for sufficient number of simulations. As the spectral methods not model the parameters directly, we cannot calculate the prg-bbbbaiilities they assign to the ground truth parameters. Overall, FNOPE is the only
7

---

Ground truth
FNOPE (fix)
FNOPE
NPE (spectral)
FMPE (spectral)
FMPE (raw)
8
C
x10
Posterior log prob
0.2
lower bound
Pred. MSE
SBC EoD
3
-4
10
10
10
10
10"
10
# simulations
# simulations
simulations
FNOPE (fix)
FNOPE
NPE (spectral)
FMPE (spectral)
FMPE (raw)
Figure 5: Darey flow. (a) Ground truth parameter and posterior samples for a simulation budget of 10a training samples (more posterior samples in Fig. (S6). (b) MSE of posterior predictives to the ground truth observation (FNOPE, FNOPE (fix) and FMPE (raw) visually overlap). (c) Simulation- based calibration Error of Diagonal (EoD) for 50 dimensions. (d) Posterior log-probability of ground truth samples normalized by the number of dimensions (higher is better).
method that consistently performs well on all presented metries. additional ablation experiments (Fig. S4). we show that FNOPE attains strong performance across different hyperparameter choices, and that our lengthscale heuristic (Sec. 3.T is an appropriate choice for this task.
4.5 Mass balance rates of Antarctic ice shelves: Real world application
Finally, we turn real-world task from glaciology: Inference of snow accumulation and basal melt rates of Antarctic ice shelves from radar internal reflection horizons (IRHs) [39/41]. Snow continuously accumulates on top of the shelf. Over time, it transported to larger depths where the former surfaces are further deformed by ice flow and form internal layers of constant age, which are measured by radar (Fig. (b..b. The inference of accumulation and melt rates challenging SBI task, where the model is misspecified, as it cannot accout for all real-world effects. We use an isochronal advection scheme forward model deseribed in [39]. In this work, the authors consider simulations on grid of 500 points along one-dimensional spatial domain and directly infer 50 parameters on fixed downsampling of this domain using NPE, We refer to this approach as NPE (raw) and compare it to FNOPE and the baseline methods.
First, we evaluate all methods on test of simulations (Fig. 6c). As with the previous tasks, the performance of FNOPE at 10xh simulations is comparable to the performance of the other methods at 10s simulations in terms of predictive MSE, and is only marginally worse 102 simulations. All methods show a reasonable calibration in terms of SBC EoD at all simulation budgets (Fig. (dd). We then test the performance of all methods on real data (as in (.). Posterior predictive samples from FNOPE match the observation very well (Fig/op), and while FNOPE still performs better at low simulation budgets than the other methods, the relative improvement compared to the baselines is smaller than the one observed on synthetic data (Fig. (ee. We note that this modeling problem was explicitly set up by Moss et al. (39 so that NPE (raw) can infer the posterior using feasible number of simulations. FNOPE achieves the same performance with two orders of magnitude fewer

---

Elevation [m.a.s.]]
100
lce shelf
Posterior
Observed
50
250
-750
20
40
60
80
100
120
Distance from grounding line [km]
0.2
Pred. MSE
(synth. obs.)
0.6
SBC EoD
lower bound
Pred. MSE (real obs.)
0.6
0.0
0.0
10
10
10
10
10
10
10"
10
10
10
10
10"
# simulations
# simulations
# simulations
FNOPE
NPE (spectral)
FMPE (spectral)
NPE (raw)
FMPE (raw)
Figure 6: Mass balance rates of ice shelves. (a) Measurement transect of the radar data in Ekström Ice Shelf, Antarctica (adapted from [39], published under Creative Commons CC BY license). (b) Posterior predictive results obtained with FNOPE (trained on 10 simulations), compared to radar- based observation one layer the ice shelf. (c)-(e) Performance measures on simulations and the real observation, where NPE (raw) refers to the method used in [39
simulations. Additionally, FNOPE able to infer the full parameter dimensionality (500) instead of downsampling to 50 dimensions (Fig. S5
Discussion
We present FNOPE, a simulation-based inference method using Fourier Neural Operators to efficiently infer function-valued parameters. On variety of task, we showed that FNOPE can infer posteriors for function-valle valuedd data at very small simulation budgets compared to baseline methods, especially for high-dimensional problems. In addition, by building upon existing work for FNOs on non-uniform discretizations of the domain, FNOPE can generate samples from posteriors and observations defined on any discretizations of the domain, even if these discretizations were never seen during training.
Related work Scaling SBI methods to high- digensinnni parameter spaces has been the focus of many works which make use of state of the generative modeling techniques such as generative adversarial networks [7]. diffusion models [ [0] I1 and flow matching [9] In particular, recent use a transformer architecture to tokenize function-valle valued parameters, allowing for complete flexibility in estimating conditional distributions. However, as these methods explicitly model each point for discretized function cunctioo-valued parameters, they are limited in terms of scalability. Our FNO-based approach allows us compactly represent the parameters, significantly lowering the computational costs as the number parameters grows. Finally, for applications where only the one-dimensional marginal distributions of the Bayesian posterior are needed, it is possible to scale SBI methods to higher dimensions, as the correlation between the parameters does not need to be captured (4). However, in most scientific applications the correlation structure is essential object of interest and marginal distributions are only of limited use.
works (30 42
Estimating function-valus parameters using FNOs 44.46 and other neural operators 47 44 has
been explored previous work. majority of these approaches consider deterministic inversion, or estimating single value for the parameters opposed to targeting the Bayesian posterior. While

---

probabilistic generative models such as invertible Fourier Neural Operators (iFNOs, Long et al. (50)) estimate a conditional distribution, they do not explicitly target the Bayesian posterior. We compare the performance of iFNO on the Darey task, and while it has comparable performance in terms of predictive MSE, the uncertainty estimates are not well-calibrated and the conditional distribution collapses to a tiny parameter region (Appendix S8
The closest neighbours our work are Lingsch al. [51] who use FNO architecture with an FMPE objective to learn vector-valued parameters and, additionally, learn an emulator producing cunclion-valued observations. The crucial difference to our approach that their simulators are deterministic and the inferred parameters are not canction-valuedd Recently, Lim et al. [37 developed an approach for score-based modeling function spaces using FNOs. This enables high-dimensional posterior inference, but their approach is limited to uniform grids and does not allow for flexible conditioning.
Another related approach is diffusion posterior sampling (4), which seeks to learn a high- dimensional prior distribution from samples using score-based models. The learned priors can then be used to generate samples from the posterior using analytically tractable model likelihoods [55] Other works extended such approaches for intractable likelihoods [57] 58). Similarly diffusion posterior sampling, we only require prior samples. However, instead of learning the prior distribution, we learn the posterior distribution directly.
Limitations The FNO-backbone used by FNOPE inherently makes assumptions about the structure of the parameters. These assumptions enable computationally efficient inference, but result in some limitations. First, the FNO assumes limited high high-frequeccy information in the parameter and observation domains. Therefore, they are ill-suited to infer parameters with high power in higher frequencies example, parameters with discontinuities. This could potentially be addressed by neural operators using other transforms, such as wavelet transforms (59]. Furthermore, to (accurately) compute the FFT or NUDFT of the observations, we require sufficiently many points in their discretization. Therefore, unlike other flexible methods [30, 42), our approach cannot perform inference using extremely sparse observations. Still, the SIRD experiment shows that even for 20 points we get reasonable estimations (Fig. S2).. Finally, the computational complexity of our approaches still scales exponentially with the domain dimension, which could be challenging in high-dimensional domains.
Conclusion We presented FNOPE, a simulation-based inference approach for inferring function- valued data. FNOPE can be applied to non-uniform, unseen discretizations of the domain, can scale to large parameter dimensions, and can be trained using comparatively small simulations budgets. As we show in various experiments, FNOPE can therefore tackle spatiotemporal inference problems that were previously challenging or even intractable for simulation-bast based inference.
Acknowledgments and Disclosure of Funding
This work was funded by the German Research Foundation (DFG) under Germanys Excellence Strategy EXC number 2064/1 390727645 and SFB 1233 Robust Vision (276693517) and DFG (DR 822/3-1) and the Heinrich-Bölll-ttffne Söll-Sttnn,,.,..g..... Data collection was supported by Alfred Wegener Institute through logistic grants SWI_ANT 18. The authors acknowledge support by the state of Baden- Wirtnn--Wrettbrerggg through bwHPC. GM
is a member of the International Max Planck Research School for Intelligent Systems (IMPRS-IS). We thank Manuel Gloeckler for providing the training data and Simformer results for the SIRD experiment. We thank Daniel Gedon and Julius Vetter, and all members of Mackelab for discussions and feedback on the manuscript.
References
[1] George Papamakarios and lain Murray. Fast e -free inference of simulation models with bayesian conditional density estimation. In Advances in Neural Information Processing Systems, volume 29, 2016.
[2] Jan-Matthis Lueckmann, Pedro Goncalves, Giacomo Bassetto, Kaan Öcal, Marcel Nonnenmacher, and Jakob H Macke. Flexible statistical inference for mechanistic models of neural dynamics. In Advances in Neural Information Processing Systems, volume 30, 2017.
10

---

[3] David S. Greenberg. Marcel Nonnenmacher, and Jakob H. Macke. Automatic posterior transformation for likelihood-free inference. In International Conference on Machine Learning. 2019.
[4] George Papamakarios, David Sterratt, and lain Murray, Sequential neural likelihood: Fast likelihood free inference with autoregressive flows. In The 22nd international conference on artificial intelligence and statistics. PMI.R 2019.
[5] Conor Durkan, lain Murray, and George Papamakarios. On contrastive learning for likelihood inference. In Proceedings of the 37th International Conference on Machine Learning, volume 119. PMLR, 2020.
[6] Stefan T. Radev, Ulf Kai Mertens, Andreas Voss, Lynton Ardizzone, and U. Köthe. Bayesflow: Learning complex stochastic models with invertible neural networks. IEFE Transactions on Neural Networks and Learning Systems, 33, 2020.
[7] Poornima Ramesh, Jan-Mattin Matthis Lueckmann, Jan Boelts, Álvaro CanttrrCantero, David S. Greenberg. Pedro Gonealves, and Jakob Macke. GATSBI: Generative adversarial training for simulation based inference. International Conference on Learning Representations, 2022.
[8] Tomas Geffner, George Papamakarios, and Andriy Mnih. Compositional score modeling for simulation- based inference. In International Conference on Machine Learning, 2022.
[9] Jonas Wildberger. Maximilian Dax, Simon Buchholz, Stephen Green, Jakob Macke, and Bernhard Schölkopf. Flow matching for scalable simulation hassed inference. In Advances in Neural Information Processing Systems, volume 36, 2023.
[10] Marvin Schmitt, Valentin Pratz, Ullrich Koethe, Paul-Christian Bürkner, and Stefan T. Radev. Consistency models for scalable and fast simulation-basee inference. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024.
[11] Julia I. Linhart, Gabriel Cardoso, Alexandre Gramfort, Sylvain Le Corff, and Pedro I.. C. Rodrigues. Diffusion posterior sampling for simulation
inference in tall data settings. ArXiv, 2024.
[12] R. Hull, Leonarduzzi, L. De La Fuente, H. Viet Tran, A. Bennett, P. Melchior, Maxwell, and
L. E. Condon. Simulation- based inference for parameter estimation of complex watershed simulators. Hydrology and Earth System Sciences, 28, 2024.
[13] Zongyi Li, Nikola Borislavov Kovachki, Kamyar Azizzadenesheli, Burigede liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. In International Conference on Learning Representations, 2021.
[14] Lu Lu, Pengzhan Jin, Guofei Pang. Zhongqiang Zhang, and George Em Kamiadakis. Learning nonlinear operators via deeponet based the universal approximation theorem of operators. Nature Machine Intelligence, 3, 2019.
[15] Nikola Kovachki. Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces with applications to pdes. Journal of Machine Learning Research, 24, 2023.
[16] Nikola B. Kovachki, Samuel Lanthaler, and Andrew M. Stuart. Operator learning: Algorithms and analysis. arXiv, 2024.
[17] Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations, 2023.
[18] Xingchao Liu, Chengyue Gong, and qiang liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Representations, 2023.
[19] Marin Biloš, Kashif Rasul, Anderson Schneider, Yuriy Nevmyvaka, and Stephan Günnemann. Modeling temporal data as continuous functions with stochastic process diffusion. In Proceedings of the 40th International Conference on Machine Learning, volume 202. PMLR, 2023.
[20] Alexander Tong, Kilian Fatras, Nikolay Malkin, Guillaume Huguet, Yanlei Zhang, Jarrid Rector- Rector-Brooks, Guy Wolf, and Yoshua Beagio. Improving and generalizing flow- generative models with minibatch optimal transport. Transactions on Machine Learning Research, 2024.
[21] M.L. Stein. Interpolation of Spatial Data: Some Theory for Kriging. Springer New York, 2012. ISBN 9781461214994.
11

---

[22] Christopher KI Williams and Carl Edward Rasmussen. Gaussian processes for machine learning, volume 2. MIT press Cambridge. MA, 2006.
[23] Levi Lingsch, Mike Y. Michelis. Emmanuel De Bézenac. Sirani M. Perera, Robert K. Katzschmann, and Siddhartha Mishra. Beyond regular grids: Fourier-hased neural operators on arbitrary domains. In Proceedings of the 41st International Conference on Machine Learning, 2024.
[24] Leslie Greengard and June- Yub Lee. Accelerating the nonuniform fast fourier transform. SIAM Review, 46, 2004.
[25] Boris Bonev, Thorsten Kurth, Christian Hundt, Jaideep Pathak, Maximilian Baust, Karthik Kashinath, and Anima Anandkumar. Spherieal fourier neural operators: Learning stable dynamics on the sphere. In International Conference on Machine Learning, 2023.
[26] Zongyi Li, Nikola Borislavov Kovachki, Chris Choy, Boyi Li, Jean Kossaifi, Shourya Prakash Otta, Mohammad Amin Nabian, Maximilian Stadler, Christian Hundt, Kamyar Azizzadenesheli, and Anima Anandkumar. Geometry-inforaae neural operator for large-scale 3d PDEs. In seirty-seventh Conference on Neural Information Processing Systems, 2023.
[27] Nicolas Bonncel, Julien Rabin, Gabriel Peyré, and Hanspeter Pfister. Sliced and radon wasserstein barycenters of measures. Journal of Mathematical Imaging and Vision, 51, 2015.
[28] Sean Talts, Michael Betancourt, Daniel Simpson, Aki Vehtari, and Andrew Gelman. Validating bayesian inference algorithms with simulation based calibration. ArXiv, 2018.
[29] Jan Boelts, Michael Deistler, Manuel Gloeckler, Álvaro Tejero- Cantero, Jan-Matthis Lueckmann, Guy Moss, Peter Steinbach, Thomas Moreau, Fabio Muratore, Julia Linhart, Conor Durkan, Julius Vetter, Benjamin Kurt Miller, Maternus Herold, Abolfazl Ziacemehr, Matthijs Pals, Theo Gruner, Scbastian Bischoff, Nastya Krouglova, Richard Gao, Janne K. Lappalainen, Bálint Mucsányi, Felix Pei, Auguste Schulz, Zinovia Stefanidi, Pedro Rodrigues, Cornelius Schröder, Faried Abu Zaid, Jonas Beck, Jaivardhan Kapoor, David S. Greenberg, Pedro Gonçalves, and Jakob Macke. sbi reloaded: toolkit for simulation-based inference workflows. Journal of Open Source Software, 10, 2025.
[30] Manuel Gloeckler, Michael Deistler, Christian Weilbach, Frank Wood, and Jakob H. Macke. All-in-one simulation-basee inference. In Proceedings of the 4t International Conference on Machine Learning, 2024.
[31] Jan Jan-Matthis Lueckmann, Jan Boelts, David Greenberg, Pedro Goncalves, and Jakob Macke. Benchmarking simulation-based inference. In International conference on artificial intelligence and statistics, 2021.
[32] William Ogilvy Kermack and Anderson G McKendrick. A contribution the mathematical theory of epidemics. Proceedings of the royal society of london. Series A, Containing papers of mathematical and physical character, 115, 1927.
[33] Yi-Cheng Chen, Ping-En Lu, Cheng- Shang Chang, and Tzu-Hsuan Liu. time-dependent sir model for covid-19 with undetectable infected persons. leee transactions on network science and engineering, 7, 2020.
[34] Jonathan Schmidt, Nicholas Krämer, and Philipp Hennig. A probabilistic state space model for joint inference from differential equations and data. Advances in neural information processing systems, 34, 2021.
[35] Wolfgang Nowak and Olaf A. Cirpka. Geostatistical inference of hydraulic conductivity and dispersivities from hydraulic heads and tracer data, Water Resources Research, 42, 2006.
[36] Zongyi Li, Hongkai Zheng, Nikola Kovachki, David Jin, Haoxuan Chen, Burigede Liu, Kamyar Aziz- zadenesheli, and Anima Anandkumar. Physics-informed neural operator for learning partial differential equations. ACM/JMS Journal of Data Science, 1, 2024.
[37] Jae Hyun Lim, Nikola B Kovachki, Ricardo Baptista, Christopher Beckham, Kamyar Azizzadenesheli, Jean Kossaifi, Vikram Voleti, Jiaming Song, Karsten Kreis, Jan Kautz, et al. Score-based diffusion models in function space. ArXiv, 2025.
[38] NeMo Contributors. Nvidia physicsnemo: An open-source framework for physics-hased deep learning in science and engineering. version: 2023.
[39] Guy Moss, Vjeran Vinjevi, Olaf Eisen, Falk M. Oraschewski, Cornelius Schröder, Jakob H. Macke, and Reinhard Drews. Simulation-base inference of surface accumulation and basal melt rates of an antarctic ice shelf from isochronal layers. Journal of Glaciology, 71, 2025.
12

---

[40] Edwin D. Waddington, Thomas A. Neumann. Michelle R. Koutnik, Hans Peter Marshall, and David L. Morse. Inference of accumulation-rate patterns from deep layers in glaciers and ice sheets. Journal of Glacialogy, 53, 2007.
[41] J. Wolovick, J. C. Moore, and I.. Zhao. Joint inversion for surface accumulation rate and geothermal heat flow from ice-penetrating radar ohservations at dome a, cast antarctica. part i: Model description, data constraints, and inversion results. Journal of Geophysical Research: Earth Surface, 126, 2021.
[42] Paul E. Chang, Nasrulloh Loka, Daolang Huang, Ulpu Remes, Samuel Kaski, and Luigi Acerbi. Amortized probabilistic conditioning for optimization, simulation and inference. ArXiv, 2024.
[43] Luca Ambrogioni, Umut Güęlü, Julia Berezutskaya, Eva van den Borne, Yagmur Güęlütürk, Max Hinne, Eric Maris, and Marcel van Gerven. Forward amortized inference for likelihood free variational marginal- ization. In Kamalika Chaudhuri and Masashi Sugiyama, editors, Proceedings of the Twenty Secoon International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pages 777-786. PMLR, 16-18 Apr 2019.
[44] Md Ashiqur Rahman, Manuel Florez. Anima Anandkumar, Zachary Ross, and Kamyar Azizzade- nesheli. Generative adversarial neural operators. Transactions on Machine Learning Research, 2022. ISSN 2835-8856.
[45] Jacob H Seidman, Georgios Kissas, George J. Pappas, and Paris Perdikaris. Variational autoencoding neural operators. In Proceedings of the 40th International Conference on Machine Learning, volume 202.
PMLR, 2023.
[46] Abdolmehdi Behroozi, Chaopeng Shen, and Daniel Kifer. crnsitivity-constrainee fourier neural operators for forward and inverse problems parametric differential equations. In The Thirteenth International Conference on Learning Representations. 2025.
[47] Zhongkai Hao, Zhengyi Wang. Hang Su, Chengyang Ying. Yinpeng Dong, Songming Liu, Cheng, Jian Song, and Jun Zhu. GNOT: A general neural operator transformer for operator learning. In Proceedings of the 40th International Conference on Machine Learning, volume 202. PMI.R, 2023.
[48] Tian Wang and Chuang Wang. Latent neural operator for solving forward and inverse pde problems. In Advances in Neural Information Processing Systems, volume 37, 2024.
[49] Haixu Wu, Huakun Luo, Haowen Wang, Jianmin Wang, and Mingsheng Long. Transolver: A fast transformer solver for pdes general geometries. In International Conference on Machine Learning. 2024.
[50] Da Long, Zhitong Xu, Qiwei Yuan, Yin Yang, and Shandian Zhe. Invertible fourier neural operators for tackling both forward and inverse problems. Proceedings of The 28th International Conference on Artificial Intelligence and Statistics, volume 258, PMLR, 2025.
[51] Levi E. Lingsch, Dana Grund, Siddhartha Mishra, and Georgios Kissas. FUSE: Fast unified simulation and estimation for PDEs. In The Thirty-eighth Annual Conference on Neural Information Processing Syxtems, 2024.
[52] Yang Song, Jascha Sohl-DDcksttein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
[53] Jiaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz. Pscudoinverse- guiddd diffusion models for inverse problems. In International Conference on Learning Representations, 2023.
[54] Hyungjin Chung, Jeongsol Kim, Michael Thompson Mccann, Marc Louis Klasky, and Jong Chul Ye. Diffusion posterior sampling for general noisy inverse problems. The Eleventh International Conference on Learning Representations, 2023.
[55] Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. In The Eleventh International Conference on Learning Representatios, 2023.
[56] Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, 2022.
[57] Gabriel Cardoso, Yazid Janati el idrissi, Sylvain Le Corff, and Eric Moulines. Monte carlo guided denoising diffusion models for bayesian linear inverse problems. In The Twelfth International Conference on Learning Representations, 2024.
13

---

[58] Zihui Wu, Yu Sun, Yifan Chen, Bingliang Zhang, Yisong Yue. and Katherine Bouman. Principled probabilistic imaging using diflusion models plug-and-play priors. In The Thirty-eighth Annual Conference on Nearal Information Processing Systems, 2024.
[59] Tapas Tripura and Souvik Chakraborty. Wavelet neural operator for solving parametric partial differ- ential equations computational mechanies problems. Computer Methods Applied Mechanics and Engineering, 404, 2023.
14

---

Supplementary Material
S1
Software and Computational Resources
For all baseline SBI methods, the sbi toolbox [29], for the Simformer baseline the
publicly available code from Gloeckler et al. 30 We use an optimized solver to solve the Darcy Flow PDE 38
We use various compute resources for the different experiments. For each experiment, we run the training and evaluation for each method, for each simulation budget, and for each of the three random seeds separately. The summary network sizes and compute times for all tasks are provided in Tab. ST
For the Linear Gaussian and SIRD experiments, we perform our experiments on Nvidia RTX 2080ti GPU nodes. Both simulators have negligible wall-clock costs on these GPU nodes. The Darey flow experiment required GPUs with higher VRAM to accommodate the large (=s 16k dimensional) parameters and observations. We performed these experiments Antarctic Ice experiment, we perform training and evaluation on CPU, namely Intel Xeon Gold 16 cores, 2.9GHz. We perform this experiment on CPU as the main computation cost is in running the simulations for the predictive MSE check, and the implementation of the model not accelerated by of GPUs. The simulation costs are described in Moss et al. [39]
Nvidia A 100 GPUs. For the
Table S1: Network sizes and training times for Gaussian Linear (GL) and Darey Flow (Darcy) tasks. We report the mean over 3 runs with training budget of 10k simulations. Baselines estimating the "raw" (untransformed) parameters are denoted (R) and those estimating Fourier coefficients are
denoted (S). Linear Gaussian task was run on 2080ti GPU, Darcy flow A100 GPU.
Task
Metric
FNOPE
FNOPE (fix)
NPE (S)
FMPE (S)
FMPE (R)
GL
# params.
110K
109K
567K
86.3K
299K
GL
train (Tot.) [m]
3.12
1.98
6.77
2.46
5.14
GL
train (/epoch) [s]
0.99
0.87
2.17
0.24
0.31
GL
sample (/sample) [ms]
2.16
1.06
0.05
0.24
0.20
Darcy
# params. train (Tot.) [m]
11.6M
11.6M
3.54M
898K
9.18M
Darcy
72.2
41.8
20.5
10.9
12.2
Darcy
train (/epoch) [s]
20.3
26.2
2.82
0.66
0.73
Darey
sample (/sample) [ms]
280
35.2
0.21
1.95
2.70
Table S2: Network sizes and training times for SIRD. We report the mean over runs with training budget of 10k simulations. Both methods were trained and evaluated on 2080ti GPU.
Task
Metric
FNOPE
Simformer
SIRD
params.
117K
286K
SIRD
train (Tot.) [m]
6.95
47.17
SIRD
train (/epoch) [s]
1.19
9.43
SIRD
sample (/sample) [ms]
0.22
0.22
Table S3: Network sizes and training times for Antarctic Ice (Ice) task. We report the mean over 3 runs with training budget of 10k simulations. Baselines estimating the "raw" (untransformed) parameters are denoted (R) and those estimating Fourier coefficients are denoted (S). All methods trained and evaluated CPU.
Task
Metric
FNOPE
NPE (S)
FMPE (S)
NPE (R)
FMPE (R)
Ice
# params.
25.3K
236K
92.3K
361K
96.2K
Ice
train (Tot.) [m]
47.4
19.7
31.6
6.80
33.5
Ice
train (/epoch) [s]
9.82
4.21
1.92
5.64
2.02
Ice
sample (/sample) [ms]
22.8
1.50
16.6
1.53
23.9
15

---

S2 Evaluation details
We here describe more details about our evaluation procedures. We evaluate heldout test set (i. 725- x3. (1) where Jest is the number of test simulations. Given approximate posterior distribution 9:0 (6, x. (2)) and a test observation (r), we draw K Kus posterior samples (Okj:Ty)) ~ % (0, 7 x3. ( In the case where no vector-valued parameters are present, they can be omitted. Similarly, for methods which do not explicitly use the positions 1 1 (e.g. FNOPE (fix)), the positions can be omitted, as we do not apply these methods to tasks where we consider arbitrary discretizations.
We report the average and standard error over all Just test simulations.
S2.1 Sliced Wasserstein Distance
Following Bonneel et al. (27), we define the (empirical) sliced Wasserstein(-2) distance (SWD) between N samples from two probability distributions p and q as
K
1/27
SWD(p, 9) = (3
1 ya
K
1=1
where 24 p(x), Us (()) are samples from the two distributions, u are uniformly randomly sampled vectors on the unit sphere SD-1, and (k) (k) yuu are the I-dimensional i-th order statistics of the projections Ik; u T yı respectively. We calculate the SWD with 50 random projections u and K 1000 posterior samples.
S2.2 Simulation-based Calibration Error of Diagonal
Simulation- based calibration (SBC) 28 is a standard measure of the calibration of approximate posterior distributions (in terms of over- or underconfidence). We obtain ranks rty for each sample (05.2²) in the test set using SBC with the 1-dimensional marginal distributions used the reducing functions. That is, for each of the dimensions i of 0, the rank ri is integer in (1, K post + 1). This results in Just ranks. The cumulative distribution function of ranks is therefore
(D)
Jrost E Lrris/Koos a).
The SBC Error of Diagonal (SBC EoD) is then the mean absolute distance between this cumulative distribution and the cumulative distribution function of uniform distribution.
L (CDF;(a) alda.
SBC FoD(i)
In contrast to the SBC area under the curve (SBC AUC), the EoD will detect poor calibrations for posteriors that are overconfident at low confidence levels and underconfident at high (or Acce-versa,.Finaly we report the average SBC FoD across the dimensions 0,
SBC EoD
SBC EoD(1).
No
For SIRD, since the posterior dimensionalities are low, we compute average SBC EoD over all one-dimensional marginals the posterior. For the Darcy Flow and Antarctic Ice tasks, we select subset of 50 marginal distribution for computing the SBC EoD, regularly spread across the domain.
S2.3 Predictive MSE
To calculate the MSE for posterior predictive samples, run for each posterior sample (Ouj Meg)) and each true observation z9. the simulator øıj Ptş (x Oky, 10, Tky)) We then compute the average mean square error of the simulation økj to the corresponding observations 25.
Jua
1
1
MSE
[akk 211122.
Juea K post A 1,j=1
16

---

where | is the number of points in the discretization 1 and therefore the dimensionality of 23. We use this metric since the simulators considered in this work correspond to (unknown) unimodal likelihood functions-a correctly estimated posterior will produce simulations clustered around the true observation. We opt for this metric to quantify predictive performance due its clear interpretability. However, for multimodal likelihood functions, this metric can be replaced with scoring rule.
S2.4 Posterior Log Probability
For the Darey Flow task, we additionally report the posterior log- bog-rrbbability the true parameters 31,. normalized by the number of pixels:
log-probability per pixel
log 9/o (09 13)
For the spectral methods NPE/EMPE (spectral), we cannot directly compute the posterior- probabilities, as we can only compute the posterior-log- spectral decomposition of the ground truth parameters 02. However, by discarding the information of the higher modes, remove the information which these baseline methods cannot capture, thus biasing the resulting pogaprobabilities favor these baselines.
poperrio--og-prbbbblliiieee probabilities of the first M modes of the
S3 FNOPE details
S3.1 Bochner's Theorem
state Bochner's theorem following Williams and Rasmussen [2 A complex-valued function on Rd is the covariance function of weakly stationary mean continuous complex-valle valued random process on 2 and only if it can be represented as
/ exp2xif-r du(s)
k(T)
(5)
JRD
for some positive finite measure μ.. Crucially, the less general but relevant case that admits density P(f), the integral is a Fourier transform between the kernel i<()) and the spectral density P()). We apply this result to relate the lengthscale of the square exponential kernel to the spectral density of its Fourier decomposition in Sec. 3.1
S3.2 Kernel lengthscale heuristic
The spectral density samples from a Gaussian Process with a square exponential kernel of lengthscale / is stated in Sec. 3. as
P()) (2mt2ydo/2 exp_) (2
This is also Gaussian density, and trivially we see that the full spectral power is
P P= f Rd0 Pf
We consider discretizations normalized to [0, 1] in each dimension, and so the power contained in the first M spectral modes, Pas. corresponds to the above integral within the domain || S M/2, i.c. where all the components of are within [- M/2.. Therefore Pa simplifies to the product of the Gaussian integrals
(
M/2
PM
(2 exp( exp((2x2
-M/2
TIM
erf
1/2
do
.M 1
erf
.M2 +
17

---

where erf is the Gauss enror function, and in the last line we substituted our heuristic x(M//+1) This value saturates the enror function and produces values very close to 1. For example, for the Darcy Flow example (dg = 2), we set M = 32. The resulting spectral power in the first 32 modes is Ps2 0.9997. While individual samples from the Gaussian process can result in discrete Fourier transforms where this spectral property is not fulfilled, is clear that the majority of the spectral power will be contained in the first M modes for all samples.
2
S3.3 Flexible discretization
We provide further detail of the data augmentation scheme introduced in Sec. 3.2 First, we describe why this is necessary despite the use of the non-uniform fast fourier transform (NUDFT). The NUDFT applied matrix multiplication. O V((")a )0, where θ is vector containing the first M spectral transformation matrix. The inverse NUDFT'S is
components of 0, and V is the discretization-deeenne similarly implemented as a matrix multiplication 0 V (1" (,, where V T is the conjugate transpose of V. This approach enables the computational efficiency of the NUDFT, as the exact inverse matrix does not need be computed at runtime. However, V is only an approximate inverse of V. with the approximation error increasing for increasing non-uniformity of the discretization.
Consider the common case where the simulation dataset S (Sec. 3 provides parameters Og and observations vy always discretized on the same, uniform simulation domain. Without data aug- mentation, we always apply the NUDFT and its inverse without approximation error. However, if we wish to condition a posterior on x" measured at some non-uniform discretization (", then the NUDFT and its inverse will produce some error, which was unseen during training. This could lead to unpredictable, out of distribution errors at evaluation time. By explicitly passing the positions 1 and 1st to the network, as well as augmenting them to ensure the network is not always applied on uniform discretizations during training, we give FNOPE capacity to learn to counteract these approximation errors.
Masking We define a uniform distribution over the binary mask vectors with a fixed num- ber of nonzero entries, Na. Suppose we are given simulation (0,1%, x, 1"), where 6, 1 con- sist of Na points, and x, 1 consist of N₂ points. We construct two random binary masks, M 10, 1) M E {0, 1) each with exactly Nas nonzero entries. We then remove the cor- responding elements of 0,1 where Mg is zero, and similarly remove the corresponding elements of 1st where M, zero. For a minibatch of simulations, we independently sample the masks M M for each simulation. If Na. > No or Na. > Nz, we leave the corresponding value and position vector unchanged. The value of N& used in our work is reported for each experiment in Appendix S6
Positional noise We additionally add small, independent gaussian noise 6 (N, o²) to each point in 1 and 1s in the unmasked positions. This reduces generalization error for simulation datasets where the discretization of 0 and x fixed. The value of 0² can be set according to the spacing of the discretization. In our experiments, we always normalize the simulation domain to [0, 1] in all dimensions, and set 10
S3.4 Fixed discretization
For applications with uniform grids, we provide FNOPE (fix). Here, we use the FFT instead of the NUDFT in the FNO blocks to transform the data from physical to spectral space and back. In addition, we do not mask any parameters during training and do not add any positional noise. We do not introduce additional
expect this method to have improved performance on uniform grids noise through the data augmentation process described above.
Another potential advantage of FNOPE (fix) is its computational efficiency. Consider the case of a parameter discretized uniformly in a D-dimensional domain, with L points per dimension, leading to a total of _ points. The computational cost of computing the spectral decomposition of the parameters using the FFT is OII log I.P) O(DLP log L). Assuming the maximum number of modes in each dimension modeled by the FNO is M, this leads to MD total modes to compute. Therefore, the computational cost of the frequency- limited NUDFT O(MPI, I,P. For one-dimensional domains, the NUDFT may well be faster to compute than the FFT. However, for
18

---

higher dimensions, the NUDFT scales exponentially with both M and L. This discrepancy increases with both the dimensionality of the domain, and the number of modes M modeled by the FNO blocks.
S3.5 Additional Parameters
The näive extension of the FMPE objective as stated in Sec. 3.3 is to minimize the L loss 1 4z 1
where v -1o [5] and [u, (E,)0), u, (2(22/1)) However, the scale of the loss for the continuous parameters, ||0j (Ex varies with the number of points in the discretization 1 which we denote No. To ensure that the loss is balanced for the function- and vector-valued parameters, in practice add their 2 losses, normalized by their respective vector dimensionalities. That is, we minimize
E
No do ue (E((0))
| ue (z(ln)))
N
where N,, is the fixed dimensionality of 17. The expectation is over the same random variables as for the statement of the loss L2 in Sec. 3.3
S4
Baseline Methods
S4.1 Spectral preprocessing
For spectral NPE/PMPE we first apply a Fourier transformation to the parameters, take the first M Fourier modes and expand these M complex values to a real vector of dimension 2M - representing the real and imaginary parts (the imaginary part of the first component is always 0, hence it is discarded). For two dimensional data, the same preprocessing results in 2.2² 1 parameters. After inferring the posterior of the parameters in Fourier space and sampling from we apply the inverse Fourier transform to get samples in the spatial domain.
For the dimensional problems (Linear Gaussian and Ice Shelf) first pad the data by replicating the first/last value and perform a real FFT with torch. fft. rfft. We then use the first M fourier components and expand these complex numbers input to NPE/EMPE. For the Linear Gaussian and Ice Shelf we use M = 50, 10, respectively. We then revert this process for samples from the posterior with torch fft. irfft with the corresponding settings.
real tensor of dimension 2MM, which we as
For the two dimensional Darcy flow, the padded data (in mode replicate). We then center the frequencies before cropping to the first Fourier components in both dimensions, and expanding it to real tensor of dimension 2. For posterior samples we again revert this process with the corresponding settings.
the two dimensional FFT implemented in pytorch on
$4.2 NPE (spectral)
For NPE (spectral) we infer the posterior over the coefficients the first M Fourier modes following the spectral preprocessing described above. We use NPE 3 with normalizing flows. We do not apply spectral preprocessing to the observations but pass raw observations through an embedding net as common practice in NPE.
S4.3
NPE (raw)
For the mass balance experiment (Sec. 4.5). we also compare to the approach of Moss et al. [39]. which we refer to as NPE (raw). This approach infers the mass balance parameters on a fixed discretization of 50 gridpoints. The authors use a Neural Spline Flow with transformations, two residual blocks of 50 hidden units each, ReLU nonlinearities and 10 bins. The embedding net used to embed the 441 dimenssiii observation is a CNN with two convolutional layers of kemne size 5, with ReLU activations and max pooling of kernel size 2. The convolutional layers are followed by two linear layers with 50 hidden units and output dimension 50. The same settings are used in the 500-dimensional experiment (Appendix S7.44. Training is performed with a batch size 200 and an Adam optimizer with learning rate of 0.0005.
19

---

S4.4 FMPE (spectral)
As with NPE (spectral), in this approach we apply spectral preprocessing to the parameters, and infer the coefficients of the top M Fourier modes, but process the observations directly using embedding net. We use MLPs to estimate the flows, as in Wildberger et al. [9) as implemented the sbi toolbox [29]. This implementation also uses the rectified flow [88] objective for FMPE, and use independent Gaussian noise as the noise distribution. The flow networks are conditioned on time by concatenating the time to the inputs.
S4.5
FMPE (raw)
In this approach, we infer the parameters directly on a fixed discretization of the domain. We again use embedding nets encode the observations, and MLPs to learn the flows. As with FMPE (spectral), we use rectified flow objective, and independent Gaussian noise as the noise distribution, as in the sbi toolbox. The flow networks are conditioned on time by concatenating the time the inputs.
S4.6 Simformer
For the SIRD experiment, we apply Simformer with the same settings as in Gloeckler et al. [30] That is, we use a transformer model with token dimension of 50, 8 layers, and heads. The widening factor is 3, and the training was performed with batch size of 1000 and an Adam optimizer. We train Simformer learn all conditionals, and so uniformly draw between the posterior. joint, and likelihood masks, as well as two random masks drawn from Bernoulli distributions with p 0.3 and p = 0.7 respectively. For both STRD experiments, where we evaluate on 20 and time points respective, we use the same Simformer model which is trained using 20 randomly sampled time points.
S5
Simulators
S5.1 Linear Gaussian model
The Gaussian Simulator is inspired by Lueckmann et al. 31, but instead of a 10 dimensional Gaussian distribution with independent dimensions expanded the problem to 1000 dimensions and use Gaussian Process prior (see below). Draws from the simulator are still drawn independently per dimension x NN,, o20, where o² 0.1 (as in 31
Prior The prior defined as a Gaussian process GP on [o, 1] with an equidistant discretization with 1000 timepoints. A draw from the prior is therefore defined ~ (O, k(::)), where l: is the squared exponential kernel, k(t, t') = exp(
We set the lengthscale = 0.05 and variance 1. Only in the ablation experiments (Fig. (labb we changed the lenghtscale I 0.005.
212
Evaluation parameters
The results for Fig.
3 is based on 100 observations and 1000 posterior
samples for each observation.
S5.2 SIRD model
Similar to Gloeckler et al. 30 we extend the SIRD (Susceptible, Infected, Recovered, Deceased) model to have time-dependent contact rate. Compared to the classical SIR framework the model additionally incorporates a deceased (D) population. Similar models were explored by Chen et al. 333, Schmidt et 34. This addition is important for modeling diseases with significant mortality rates. The SIRD model, including time-dependent contact rate B(t), is defined by the following set
20

---

of differential equations:
ds B(t)SI, di
dl 8()ST - --1 1 pT,, dt II dD ul.
dR
Here, S. I. R and D are the susceptible, infected, recovered, and deceased population, ((t) is the time dependent contact rate, and y and μ are the recovery and mortality rates among the infected population. We simulate on a dense uniform grid of 100 time points for parameters and observations. The simulations are additionally contaminated by an observation noise model, which described by a log-normal distribution with mean S() and standard deviation a 0.05.
We impose the same prior as Gloeckler et [30: the global variables y and μ are drawn from uniform distribution, Y, Unif(O, 0.5). For the time-dependent contact rate we define Gaussian process prior which is further transformed by a sigmoid function to ensure that (1) E [0, I] for all t. For the Gaussian process
Prior
RBF kernel defined as k(t,, t') cxp( ||| ||--611"
2-7a
Evaluation parameters MSE as well as SBC EoD based on 100 observations with 1000 posterior samples each (Fig. 4e and Fig- (2.). To calculate the posterior predictive, sample the initial condition ((0) from the Simformer prediction for both methods, as opposed to the prior defined above, to match the setting of Gloeckler et al. 30
S5.3 Darcy Flow
Details for the Darcy model are already given in the main text. We additionally scale the log- permebility by the scale factor of 1000 before taking the exponential this results in permeabilities which produce sufficiently variable solutions using the Darcy flow simulator, and the permcabilities on the same scale reported in Lim al. 37 The simulation output is additionally corrupted by an
independent Gaussian observational noise per pixel Ç ~ (0, of) ) We set a; E(u, /3// where the expectation is per simulation batch, resulting in signal to noise ratio (SNR) of 30.
Evaluation Parameters All metrics are calculated over a test set of 10 observations (Fig. b-d). For MSE and SBC FoD, 100 posterior samples used for each observation and for each method. Finally, for SBC FoD, we use a subset of 50 pixels of the full 129 x 129 as the marginals used for the reducing functions. The same pixels were used across all methods and all observations.
S5.4 Mass balance rates of Antarctic Ice Shelves
use the same simulator as described in Moss et al. (39. This model takes in spatially varying surface accumulation rates ā()), which are related to the basal melt rates b(l) through the total balance condition rin = à(l) b(I) ri)() is known and fixed across all simulations.
The layer prediction model consists of set of isochronal layer with prescribed thicknesses (h, (1), (z)))...., hx (4)) such that Σk (1) h(l), where h(t) is the known and fixed to- tal ice shelf thickness. At each time step, the thickness of the layers are simultaneously updated through an advection equation,
Th
(hgu),
Øt
where u(!) is the known velocity profile of the ice shelf which is fixed across simulations. Additional layers are added at the top of the ice shelf and removed from the bottom according to a(!), b(t) accordingly. The noise model approximates the observation noise of the radar measurement given an assumed density profile of the ice shelf. At the final timestep I", the layer that most closely matches
21

---

the ground truth observation x" according to the - is selected as the simulator output. Full details are described in Moss et al. 39
Prior The prior is defined over the accumulation rate parameter ä()), and is motivated by physical observations at Ekström ice shelf. Prior samples are drawn using
a(I) a + Polts
(6)
where Hoff (..5,. ose ~ U( 0.3)), and á is drawn from Gaussian Process with unit-variance, zero-mean Matérn-v kernel of lengthscale 2500 and v = 2.5.
Evaluation Parameters For SBC EoD, as well as the predictive MSE on synthetic test simulations, we use 100 test observations and sample 10 posterior samples for each observations and for each method. The real test data consists of one field observation (shown in Fig. (oh-b), and the posterior predictive was estimated using 1000 posterior samples.
S6
Experimental details
$6.1 Linear Gaussian
For all the baseline methods, we train the networks using an Adam optimizer with a learning rate of 0.0001, and batch size of 200. For NPE/EMPE (spectral), we use 50 modes, leading to 100 parameters leam, and a pad width of 20 for the spectral preprocessing (Appendix (S4T)) For NPE (spectral) the density estimator is a Neural Spline Flow (NSF) with 2 residual blocks with 50 hidden dimensions each, 5 transforms, with RELU activations. For FMPE (spectral), we use an MLP with linear layers with 64 hidden dimensions to estimate the flow, with ELU activations. In both cases, we embed the 1000 dimensional observation into a -0-dimensi vector using an MI.P with 2 layers and 50 hidden units and RELU activations. For FMPE (raw), we use an MLP with 5 layers and 64 hidden features, with ELU activations.
For FNOPE and FNOPE (fix) we use 50 Fourier modes for the FNO blocks. We use 5 FNO blocks with 16 channels, while the context is embedded into 8 channels. We train for a maximum of 500 epochs with an early patience of 50, We used a training batch size of 512 and learning rate of 0.001. For FNOPE, we use 4 channels each for the positional and time embeddings and the target gridsize Nas = 256 (for FNOPE (fix), no positional embedding is included). All nonlincarities are GELUs.
S6.2 SIRD
For FNOPE we use 32 Fourier modes for the FNO blocks. We use 5 FNO blocks with 16 channels, while the context is embedded into 8 channels. We train for a maximum of 1000 epochs with an early patience of 50. We use a training batch size of 200 and a learning rate of 0.001. The discretization positions and flow times are embedded into 4 channel dimensions each, and the target gridsize N& = 40. This experiment additionally included vector-ve vector-valued parameters IR.2. These are embedded into 16-dimensional vector using a 1-layer MI.P with hidden dimension of 64. The flow for the vector-valued parameters is estimated using an 1 -llayer MLP with a hidden dimension of 64. The spectral decomposition of the output of the FNO blocks, well as the spectral decomposition of the observation, are embedded into a 32-dimensional vector and concatenated to the input of the MLP. All nonlinearities were GELUs.
The training hyperparameters for simformer are described in S4.6
S6.3 Darcy Flow
For all the baseline methods, we train the network using an Adam optimizer with a learning rate of 0.0001, and batch size of 200. For NPE/FMPE (spectral), we use 16 modes, leading to 2 x 162 = 512 parameters to learn, and pad width of 20 in each dimension for the spectral preprocessing (Appendix [S4.I). For NPE (spectral) the density estimator is NSF with residual blocks with 50 hidden dimensions each, 5 transforms, with RELU activations. For FMPE (spectral), we use an MI. with 8 layers with 256 hidden dimensions to estimate the flow, with ELU activations. For FMPE (raw),
an MLP with 8 layers and 256 hidden features, with ELU activations. All
22

---

baseline methods embed the observation with a CNN embedding net into 100 100-dimensional vector using 4 convolutional layers with kernel size followed by max pooling of kemnel size 2, followed by a 4-layer MLP with 100 hidden units, with RELU nonlinearities throughout.
For FNOPE and FNOPE (fix) we use 32 Fourier modes for the FNO blocks. The network is made of 5 FNO blocks with 32 channels, while the context is embedded into 32 channels. We train for maximum of 300 epochs with an early patience of 50, We use training batch size of 200 and learning rate of 0.0005. For FNOPE, we set the target gridsize N&s 2048. The architecture includes 8 channels for positional embedding, and channels for time embedding. All nonlinearities are GELUs.
S6.4 Mass balance rates of Antarctic Ice Shelves
For all the baseline methods, we train the network using an Adam optimizer with a learning rate of 0.0001, and batch size of 200. For NPE/FMPE (spectral), we use 10 modes, leading to 20 parameters to learn, and a pad width of 20 for the spectral preprocessing (Appendix (S4.T). For NPE (spectral) the density estimator a NSF with 2 residual blocks with 50 hidden dimensions each, transforms, with RELU activations. For FMPE (spectral), we use an MLP with 5 linear layers with 64 hidden dimensions to estimate the flow, with ELU activations. For FMPE (raw), we use an MLP with 5 layers and 64 hidden features, with ELU activations. For all baseline methods, we used the same embedding as Moss al. (39 which was a CNN embedding the 441- 441-dimensional observation into a 50-dimensional vector using 2 convolutional layers with kernel size 5 followed by max pooling of kernel size 2, followed by a 2-layer MLP with 50 hidden units, with RELU nonlinearities throughout. The configuration of NPE (raw) is described in Appendix S4.3
For FNOPE we 10 Fourier modes for the FNO blocks. The network is made of 5 FNO blocks with 16 channels, while the context is embedded into 8 channels. We train for a maximum of 1000 epochs with an early patience of 50. We use training batch size of 200 and a learning rate of 0.001. We do not include the data augmentation procedure for this experiment, as the discretizations of both observations and parameters was fixed the setting of (39]. We still include positional embedding due to the parameter and observations being discretized differently to one another: the architecture included 4 channels for positional embedding, and 4 channels for time embedding. All nonlinearities are GELUs.
For the experiments 500 gridpoints (Fig. S5
we use the same hyperparameters.
S7
Ablation experiments
$7.1 Linear Gaussian
To investigate the influence of different hyperparameters on the performance of FNOPE, several ablation experiments. First, we studied the performance of FNOPE in the deliberately insufficient setting where the prior distribution over the parameter contains higher-frequenc, modes than what is modeled by the FNO blocks. To this end, we changed the lengthscale of the prior distribution for the Linear Gaussian task (Sec. (4.2) by factor of 10 to 0.005, resulting in higher frequency components for the parameter 0. We then trained FNOPE with a varying number of spectral modes in the FNO blocks. With lower number of modes, the performance of FNOPE degrades, and the posterior samples clearly miss the high frequency components present in the ground truth observations (Fig. (S...). Second, we varied the number of unmasked points in the FNOPE training procedure (Sec. B.I.. While the SWD decreases with the number of unmasked points, it saturates at 512 unmasked points, which is approximately half of the observed data (Fig. (T)).. We then consider changing the lengthscale heuristic used to define the base distribution for FNOPE (Sec. 3.1.. by detining/ / Lu/M for the number of modes AM and some lengthscale scaling factor Lo- In the extreme case Lp = 0, this corresponds to the base distribution sampling uncorrelated white noise (WN). The lengthscale heuristic used in our work corresponds approximately to Lo = 4/TT 1.27. In contrast to the other hyperparameter ablation experiments, varying the lengthscale scaling factor does not seem to impact the performance of FNOPE considerably for this task. FNOPE performs well over wide range of lengthscale scaling factors (Fig. (S.H) Finally, we consider version of FNOPE which keeps the masking scheme as described in Sec. 3.T] but without adding positional noise to the remaining positions, which effectively only sees positions on the simulation grid during training
23

---

(FNOPE (no jitter)). On this task, we evaluate on an equispaced grid, and we see that FNOPE (no jitter) performs similarly FNOPE (Fig. (STp). Therefore, the performance of FNOPE does not degrade from the inclusion of positional noise in this task.
b
modes
10
SWD
Power
128
256
10
ground Iruth
8
16
128
256
100
# modes
Frequency
d
0.15
0.15
0.15
SWD
0.00
0.00
0.00
128
1000
WN
0.1
1.0
10.0
10
10
10
10"
# unmasked points
Lengthscale Scaling Factor
simulations
FNOPE
FNOPE
FNOPE jitter)
Figure S1: Linear Gaussian ablation experiments. (a) Performance for 10 training samples in terms of SWD for the Linear Gaussian experiments with varying number of Fourier modes in the FNO block. Note that we changed the lengthscale of the simulator prior by factor of 10 to 0.005 (Appendix S5 (b) Power analysis of FNOPE (fix) samples and ground truth for different number of used modes the FNO block. (e) Performance terms of SWD for different numbers of masked points in FNOPE. (d) Influence of the lengthscale of the noise process the SWD. "WN" corresponds to white noise (an uncorrelated Gaussian distribution). (e) Same as Fig. with version of FNOPE in which we omit the adding of positional noise (FNOPE (no jitter)..
$7.2 SIRD
First, compare the performance of FNOPE to Simformer the SIRD task when using observations 20 (instead of 40) randomly sampled times (Fig. S2). The performance of FNOPE slightly degrades, while Simformer still performs robustly. Second, adapt the lengthscale heuristic used to define the base distribution for FNOPE (Sec. (Sec.3..I) (.1). by defining = Lo/ Lo/M /M for the number of modes M and some lengthscale scaling factor Lo. The lengthscale heuristic used in our work corresponds approximately to Lo = 4/π ze 1.27. We observe that FNOPE performs robustly for a wide range of lengthscale scaling factors.
S7.3 Darey Flow
In the Darcy task, we ablate the number of modes used in the FNO block for FNOPE. We see that while the MSE of posterior predictive simulation and posterior calibration as measured by the SBC EoD of marginal distributions is not strongly affected by the number of modes used, the posterior log-proo probability per pixel increases with the increased model capacity (Fig. (S4h). This is expected, as more modes allow FNOPE to estimate a more constrained posterior distribution, leading to higher posterior densities. Second, we adapt the lengthscale heuristic used to define the base distribution for FNOPE (.1), by defining = Lo/ M for the number of modes M and some lengthscale scaling factor T.g The lengthscale heuristic used our work corresponds approximately to Lo 4/π n ze 1.27. We see that increasing the lengthscale scaling factor improves the calibration

---

Contact rate
1.0
Death rate
0.4
0.2
0.0
20
40
0.2
0.4
Time [days]
Recovery rate
x
0.2
FNOPE Simformer
Infected
Pred. MSE
Data
0.0
Recovered
0.0
0.2
lower bound
0.4
SBC EoD
Deceased
Mxx
0.0
0.0
20
10"
10"
Time [days]
simulations
Figure S2: SIRD model on 20 conditioning points. (a) Posterior conditioned on 20 time points. left: Posterior (mean + std.) of time-varying parameter and ground truth parameters (dashed). right: Two dimensional posterior of vector-valued parameters and ground truth parameters (dot). (b) Posterior predictive (mean + std.) of infected, recovered and deceased populations with observations marked. (c) upper: MSE of posterior predictive samples to observations. lower: Simulation- based calibration error of diagonal (SBC EoD). 'Lower bound' refers to the SBC EoD for uniformly sampled posterior ranks (details in Appendix S2) See Fig. for the results with 40 conditioning points.
x10
0.2
8
Pred. MSE
SBC EoD
0.1
0.0
1.0
10.0
0.1
10.0
Lengthscale scaling factor
Lengthscale scaling factor
Figure S3: SIRD ablation experiments. Influence of the lengthscale of FNOPE's base distribution (Sec. (.I1) on posterior quality terms of MSE of posterior predictive simulations and Simulation- based calibration error of diagonal (SBC EoD).
of the posterior learned with FNOPE. We observe that FNOPE achieves its best performance in terms of posterior log-probayility when the base distribution lengthscale scaling factor is set to a value around our lengthscale heuristic (Fig. S4p).
$7.4 Mass balance rates of Antarctic Ice Shelves
repeat the Antarctic ice mass balance experiment (Sec. 4.5p. without downsampling the parameter space from simulation to inference, i.e. inferring the full 100-dimmnsiionl posterior distribution over the mass balance parameters. Overall, we observed that FNOPE maintains its performance on this higher-dimensional problem (Fig. S5 For low simulation budgets, the performance of
25

---

a
x10
4
0.2
Posteriar log prob
1.5
Pred. MSE
SBC SBC EoD
0.1
1.2
1.0
0
0.0
# modes
# modes
# modes
b
0.2
Posterior log prob
Pred. MSE
SBC EoD
2.0
2
0.1
1.0
0
0.0
0.1
1.0
10.0
0.1
1.0
10.0
0.1
1.0
10.0
Lengthscale scaling factor
Lengthscale scaling factor
Lengthscale scaling factor
Figure S4: Darey ablation experiments (a) Influence of number of used modes in the FNO block different performance measures. The original experiments used 32 modes. Measures are the same as in Fig. -p.d (b) Influence of the noise length scale on different performance measures. The original experiments used a lengthscale of zs 1.27. Measures are the same as in Fig. 55-d.
FNOPE exceeds the other methods in terms of predictive MSE for both synthetic and real observations. As expected, spectral baseline methods significantly outperform the other baselines, especially at low simulation budgets.
0.2
Pred. MSE (synth. obs.)
lower bound
0.6
SBC EDD
Pred. MSE (real obs.
0.6
0.0
0.0
0.0
10
10"
10
10
10"
10
10
10"
10
# simulations
# simulations
# simulations
FNOPE
NPE (spectral)
FMPE (spectral)
NPE (raw)
Figure S5: Mass balance rates of ice shelves. Inference of mass balance rates where the parameter discretization uses 500 gridpoints (observation discretization remains unchanged). Performance measures on test simulations and the real observation, where NPE (raw) refers to the method used in Moss al. B3] Results for FMPE (raw) omitted as this baseline was not able to always produce samples within the prior bounds across all test observations. See Fig. for the results based on 50 grid points.
26

---

S8
Additional Results
S8.1
Additional Darcy results
01
03
6
0
Ground truth
FNOPE (fx)
FNOPE
NPE (spectral)
FMPE (spectral)
FMPE (raw)
Figure S6: Posterior samples Darcy flow experiment. Additional posterior samples for the Darcy flow experiment for six distinct observations simulated from ground truth parameter 0,.
27

---

81
02
0g
1.8
Ground truth
1.6
FNOPE (fb))
1.4
1.2
FNOPE
1.0
NPE (speciral)
0.8
FIMEE (spectral)
0.6
FIME (raw)
0.4
0.2
Figure S7: Posterior means for Darcy flow experiment. Posterior means (based on 100 samples) for the Darcy flow experiment for six distinct observations simulated from ground truth parameter 0,.
28

---

01
02
93
04
0g
0
0.6
FNOPE (fox)
0.5
FNOPE
0.4
NPE (spectral)
0.3
FMPE (speciral)
0.2
0.1
FMPE (raw)
0.0
Figure S8: Posterior standard deviation for Darey flow experiment. Posterior standard deviations (based on 100 samples) for the Darey flow experiment for distinct observations simulated from ground truth parameter 0,.
29

---

S8.2 Comparison to invertible Fourier Neural Operator
As an additional comparison, we apply the work of Long al. [50] and train an invertible Fourier Neural Operator (iFNO) to solve the Darcy Flow inverse problem, using the same training data and simulation budgets. We train the iFNO with the same settings as described in Long al. [50] for the D-CURV experiment, summarized below. We observe that while iFNO achieves comparable performance to FNOPE in terms its predictive MSE, the posterior distribution not calibrated as measured by the SBC EoD (Fig. (S9..) and essentially collapsed to point estimate. The SBC FoD value measured for iFNO (around 0.25) is consistent with this point estimate, because the one-dimensional marginal of point mass distribution either always overestimates underestimates the ground truth value. Hence, recalling the definition of SBC BoD (Sec. S2.2), the ground truth will have rank Tij. mass distribution is equally likely to underestimate or overestimate the ground truth, the cumulative distribution of the ranks will be given by (a) 0.5 for all significance levels (0, 1). Given the definition of the SBC EoD, point mass distribution results in a SBC EoD value of
raj K post + 1. Supposing that over different observations rz, the point
J CDF - ad = 0.25
for each dimension i. This overconfidence is also reflected in the standard deviations the estimated distributions (Fig. 59b), which show that iFNO essentially estimates point mass for this task.
iFNO hy hyperparameters We trained iFNO with FNO blocks and Fourier modes. The number of training epochs for the invertible Fourier blocks training 100. The architecture of the 3-VAE was the same as in Long et al. [50] with rank 32, as well the value 3 10-6 We used minibatch size of 10 for joint training and 20 for the VAE and iFNO pretraining.
to 100, for the B- 3-VAE 1000, and for joint
X10
Ground Truth
Posterior Mean
|Mean GTi
Posterior
1.30
FNOPE
Pred. MSE
1.28
1.26
FNOPE (fhx)
b
SBC EoD
IFNO
0.00
10
10"
simulations
lower
bound
FNOPE
FNOPE (fix)
Figure S9: Darcy Flow, comparison of FNOPE to iFNO [5]. (a) MSE of posterior predictives to ground truth observation (zoomed relative to Fig. 5). (b) Simulation- calibration Error of
Diagonal (SBC EoD) for different training budgets. (c) Ground truth 6, Posterior means, pixelwise error of means relative to ground truth, and posterior standard deviations. The color bars of the means match Fig. S7 and for errors and standard deviations they match Fig. S8
30

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We state the contributions of our work: We introduce FNOPE, simulation- based inference method for inferring function-valuee valued parameters. We demonstrate on synthetic and real world tasks its ability generalize new and non-uniform discretizations the parameter and observation domains, as well as its scalability to large numbers of parameters.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We explicitly mark discussion the limitations of our approach, including limitations of the low-frequency assumption Fourier Neural Operators, and limitations in higher-dimensional parameter domains than those considered in our experiments.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specificattin, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
31

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Our work includes a lengthscale selection heuristic for the FNOPE noise distribution, for which we provide derivation for in Appendix S3.2 and appropriately refer the reader to this derivation in the main text.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide all experimental details appropriately labelled subsections the appendix, including simulator configurations, model hyperparameters for each experiment, and details on the evaluation metrics. In addition, we provide our code to reproduce all experiments in the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
32

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We provide our code and document how run the experiments included in our paper. These include the simulators, as well as the random seeds used to generate the training data from the simulators.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
for more details.
While we encourage the release code and data, we understand that this might not be possible, so No is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips t/tpps.ccppublll.// wttpc///wwwcgggddeeeeeeeeeeeeeee
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We report all experimental details, including data splits, model hyperparame- ters, and optimizers in Appendix (S6) We provide the random seeds used to run training and evaluation scripts to allow full reproducibility our experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
33

---

We report the standard enrors in our results over 3 independent runs for each experiment. The size of the test set is provided for each experiment in the appendix and in the configuration files provided with the code.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative eтror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the specifications of our compute nodes well further informa- tion in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We confirm that this work conforms to the NeurIPS Code of Ethies in all
respects.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
34

---

10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
Our work presents methodological development in performing simulation- based inference, with goal enabling scalable and flexible inference in various scientific domains. We do not target applications with direct societal impact, and do not foresee potential for misuse.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work only synthetic publicly available data. Our work is method- ological and empirical, and therefore we do not see risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
35

---

Answer: [Yes]
Justification: We credit all external asserts in our work, including datasets. We reproduce a figure component from existing published work which is correctly attributed and licensed, with some details omitted for the review stage to preserve anonymity.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We provide the code for our method with appropriate documentation. No other assets are produced in this work.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This work does not include crowdsourcing or research with human experts.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
36

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No human subjects were involved in this study, nor data involving human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: LLMs were not used in the development, implementation, or evaluation of the proposed method according to the NeurlPS LLM policy.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non sta-standard components.
Please refer to our LLM policy (https :ettps://neuue //terr/ps.....p h//Coofffeennee//// for what should or should not be described.
37


Reviews:
[{"review_id": "9eGuHEKEwJ", "reviewer": "Reviewer_pNZ3", "summary": "The paper proposes to combine flow matching and Fourier neural operators to extend simulation-based inference to model posteriors over function-valued parameters, e.g., parameters that vary in time and/or space. To do this, the authors use Fourier neural operators to model the flow field and further adapt them to handle non-uniform input data using a non-uniform fast Fourier transform. The full model is able to handle both function-valued and vector-valued parameters.\n\nThe authors test the proposed model on several synthetic benchmark tasks, as well as a real-world example involving glacial flow. The method is generally on par with or outperforms baselines, requiring orders of magnitude fewer simulations to achieve the same accuracy, and having more realistic posterior samples.", "questions": "1. Did you measure the training and evaluation (wall-clock) times for each of the methods you consider? I'm curious to see how FNOPE compares to baselines. Also, how stable is the training?\n2. The hyperparameters of the model seem to have been chosen somewhat heuristically. Did you investigate other choices? For instance, did you try other kernels for the Gaussian process prior, e.g., Matérn?\n3. Further to the question above, you choose to fix the length scale of the Gaussian process kernel to a particular value, but would it not be possible to optimise it? Is there a reason not to do so?\n4. Simformer seems like a strong baseline. What is the reason for not comparing against it in all of your experiments?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. The paper presents an interesting solution to an important problem in simulation-based inference by combining two recent and promising techniques, flow matching and Fourier neural operators. \n2. The model appears to work well in practice and requires many fewer simulations than the baselines, although there still are some computational concerns (see weaknesses).\n3. The paper is well-written and should be of broad interest to the NeurIPS community.\n\n**Weaknesses**\n1. The main weakness of the paper is the computational complexity of the proposed model, which scales exponentially with the parameter dimensionality, both in the number of parameter discretisation points and the number of Fourier modes considered. So while the model seems like an interesting approach, it is unlikely to be useful in many scenarios in its current form.\n2. The model appears to be powerful in the sense that it generally seems to require only a fraction of the number of simulations that the baselines do, but it is unclear how big the advantage is in actual compute since training and evaluation times are not reported.\n3. The hyperparameters of the model seem to have been chosen heuristically and not compared to other choices.", "was_revised": true, "final_justification": "The authors addressed my questions and concerns, and provided that the submission is extended with the new experiments, I think it should be accepted.", "rebuttal": "We thank the Reviewer for their positive and constructive feedback. We note that the other Reviewers also highlighted the applicability of FNOPE to non-uniform discretizations (all Reviewers), and were positive about the simulation-efficiency (Reviewers 5bMC, u8xK), and the clarity of our presentation (Reviewer 5bMC).\n\nTo address the Reviewers’ concerns, we clarify the computational complexity which was incorrectly stated to be exponential in the number of discretization points, and report compute times, which are all in a reasonable range. Additionally, we ran extensive hyperparameter experiments to show the robustness of FNOPE across different settings. \n\nWe provide tables as supporting information to our answers. Where appropriate, this information will be expressed in figures in our revised manuscript.\n\n## Computational complexity and compute time\n\nIt is not the case that the computational complexity “scales exponentially … in the number of discretization points”. Instead, in the FNO architecture with $M$ modes and $N$ discretization points, the computational complexity is only $O(N\\log N)$ (if using the FFT), and $O(NM)$ (if using the NUDFT). This is one of the main advantages of the FNO architecture—since higher order frequency components are discarded, the main computational complexity of the neural network is independent of the number of points in the discretization, allowing it to be efficiently applied to coarse and fine discretizations alike. It is true that the computational complexity of the FNO scales as $O(M^D)$ in the spatial dimension $D$. However, the focus of this work is on spatial-temporal problems $D=1, 2, 3, 4$ , which cover many scientific applications, and so this exponential complexity is not a critical limitation of the FNO architecture. \n\nWe believe the misunderstanding regarding computational complexity could stem from the statement in our limitations section. We apologise for this confusion, and we will clarify the distinction between the dimensionality of the spatial domains and the number of discretization points in our revisions.\n\nAdditionally, we provide a full breakdown of our experiments in terms of training time, sampling time, and network sizes of FNOPE and the baseline methods (Table R1.1). Even for the largest inference task considered (Darcy Flow), training was reasonably fast at around 1 hour on a single GPU. Sampling times are larger for FNOPE on the Darcy Flow task compared to the baseline methods, mostly due to the required solution of solving high-dimensional ordinary differential equations. However, even in this case, sampling times are sufficiently fast that this difference is negligible for our applications. For all other experiments, FNOPE is faster than or equivalent to the other methods in terms of training and evaluation times.\n\nWe hope that the clarification regarding computational complexities, as well as the empirical validation of the compute times, are enough to alleviate this Reviewer’s main concern regarding the applicability of our approach.\n\nTable R1.1: Network sizes and training times for the main experiments. We report the mean over 3 runs with a training budget of 10k simulations. We omit showing any variation for more readability and since the results are robust across random seeds. Linear Gaussian and SIRD tasks were run on a 2080ti GPU, Darcy flow on an A100 GPU, and Antarctic ice on CPU.\n| Task | Metric | FNOPE | FNOPE (fix) | NPE (spectral) | FMPE (raw) | FMPE (raw) | NPE (raw) | Simformer |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Linear Gaussian | # parameters | 109,797 | 108,581 | 566,590 | 299,310 | 86,302 | - | - |\n| Linear Gaussian | training (Tot.) [min] | 3.12 | 1.98 | 6.77 | 5.14 | 2.46 | - | - |\n| Linear Gaussian | training (/epoch) [s] | 0.99 | 0.87 | 2.17 | 0.31 | 0.24 | - | - |\n| Linear Gaussian | sampling time (/sample) [ms] | 2.16 | 1.06 | 0.05 | 0.20 | 0.24 | - | - |\n|  |  |  |  |  |  |  |  |  |\n| SIRD | # parameters | 116,581 | - | - | - | - | - | 285,579 |\n| SIRD | training (Tot.) [min] | 6.95 | - | - | - | - | - | 47.17 |\n| SIRD | training (/epoch) [s] | 1.19 | - | - | - | - | - | 9.43 |\n| SIRD | sampling time (/sample) [ms] | 0.22 | - | - | - | - | - | 0.22 |\n|  |  |  |  |  |  |  |  |  |\n| Darcy flow | # parameters | 11,555,673 | 11,552,521 | 3,537,566 | 9,180,033 | 897,664 | - | - |\n| Darcy flow | training (Tot.) [min] | 72.24 | 41.83 | 20.52 | 12.20 | 10.95 | - | - |\n| Darcy flow | training (/epoch) [s] | 20.30 | 26.23 | 2.82 | 0.73 | 0.66 | - | - |\n| Darcy flow | sampling time (/sample) [ms] | 279.55 | 35.20 | 0.21 | 2.70 | 1.95 | - | - |\n|  |  |  |  |  |  |  |  |  |\n| Antarctic ice | # parameters | 25,317 | - | 235,608 | 96,210 | 92,340 | 360,933 | - |\n| Antarctic ice | training (Tot.) [min] | 47.35 | - | 19.69 | 33.53 | 31.63 | 6.80 | - |\n| Antarctic ice | training (/epoch) [s] | 9.82 | - | 4.21 | 2.02 | 1.92 | 5.64 | - |\n| Antarctic ice | sampling time (/sample) [ms] | 22.79 | - | 1.50 | 23.85 | 16.61 | 1.53 | - |\n\n\n## Simulation efficiency\n\nAs the Reviewer pointed out, one strength of FNOPE is that it requires fewer simulations than existing methods, which could enable inference on computational expensive simulators. To emphasise this strength, we conducted experiments using only 100 training simulations. For the Linear Gaussian task the SWD is considerably smaller for FNOPE and FNOPE (fix) than for the baseline methods with 100 simulations (Table R1.2). Similarly, on the Antarctic ice task, FNOPE is more robust than the baseline methods (Table R1.3), even though the performance degrades for all methods, .\n\nTable R1.2: SWD means and standard errors of the Linear Gaussian experiment for different numbers of training simulations.\n| Method ↓ / # simulations →   | $10^2$              | $10^3$              | $10^4$              | $10^5$              |\n|----------------------------------|-------------------|-------------------|-------------------|-------------------|\n| FNOPE                            | 0.118 ± 0.002   | 0.144 ± 0.002   | 0.089 ± 0.001 | 0.086 ± 0.000 |\n| FNOPE (fix)                      | 0.042 ± 0.001 | 0.026 ± 0.000 | 0.016 ± 0.000 | 0.010 ± 0.000 |\n| NPE (spectral)                     | 1.237 ± 0.008   | 1.084 ± 0.006   | 0.683 ± 0.008  | 0.034 ± 0.001 |\n| FMPE (raw)                         | 1.053 ± 0.005   | 0.944 ± 0.002  | 0.884 ± 0.000 | 0.874 ± 0.000 |\n| FMPE (spectral)                    | 1.306 ± 0.008   | 1.102 ± 0.008   | 0.821 ± 0.008   | 0.172 ± 0.008 |\n\nTable R1.3: Predictive MSE means and standard errors of the Antarctic ice experiment for different numbers of training simulations.\n| Method ↓ / # simulations → | $10^2$ | $10^3$ | $10^4$ | $10^5$ |\n|---|---|---|---|---|\n| FNOPE | 0.108 ± 0.002 | 0.069 ± 0.002 | 0.054 ± 0.001 | 0.046 ± 0.001 |\n| NPE (spectral) | 0.355 ± 0.006 | 0.180 ± 0.003 | 0.071 ± 0.002 | 0.062 ± 0.001 |\n| FMPE (raw) | 0.244 ± 0.003 | 0.238 ± 0.003 | 0.074 ± 0.001 | 0.058 ± 0.002 |\n| NPE (raw) | 0.246 ± 0.003 | 0.287 ± 0.004 | 0.093 ± 0.002 | 0.045 ± 0.001 |\n| FMPE (spectral) | 0.365 ± 0.005 | 0.165 ± 0.004 | 0.091 ± 0.002 | 0.075 ± 0.002 |\n\n\n\n## Extensive Hyperparameter Ablations\n\nOverall, FNOPE performs robustly over a wide range of different hyperparameters (see response to Reviewer u8xK for more details and result tables). First, we performed an ablation study on the impact of the Gaussian process kernel lengthscale on the performance for the Linear Gaussian and SIRD experiments (Tables R3.3, R3.4), which shows that FNOPE performs well for different lengthscales, even in the limit of white noise. Second, we investigated the dependence on the number of Fourier modes used for the Linear Gaussian and the Darcy flow experiment (Table R3.1, R3.2): FNOPE’s performance increases with the number of modes, but saturates for a higher number of modes. Third, we investigated the effect of the number of unmasked training data points on the performance of our method (Table R3.5). Here, we focus on the Linear Gaussian experiment. FNOPE performs well for all numbers of unmasked training points, but the performance increases slightly with the number of unmasked training points and exhibits saturating behavior with higher numbers of unmasked points. Figures for these ablation studies will be included in the Appendix of the paper. \n\n\n## Different base distributions (e.g. Matérn kernel)\n\nUsually, the base distribution in flow-matching models is the unit Gaussian, $\\mathcal{N}(0,I)$. Our main motivation in introducing the Gaussian Process base distribution is to produce base distribution samples which satisfy the smoothness assumptions of our FNO architecture, while still providing a distribution which can be efficiently sampled and evaluated. Thus, the main design choice in the base distribution is the power spectrum of the samples, which can be controlled via the lengthscale (see discussion above on hyperparameter ablations). A Matérn kernel could also present a valid choice of kernel, but we do not expect it to lead to different results.\n\n## Simformer as baseline\nThe computational cost of Simformer is at least $O(N^2)$ in the number of parameters, which is infeasible for a very large number of points as in the Darcy Flow Task ($\\sim 16$k).\n\nOur goal in comparing FNOPE to Simformer on the SIRD task is to demonstrate that FNOPE is comparable to an established approach at the challenging task of inferring parameters on non-uniform positions, as well as inferring additional vector-valued parameters. More specifically, for the SIRD experiment, we infer the parameters on much sparser grids than the rest of our experiments (40 gridpoints, compared to e.g. 1000 for the Linear Gaussian). However, FNOPE can be scaled to finer discretizations in contrast to Simformer."}, {"review_id": "KvQ0dmunRw", "reviewer": "Reviewer_5bMC", "summary": "The authors propose a method for performing simulation-based inference (SBI) of function-valued latent variables. The method is a combination of flow-matching posterior estimation (FMPE) and Fourier neural operators (FNOs): FMPE provides the objective function, while FNOs provide a means of parameterizing function-valued outputs given $x$ (plus its discretization) as input. A key innovation of the authors is their explicit conditioning on the (variable) discretization positions $l^\\theta, l^x$ in both the generative (simulation) process and the inference procedure. This formulation allows for flexible choice of discretization grid, in particular the use of *any* grid desired at inference time once the model is fitted.", "questions": "Let me start by providing more detail on the marginalization point discussed above. Imagine a high-dimensional latent variable $\\theta \\in \\mathbb{R}^{100}$, for example. In NPE, a common approach to avoid high-dimensional difficulties is to use a mean-field factorization, i.e. parameterize $q(\\theta \\mid x)$  as $q(\\theta_1 \\mid x) q(\\theta_2 \\mid x) \\cdots q(\\theta_{100} \\mid x)$. This yields 100 univariate inference problems to solve. Certainly, fitting a hundred different networks is less than ideal, but it can be done. What is lost in this approach is posterior correlations between the latent variables, but in some settings this may be acceptable.\n\nFor the type of data in this work, losing correlations is unacceptable: if $\\theta$ represents evaluations of a smooth function  on a grid, $\\theta_{76}$ and $\\theta_{77}$ are highly correlated, for example; they should not deviate from each other much if the grid being evaluated on is fine enough. A discussion similar to the above I think might be a useful addition to the paper, as function-valued latent variables make a compelling case to *not* use the mean-field approach with its marginalization properties. You may want to read or cite Forward Amortized Variational Inference (Ambrogioni et al.) or works that cite this one for more on marginalization.\n\n\nQuestions:\n- The proposed method uses the FMPE objective. I wonder if you tried (or if it possible to formulate) a variant based on the NPE objective. Does this exist in other related work? Presumably it would look something like \n$$\n-\\mathbb{E}_{l^\\theta, \\theta, l^x, x} \\log q^\\phi(\\theta \\mid x, l^x, l^\\theta).\n$$\n- The section on “inferring additional parameters” seems less useful for the same marginalization argument presented above. If a user has to perform inference on both a parameter $\\eta$ and a function valued latent $\\theta$, I suspect in practice one might fit the mean-field variational posterior $q(\\theta, \\eta \\mid x) = q(\\theta \\mid x)q(\\eta \\mid x)$ with the first term fit by FNOPE and the second by a standard NPE approach. Again, unless posterior correlations between $\\eta, \\theta$ are desired, fitting a joint posterior is unnecessary.\n- The lack of discussion on marginalization clashes with the fact that SBC on the marginals is used as an evaluation metric (line 183). If the quality of the marginals is what’s being evaluated, modeling posterior correlations seems unnecessary. Adding some type of NPE competitor based on a mean-field parameterization would help resolve this point. \n- Can you explain the MSE metric in more detail? What two quantities is the MSE computed between? For a fixed evaluation point $i$, is it between the ground truth draw $\\theta_i$ and a single posterior sample at $i$? Or something else?\n- In the first experiment, can you explain intuitively why FNOPE (fix) outperforms FNOPE? \n- It’s difficult to gauge a take-away message from the second experiment given that FNOPE does not outperform the Simformer. \n- In the third experiment, NPE (spectral) seems to achieve the same MSE as the proposed methods. Yet, the images in panel (a) for NPE look much poorer relative to the ground truth. Why is this?\n- In the third experiment, is NPE (spectral) present in panel d), or not? If not, why?\n- In the third experiment, it’s stated that FNOPE uses 32 Fourier modes, while NPE/FMPE only 16 (line 248). Should these be equal for a fair comparison?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n- The exposition is clear and overall the paper is well written and easily understandable.\n- The use case for practitioners is an important one; NPE and other simulation-based methods are increasingly popular, and performing inference on increasingly complex quantities like function-valued latents  is a useful advancement (although I’m less familiar with other related work in the function-valued SBI space that may tackle similar problems).\n- The sample efficiency point is compelling; some simulators are expensive to sample, so doing well with fewer samples is necessary.\n\nWeaknesses:\n- The paper makes claims about the difficulties of high-dimensional inference problems for SBI without talking about the marginalization properties of NPE – this is an important point that I think needs to be addressed (more on this in next section).\n- The sample efficiency results are empirical in nature, rather than a theoretical guarantee. In experimental settings, it seems that NPE (spectral) can perform as well as the proposed method with enough simulations (10^4 or 10^5). \n- The experimental results do not show the degree of improvement over NPE or related methods that one might expect relative to the claims. Also, some of the metrics for evaluation of the experiments raise questions about marginalization as well. \n- The method is somewhat combinatorial, as both FMPE and FNOs are established approaches in the literature.", "was_revised": true, "final_justification": "See my more detailed comments below.", "rebuttal": "We thank the Reviewer for their positive and thoughtful feedback. We want to highlight that the other Reviewers agree that our work solves a relevant problem (pNZ3), that the sample efficiency is a particular strength of our approach (u8xK), and noted the clarity of our presentation (pNZ3, u8xK).\n\nTo address the Reviewer’s concerns, we provide a discussion on a marginalized approach as suggested by the Reviewer, which we will also add to the revised manuscript. We also provide some clarifications to answer the Reviewer’s remaining questions.\n\n## Marginalization\nWe agree with the Reviewer, that we could in principle learn a marginalized posterior in SBI. However, this is not the setting we are interested in this work. As the Reviewer mentioned correctly, a marginalized posterior is “less than ideal”, especially for function-valued parameters which are (by definition) highly correlated. But even beyond that, a lot of “low” dimensional scientific problems show highly correlated posterior distributions, and this is often what scientists care about. In neuroscience, for example, Gonçalves et al. [1] showed that the parameters of a model of the pyloric network are highly correlated, and posterior draws from the marginalized distribution fail to give reasonable simulation results. Similar properties are observed in gravitational wave detection, where correlations in the posterior are of great interest and e.g. sky position cannot be inferred with marginal inference [2]. However, we are aware that for specific problems the inference of marginal distributions might be sufficient [3]. In such cases other SBI methods might be better suited. The SIRD model pinpoints that even the low dimensional parameters $\\eta$ can have strong correlations (Fig. 4a, right) and a mean field approach would suffer from these correlations. Additionally, we want to highlight that a marginalized setup would not easily allow flexible conditioning on arbitrary spatial/temporal observations as is the case for FNOPE. \n\nIn terms of performance, we indeed measure the **calibration** of the inferred posteriors by SBC EoD on the marginals, which is a standard metric in the SBI literature [4]. Similar calibration measures for high dimensional distributions are still an active field of research. However, we want to highlight that all other metrics, e.g. SWD (Linear Gaussian) and posterior log-probability (Darcy flow) are taking the joint posterior into account. Similarly, the MSE metric (see clarification below) is based on posterior predictive samples from the joint posterior and is therefore indirectly also sensitive to correlation structures. Samples from a mean-field approximation often do not lead to predictive simulations which match the observation, as simulators often contain complicated, nonlinear interactions between the parameters. Consequently, our evaluation is **not** limited to the quality of the posterior marginals. \n\nWe thank the Reviewer for bringing the work of Ambrogioni et al. [5] to our attention. Indeed, there are relevant related tasks for which joint posterior estimation may be circumvented via marginalized approaches. For example, if there is a large number of nuisance parameters, which are themselves not of scientific interest, estimating the joint distribution including these parameters can be very challenging. A marginalized approach as in [5] can circumvent this and learn the marginal posterior distribution over the quantities of interest, without explicitly modeling the joint distribution with the nuisance parameters. We will include this discussion on marginalization with the according references in the main paper.\n\n## Sample efficiency\nAs the Reviewer pointed out, one strength of FNOPE is that it requires fewer simulations than existing methods, which could enable inference on computationally expensive simulators. To emphasize this strength, we conducted additional experiments with 100 training simulations. See the response to Reviewer pNZ3 for more details, in particular Tables R1.2 and R1.3 show that FNOPE shows strong performance even when using only 100 training simulations in the Linear Gaussian and Antarctic Ice tasks. We agree with the Reviewer that these are empirical measurements of sample efficiency, rather than theoretical guarantees. However, this is a broad limitation in the field of SBI, and not a particular limitation of this work.\n\n## Questions\n### Using an NPE objective\n\nWe are not aware of any NPE approach that additionally conditions on variable positions $l^x$ and $l^\\theta$. Standard SBI architectures such as normalizing flows are often restricted to a fixed number of input parameters, and do not provide such flexibility. This of course stands in contrast to more recent methods such as Simformer [6], which is based on transformers. This results in a very flexible framework, but is trained with a different loss function. We also note that NPE often uses embedding nets, which can be flexibly defined without adapting the NPE loss. A Neural Operator-based embedding net could be used with a similar data-augmentation scheme as presented in our work to learn posteriors conditioned on flexibly-discretized observations. However, the discretization of the parameters would still need to be fixed (and comparatively sparse) in this scenario, as the density estimator architecture is constrained and cannot trivially be replaced by a Neural Operator.\n\n### Joint posterior and SBC EoD on marginals\nAs discussed above, we are highly interested in fitting the joint posterior, including correlation structures of the function-valued and vector-valued parameters. This is also what our evaluation metrics measure (except for calibration, where appropriate calibration measures are lacking for high dimensional spaces).\n\n### MSE\nWe apologize for the confusion regarding the MSE metric. Concretely, by MSE we refer to the MSE of the posterior **predictive**. Concretely, for each test observation $x^o_j$, we sample several posterior samples using the trained model, and for each sample, we simulate a prediction, $x_{ij}$. The MSE is then calculated between each simulation and the corresponding test observation $x^o_j$, defined as $||x_{ij}-x^o_j||^2$.  The reported MSE metric is then simply the mean of these individual MSEs over the samples $i$ from the posterior, and also averaged over several different test observations $j$. For example, for SIRD, we use 100 test observations, and 1000 posterior predictive samples for each test observation. We will make this definition clearer in our revisions. The values reported are also normalized by the number of points in the discretization of $x^o$ (full details in the Appendix S2.3).\n\n### FNOPE vs FNOPE (fix) on Linear Gaussian\nAn intuitive interpretation of the slight superiority of FNOPE (fix) in comparison to FNOPE on this task is the effect of the introduced positional noise. The noise injection modifies the likelihood, and as a result, FNOPE learns a posterior under a slightly broader likelihood than what is defined by the model. Therefore, when the true posterior is very constrained as in the Linear Gaussian task, the posterior quality is slightly poorer. However, for more challenging tasks, this is an acceptable trade off, as we gain complete flexibility on the posterior evaluation points, and the influence of the noise injection on the posterior quality is small (e.g. Fig. 5 in main paper).\n\n### NPE (spectral) for Darcy flow\nFirstly, the Darcy flow simulator presented is a highly smoothing forward process. Even though NPE (spectral) has good performance in the predictive space (e.g. after pushing the posterior samples through the simulator), the statistics of the posterior samples might differ from the statistics of the ground truth parameters, which are drawn from the prior in this case. This is why we additionally included the posterior log-probabilities of the ground truth parameters as an evaluation metric in this task. Unfortunately, we cannot calculate the log-probabilities that the spectral methods (e.g. NPE (spectral) and FMPE (spectral)) assign to the ground truth parameters as they do not model the parameters directly. Hence, these methods are missing in Fig. 5d. We will make this more explicit in the manuscript. \n\n### Number of modes for Darcy flow\nThis is an excellent remark and we are thankful that the Reviewer brought it up. We additionally ran FNOPE with 16 and 24 modes. While there were essentially no differences for MSE and SBC EoD, the posterior log-probability slightly decreased (Tab. R2.1). We do not run the spectral baseline methods with 32 modes, as for this 2-dimensional domain this results in $32\\times 32=1024$ basis functions for which these methods would need to infer the coefficients for, which is prohibitively large.\n\nTable R2.1: FNOPE performance (means and standard errors) for Darcy flow task with different numbers of modes and 10k training simulations.\n| Metric ↓ / # modes →| 16 | 24 | 32 |\n|-|-|-|-|\n| MSE | 1.26e-04 ± 1.77e-07 | 1.27e-04 ± 1.79e-07 | 1.29e-04 ± 1.85e-07 |\n| SBC EoD | 0.057 ± 0.002 | 0.055 ± 0.002 | 0.068 ± 0.003 |\n| Posterior log-prob | 0.917 ± 0.015 | 1.058 ± 0.016 | 1.534 ± 0.01 |\n\n[1] Gonçalves, Pedro J., et al. \"Training deep neural density estimators to identify mechanistic models of neural dynamics.\" elife 9 (2020).\n\n[2] Dax, Maximilian, et al. \"Real-time gravitational wave science with neural posterior estimation.\" Physical review letters 127.24 (2021)\n\n[3] Miller, Benjamin Kurt, et al. “Truncated Marginal Neural Ratio Estimation.” NeurIPS (2021)\n\n[4] Talts, Sean, et al. \"Validating Bayesian inference algorithms with simulation-based calibration.\" arXiv preprint arXiv:1804.06788 (2018).\n\n[5] Ambrogioni et al., “Forward Amortized Inference for Likelihood-Free Variational Marginalization.” AISTATS (2019)\n\n[6]​​ Gloeckler, Manuel, et al. \"All-in-one simulation-based inference.\" arXiv preprint arXiv:2404.09636 (2024)."}, {"review_id": "DmvCE4v4bZ", "reviewer": "Reviewer_u8xK", "summary": "In this paper, the authors introduce a novel approach to Simulation-based inference (SBI) by integrating Fourier Neural Operators (FNO) with a flow matching objective. The method enables high-dimensional inference while requiring several orders of magnitude fewer simulations than traditional SBI techniques. The superior performance is demonstrated across different simulated case studies and one real-world case study.\n\nAdditionally, they are able to produce posterior samples on arbitrary discretizations and avoid reliance on any additional basis functions.", "questions": "1. How sensitive is the method to the number of Fourier modes retained in the FNO layers? An ablation study to assess the trade-off between accuracy and computational cost would help.\n2. How dependent is the model's performance to the choice of prior distribution used? Would deviations from the assumed prior significantly degrade performance?", "limitations": "1. Several important evaluation metrics, like SWD, are not formally defined in the appendix. For example, the loss is referenced briefly on line 869 without an accompanying equation or detailed explanation.\n2. Key theoretical concepts referenced in the main text (Bochner's theorem - line 120) are not formally stated or explained in the appendix.\n3. The paper lacks ablation studies on several key hyperparameters. For instance, an ablation on the number of discretizations, number of fourier modes retain etc could help better understand performance bottlenecks of FNOPE.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. _Innovative combination of FNO with a flow-matching training scheme:_ The FNO component effectively learns a compact representation of the data's global structure. On the other hand, the flow-matching training scheme helps perform likelihood-free inference of conditional distributions.\n\n2. _Efficient posterior estimation in high-dimensional parameter space:_ The authors of the paper demonstrate efficient performance of FNOPE on high dimensional data under limited simulation budgets compared to FMPE, NPE and other benchmarks. Crucially, it achieves this performance without relying on fixed discretizations.\n\n3. The authors demonstrate FNOPE's excellent performance across different synthetic benchmarks and a real-world application. \n\nWeaknesses. Despite the strengths, the paper has certain weaknesses that need to be addressed:\n1. The absence of ablation studies on key hyperparameters makes it difficult to assess the robustness of the method. Understanding this would clarify different performance bottlenecks. (Check limitations and questions for more details)", "was_revised": true, "final_justification": "I keep my score, I recommend acceptance.", "rebuttal": "We thank the Reviewer for their positive evaluation of our work, and for their constructive suggestions. We note that other Reviewers also acknowledge that our method can infer parameters on arbitrary discretizations (all Reviewers) and its empirical performance, especially in the low-simulation regime, (Reviewers pNZ3, 5bMC). To address the Reviewer’s main suggestion, we conducted extensive ablation studies on the hyperparameters of FNOPE. Where appropriate, the information of these tables will be expressed in figures in our revised manuscript.\n\n\n## Investigation of hyperparameters\n\nWe conducted new experiments to investigate the impact of various hyperparameters and design choices. Overall, we can show that FNOPE performs robustly over a wide range of different hyperparameters. In detail, we tested how the performance of FNOPE is affected by the number of modes of the FNO architecture, the number of unmasked training data points in training (see Sec. 3.2 of main paper), and the lengthscale of the Gaussian process used for the base distribution. \n\n## Number of modes\n\nWe investigated how the number of modes used affects the performance of FNOPE for both the Linear Gaussian and Darcy flow experiments. We augmented the Linear Gaussian task by decreasing the lengthscale of the Gaussian process prior over the parameters, resulting in a more challenging inference task with higher frequency components. The Sliced Wasserstein Distance (SWD) of the inferred posterior to the ground truth posterior improves with more modes, saturating at around 128 for both FNOPE and FNOPE (fix) (Table R3.1). This is because the FNOPE networks with fewer modes do not have the capacity to model higher-frequency components of the samples, and so cannot reproduce realistic samples. However, when increasing the number of modes, the network is sufficiently flexible to model the posterior. This is reflected in the power spectrum of the posterior samples: While samples from models with fewer modes have less power compared to the ground truth in higher frequency components, the models with 128 and 256 modes match the power spectrum of the ground truth perfectly. \n\nTable R3.1: SWD means and standard errors for FNOPE and FNOPE (fix) for the Linear Gaussian example with a simulator prior lengthscale of 0.005 and 10k training simulations.\n| Method ↓ / # modes →| 8 | 16 | 32 | 64 | 128 | 256 |\n|---|---|---|---|---|---|---|\n| FNOPE | 0.928 ± 0.006 | 0.814 ± 0.005 | 0.568 ± 0.004 | 0.251 ± 0.002 | 0.127 ± 0.001 | 0.124 ± 0.001 |\n| FNOPE (fix) | 0.933 ± 0.006 | 0.801 ± 0.006 | 0.564 ± 0.004 | 0.215 ± 0.002 | 0.035 ± 0.000 | 0.032 ± 0.000 |\n\nWe observe a similar trend for the Darcy flow task, where reducing the number of modes used by FNOPE reduces the posterior log-probability the trained model assigns to data (Table R3.2). Overall, these results highlight that the number of modes should be chosen depending on the task complexity, but the sensitivity to this hyperparameter may be limited. We will provide a discussion in the main paper. \n\nTable R3.2: FNOPE performance (means and standard errors) for Darcy flow task with different numbers of modes and 10k training simulations. \n| Metric ↓ / # modes →| 16 | 24 | 32 |\n|---|---|---|---|\n| MSE | 1.26e-04 ± 1.77e-07 | 1.27e-04 ± 1.79e-07 | 1.29e-04 ± 1.85e-07 |\n| SBC EoD | 0.057 ± 0.002 | 0.055 ± 0.002 | 0.068 ± 0.003 |\n| Posterior log-prob | 0.917 ± 0.015 | 1.058 ± 0.016 | 1.534 ± 0.010 |\n\n## Lengthscale of prior distribution\n\nWe assume that when the Reviewer asks to investigate the dependence of the model to the ``prior distribution’’, they refer to the base distribution of the flow-matching formalism, as discussed in Sec 3.1 of our work. In typical inference settings, the prior over the simulator parameters is fixed and encodes any prior knowledge we have about the problem. According to Bayes’ theorem, changing the prior will also change the posterior. In contrast, the base distribution used in the flow-matching approach is a modeling choice, and can be defined freely.\n\nWe investigate the impact of the choice of the base distribution on the performance of FNOPE on the Linear Gaussian and SIRD experiments. Concretely, we change the Gaussian process kernel lengthscales (Sec. 3.1), which we set proportionally to the inverse of the number of modes used for FNOPE. To vary the kernel lengthscale, we multiply it with different scaling factors. Both experiments demonstrate that FNOPE performs robustly over a range of different kernel lengthscales (Tables R3.3 and R3.4). We will provide figures illustrating this in the Appendix of the paper. \n\nFurthermore, we conducted the Linear Gaussian and the Darcy flow experiments using white noise as the base distribution instead of the Gaussian process, which would correspond to a Gaussian process with a lengthscale of zero. Both for using white noise and the Gaussian process as the base distribution, the SWD is close to zero for FNOPE and FNOPE (fix) (Table R3.3). Hence, FNOPE performs well over different base distribution choices. In the Darcy flow experiment, however, we observed that using white noise as the base distribution leads to visually noisier posterior samples. We will add figures showing this to the Appendix, but cannot include it in this rebuttal.\n\nTable R3.3: SWD means and standard errors for Linear Gaussian experiment for different kernel lengthscales and white noise as base distribution. The model used 50 modes and 10k training simulations. \n| Method ↓ / Scaling factor →   | 20 | 10 | 2 | 1 | 0.1 | white noise |\n|---|---|---|---|---|---|---|\n| FNOPE | 0.028 ± 0.000 | 0.032 ± 0.000 | 0.030 ± 0.000 | 0.031 ± 0.000 | 0.023 ± 0.000 | 0.027 ± 0.001 |\n| FNOPE (fix) | 0.019 ± 0.000 | 0.016 ± 0.000 | 0.017 ± 0.000 | 0.015 ± 0.000 | 0.015 ± 0.000 | 0.016 ± 0.000 |\n  \nTable R3.4: SBC error of diagonal and predictive MSE (means and standard errors) for SIRD experiment with FNOPE for different kernel lengthscales. The model used 32 modes and 10k training simulations.\n| Metric ↓ / Scaling factor →   | 20 | 10 | 2 | 1 | 0.1|\n|---|---|---|---|---|---|\n| SBC EoD | 0.036 ± 0.003 | 0.040 ± 0.003 | 0.062 ± 0.003 | 0.051 ± 0.004 | 0.036 ± 0.003 |\n| Predictive MSE | 0.003 ± 0.001 | 0.002 ± 0.000 | 0.004 ± 0.001 | 0.003 ± 0.001 | 0.004 ± 0.001 |\n\n\n## Number of unmasked datapoints\n\nWe tested the performance of FNOPE over a wide range of unmasked training data points in the Linear Gaussian experiment. While the SWD is already small for 128 unmasked datapoints, it decreases as the number of unmasked data points increases (Tab. R3.5). With an increasing number of unmasked data points, the performance improvement starts to saturate. \n\nTable R3.5: SWD means and standard errors of the Linear Gaussian experiment for different numbers of unmasked training data points. The model used 50 modes and 10k training simulations. \n| Method ↓ / # unmasked points →   | 128 | 256 | 512 | 1000 |\n|---|---|---|---|---|\n| FNOPE | 0.090 ± 0.001 | 0.058 ± 0.001 | 0.035 ± 0.001 | 0.031 ± 0.001 |\n\n## Additional clarifications\n### Sliced Wasserstein Distance (SWD)\n\nWe apologize for not including a formal definition of the SWD. The (empirical) sliced Wasserstein(-2) distance between $N$ samples from two probability distributions $p$ and $q$ is defined as \n$$\n\\text{SWD}(p,q) = \\mathbb{E}\\_{u \\sim U(\\mathbb{S}^{n-1})} \\left[ \\frac{1}{N} \\sum\\_{i=1}^N || u^\\top x\\_i - u^\\top y\\_i ||_2^2 \\right]^{1/2},\n$$\nwhere $x_i \\sim p(x)$, $y_i \\sim q(y)$ are samples from the probability distributions and $u$ are uniformly randomly sampled vectors on the unit sphere $\\mathbb{S}^{D-1}$. \nWe want to clarify that this metric is only used as an evaluation metric and not as a “loss” for the neural network.\n\n### Bochner’s theorem\n\nWe state Bochner’s theorem as in Williams and Rasmussen [1]:\nA complex-valued function $k$ on $\\mathbb{R}^{D}$ is the covariance function of a weakly stationary mean continuous complex-valued random process on $\\mathbb{R}^{D}$ if and only if it can be represented as\n$$k(\\tau) = \\int_{\\mathbb{R}^{D}}\\exp^{2\\pi is\\cdot \\tau}d\\mu(s)$$\nfor some positive finite measure $\\mu$. \n\nCrucially, in the less general but relevant case that $\\mu$ admits a density $S(s)$, the integral is a Fourier transform between the kernel $k(\\tau)$ and the spectral density $S(s)$. We apply this result to relate the lengthscale of the square exponential kernel to the spectral density of its Fourier decomposition. The definition of the SWD and the statement of Bocher’s theorem will be added to the Appendix.\n\n[1] Christopher KI Williams and Carl Edward Rasmussen. “Gaussian processes for machine learning” volume 2 (2006)."}, {"review_id": "8a6Z1WNR2U", "reviewer": "Reviewer_5iX8", "summary": "The paper proposes FNOPE, a simulation-based inference method that leverages Fourier Neural Operators with a flow-matching objective to infer function-valued parameters, including over irregular discretizations. The authors demonstrate the method across several benchmark and real-world scientific simulation problems. While the paper studies a problem of impacts, I find it difficult to identify a clear novelty in its core contribution.", "questions": "FNOPE appears to be a technically sound implementation of familiar ideas, but as presented, it lacks sufficient novelty and comparative context to justify its contributions. Please consider \n- Clarify the novelty of the method in the context of existing operator-based inverse problem literature. \n- Include direct comparisons with other neural operator methods. \n- Conduct deeper ablation studies to isolate the impact of NUDFT, GP priors, and other design decisions. \n- Strengthen the motivation for using FNOs specifically over more flexible architectures that already support irregular discretizations.", "limitations": "I suggest the authors to more clearly articulate the novel contributions of the paper, especially in light of extensive prior work on using neural operators for inverse problems in functional spaces. Please strengthen the positioning and clarify the methodological innovations.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 1, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths: Nothing stood out as commendable.\n\nWeaknesses:\n- My main concern lies in the novelty and positioning of the contribution. Using neural operators, e.g., FNOs, to approximate mappings between function spaces is well-established. Applying them to inverse problems, including probabilistic ones, has been widely explored in recent works such as Invertible FNO (iFNO), Latent Neural Operators (LNO), and Neural Inverse Operators (NIO). If the primary novelty is the integration of NUDFT to support irregular meshes, that seems incremental, as this mechanism is neither new nor unique to this work. Several transformer-based neural operators also support irregular discretizations and are known to perform well, especially under sparse observations, as acknowledged by the authors themselves in the SIRD task (Fig. 4, pg. 6).\n- Another problem is that I fail to see a fair baseline comparisons against other neural-operator-based inverse methods. It is unclear why the authors exclude these from the baseline set. Without such comparisons, it is difficult to assess whether FNOPE contributes meaningfully beyond existing neural operator architectures tailored for inverse tasks.\n- While the paper includes multiple benchmarks, it lacks deeper ablations that would clarify which components are key to the method's performance. For instance, the impact of NUDFT versus FFT, the number of retained Fourier modes, or the role of GP noise shaping is not systematically studied. This makes it hard to disentangle whether FNOPE’s gains arise from architectural choices, training strategy, or task-specific tuning.\n\n1. Long, D., Xu, Z., Yuan, Q., Yang, Y. and Zhe, S., Invertible Fourier Neural Operators for Tackling Both Forward and Inverse Problems. In The 28th International Conference on Artificial Intelligence and Statistics.\n2. Molinaro, R., Yang, Y., Engquist, B. and Mishra, S., 2023, July. Neural inverse operators for solving PDE inverse problems. In Proceedings of the 40th International Conference on Machine Learning (pp. 25105-25139).\n3. Kaltenbach, S., Perdikaris, P. and Koutsourelakis, P.S., 2023. Semi-supervised invertible neural operators for Bayesian inverse problems. Computational Mechanics, 72(3), pp.451-470.\n4. Wang, T. and Wang, C., Latent Neural Operator for Solving Forward and Inverse PDE Problems. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.\n5. Behroozi, A., Shen, C. and Kifer, D., Sensitivity-Constrained Fourier Neural Operators for Forward and Inverse Problems in Parametric Differential Equations. In The Thirteenth International Conference on Learning Representations.\n6. Zhongkai Hao, Zhengyi Wang, Hang Su, Chengyang Ying, Yinpeng Dong, Songming Liu, Ze Cheng, Jian Song, and Jun Zhu. GNOT: a general neural operator Transformer for operator learning. In Proceedings of the International Conference on Machine Learning (ICML), 2023.\n7. Wu, H., Luo, H., Wang, H., Wang, J. and Long, M., 2024, July. Transolver: A Fast Transformer Solver for PDEs on General Geometries. In International Conference on Machine Learning (pp. 53681-53705). PMLR.", "was_revised": true, "final_justification": "I’m ok with the revision and will raise my score from 2 to 3. Nevertheless, I’m surprised to hear that neural-operator approaches haven’t been applied to this specific problem yet, as the authors claim and another reviewer notes. Accordingly, I remain unconvinced about the work’s originality and novelty.", "rebuttal": "We thank the Reviewer for the concrete suggestions. Here, we clarify the key contributions of our work, provide extensive ablation experiments and benchmark against a suggested inverse method. We hope this additional context will allow the Reviewer to rethink their score.\n\n## Clarification of Main Contributions\nThe primary contribution of our work is the development of a simulation-based inference (SBI) approach for function-valued parameters. Importantly, we clarify that Bayesian inference, in general, is _not_ equivalent to ’solving an inverse problem’: In Bayesian inference, we try to estimate the full posterior distribution over parameters, and in simulation-based inference, we aim to do this without access to likelihoods (see [1]). In contrast, inverse problems typically focus on _deterministic_ models, and aim to find a _point estimate_ that optimizes some (regularized) loss, and therefore, e.g., lack uncertainty estimates which are often crucial for scientific applications (see also response to Reviewer 5bMC). Thus, our method is simply solving a different problem to the papers pointed out by the reviewer, which either address deterministic inversion methods, such as Molinaro et al., Wang et al., and Behroozi et al. [R2, R4, R5], or on the other hand, _likelihood-based_ Bayesian inference methods, such as Kaltenbach et al. [R3].\n\nIn our manuscript, we discuss related approaches that can be used for simulation-based inference of function-valued parameters, as well as how our work differs from these approaches. However, we agree with the Reviewer that clarifying the distinction between simulation-based inference and deterministic and/or likelihood-based inference in the Neural Operator context will improve the clarity of our work, and we will add this discussion in our revised manuscript.\n\n## New baseline: Invertible Fourier Neural Operators (iFNOs)\nFrom the works referenced by the Reviewer, we noticed that only iFNOs [R1] can be framed as an SBI method. We thank the Reviewer for bringing up this work as including a comparison with a Neural Operator-based method will improve our paper. However, we point out that the loss presented in [R1] does not explicitly target the posterior distribution. Instead, it trains a Neural Operator-based conditional probabilistic generative model, but it is unclear whether this model learns the posterior distribution, or a different conditional distribution. To include the iFNO as a baseline in our work, we trained an iFNO on our Darcy flow problem with 1000 training simulations. While the iFNO produces reasonable samples of the parameters in terms of predictive MSE [Table R4.1], its samples lack any uncertainty. This is evident in the simulation-based calibration metric (SBC EoD, lower is better), which is close to 0.25. This value is expected for a point estimate (see discussion at the end). We also point out that this value is considerably higher than the SBC EoD values achieved by any of the methods shown in our manuscript for any task. For this experiment, we considered the low training budget regime as this is what the authors of [R1] used in their work. Furthermore, we use the same settings as in [R1] for the `D-CURV` experiment. We will include a comparison to iFNO in the manuscript. \n\nTable R4.1: Comparison of FNOPE and iFNO performance (means and standard errors) on Darcy flow task with 1000 training simulations.\n| Method ↓ / Metric →  | Predictive MSE | SBC EoD |\n|-|-|-|\n| FNOPE | 1.30e-04 ± 2e-07 | 0.067 ± 0.004 |\n| iFNO | 1.27e-04 ± 1.8e-07 | 0.232 ± 0.001 |\n\n## Transformer-based Neural Operators\nThe other two papers mentioned by the Reviewer [R6, R7] introduce new architectures for transformer Neural Operators without investigating their use for inference problems. These works are not directly comparable to our work, but using transformer Neural Operators in our inference framework as opposed to FNOs would make for an interesting approach. Our choice of the FNO architecture was motivated by the inductive bias for smoothly varying functions that FNOs leverage to obtain relatively small network sizes. In the response to Reviewer pNZ3 we provide a full breakdown of the network sizes and training times for our experiments [Table R1.1]. Even for the very large Darcy flow task ($\\sim 16$k parameters), we observe that our network converged within just over an hour of training. While this network was comparably large ($\\sim 11$M), we repeated the experiment in our ablations (Table R4.3), reducing the number of modes from 32 to 16. Here, we observed comparable performance despite the network size being much smaller ($\\sim$3M). Nevertheless, we agree with the Reviewer that exploring transformer Neural Operators could be a beneficial future direction for our work. We will include a discussion of these papers in our manuscript.\n\n## NUDFT vs. FFT\nWe want to highlight that NUDFT is _mathematically equivalent_ to FFT if it operates on a uniform grid, whereas the FFT cannot be applied to non-uniform grids. Thus, FNOPE (fix) can be seen as an ablation of FNOPE where we replace NUDFT with FFT and add no positional noise. Additionally, we ran FNOPE on non-uniform grids, but removed the positional noise to investigate the influence of the non-uniform discretization and the corresponding processing by NUDFT. For the Linear Gaussian task there are essentially no differences of the posterior quality in terms of SWD to the ground truth posterior (Table R4.2). However, we emphasize that without positional noise, we lose the flexibility to evaluate at arbitrary positions.\n\nTable R4.2: SWD (means and standard errors) of the Linear Gaussian task with and without positional jitter.\n| Method ↓ / # simulations → | $10^2$ | $10^3$ | $10^4$ | $10^5$ |\n|-|-|-|-|-|\n| FNOPE | 0.137 ± 0.003 | 0.080 ± 0.002 | 0.033 ± 0.001 | 0.024 ± 0.000 |\n| FNOPE (fix) | 0.040 ± 0.001 | 0.028 ± 0.001 | 0.014 ± 0.000 | 0.012 ± 0.000 |\n| FNOPE (no jitter) | 0.123 ± 0.003 | 0.087 ± 0.002 | 0.035 ± 0.001 | 0.023 ± 0.000 |\n\n## Ablation Studies\nAs suggested by the Reviewer as well as Reviewers pNZ3 and u8xK, we have conducted several ablation studies to explore the effect of different hyperparameters on FNOPE. The ablations show that FNOPE performs robustly over a wide range of different hyperparameters, e.g. number of modes, lengthscale and unmasked training points. Below, we refer to tables in response to reviewer u8xK.\n\n## Number of modes\nTo investigate the influence of the number of modes, we adapted the Linear Gaussian task with a new prior, which produces samples varying at higher frequencies. Increasing the number of modes improves performance in terms of SWD as it allows the network to model higher-frequency components of the parameters more reliably (Table R3.1) and we observe a similar trend for the Darcy flow task (Table R3.2). Overall, these results highlight that the number of modes should be chosen depending on the task complexity, but the sensitivity to this hyperparameter may be limited. We refer to the response to Reviewer u8xK for a detailed discussion. \n\n## Lengthscale of base distribution\nVarying the kernel lengthscale of the base distribution (Sec. 3.1 of paper) has a limited effect on the resulting posterior quality for the Linear Gaussian task (Table R3.3), even in the limit of a white noise base distribution. \n\nAdditionally, we conducted an ablation on the Darcy flow task by varying the lengthscale of the base distribution. While the predictive MSE is robust to different lengthscales, calibration performance measured by SBC EoD is getting slightly worse with increasing lengthscale, whereas the posterior log-probability is best for intermediate lengthscales. In summary, a reasonable intermediate lengthscale scaling factor performs well across all metrics, and this is what we used in the main experiment (Table R4.3).\n\nTable R4.3: Darcy flow with 32 modes and different lengthscale scaling factors. Values reported are for 10,000 training simulations. \n| Metric ↓ / Scaling factor → | 0.1 | 0.2 | 1.0 | 2.0 | 10.0 | 20.0 |\n|-|-|-|-|-|-|-|\n| MSE | 1.26e-04 ± 1.88e-07 | 1.26e-04 ± 1.93e-07 | 1.26e-04 ± 1.83e-07 | 1.27e-04 ± 1.82e-07 | 1.27e-04 ± 1.81e-07 | 1.26e-04 ± 1.76e-07 |\n| SBC EoD | 0.125 ± 0.003 | 0.139 ± 0.003 | 0.115 ± 0.003 | 0.079 ± 0.004 | 0.062 ± 0.003 | 0.055 ± 0.003 |\n| Posterior log-prob | 1.557 ± 0.005 | 1.424 ± 0.007 | 1.714 ± 0.005 | 2.346 ± 0.007 | 0.926 ± 0.017 | 1.148 ± 0.02 |\n\n## Number of unmasked training points\nFurthermore, we conducted an ablation study to investigate the impact of the number of unmasked training data points on the performance. In summary, FNOPE performs well for different choices of unmasked training data points. For more details on this, we refer to the response to Reviewer u8xK (Table R3.5).\n\n## SBC EoD for a point estimate\nWhen training the iFNO baseline on the Darcy flow task, we obtain a Simulation-based Calibration Error of Diagonal (SBC EoD) of close to 0.25 (Table R4.1). This indicates that the distributions yielded by the iFNO are very close to point estimates. Here, we want to refer to our definition of the SBC EoD metric (Appendix S2.2). A one-dimensional marginal of a point mass distribution either always overestimates or underestimates the ground truth value. Hence, the ground truth will have rank $r_{ij}=0$ or $r_{ij}=N_\\text{post}$. Supposing that over different observations $x_j^o$, the point mass distribution is equally likely to underestimate or overestimate the ground truth, the cumulative distribution of the ranks will be given by $\\text{CDF}(\\alpha)=0.5$ for all significance levels $\\alpha\\in (0,1)$. Given the definition of the SBC EoD, a point mass distribution results in a SBC EoD value of $\\int_0^1 |\\text{CDF}(\\alpha)-\\alpha|d\\alpha = 0.25$.\n\n[1] Cranmer, Kyle, Johann Brehmer, and Gilles Louppe. \"The frontier of simulation-based inference.\" Proceedings of the National Academy of Sciences (2020)."}]

Author rebuttal:

Meta-review: Reviewers all find merit in this paper, praising the problem as important (Reviewer PNZ3, Reviewer 5bMC), the writing as clear and high-quality (Reviewer 5bMC), and the combination of Fourier neural operators and simulation-based inference as interesting (Reviewer pNZ3, Reviewer u8xK). Most of the weaknesses I saw had to do with minor details about experimental evaluation, and the fact that this is a paper whose main contribution is to combine two existing methods, making it potentially incremental (Reviewer 5iX8). I am not convinced that incrementalness is a problem here, because learning in function spaces, compared to learning over finite-dimensional vectors, is sufficiently-rich that extensions to this domain are by their nature not that straightforward. The other downside mentioned is lack of theoretical guarantees (Reviewer 5bMC), but this weakness applies to the entire research areas this paper covers, rather than just this paper - and these areas have proven themselves useful in practice, so this criticism is similarly not too convincing.

On basis of reviewer consensus, I recommend acceptance of this work
Decision: Accept (poster)