Universal Functional Regression with Neural Operator Flows
Authors: Yaozhong Shi
Angela F Gao
Zachary E Ross
Kamyar Azizzadenesheli
Venue: Accepted by TMLR ()

Abstract: Regression on function spaces is typically limited to models with Gaussian process priors. We introduce the notion of universal functional regression, in which we aim to learn a prior distribution over non-Gaussian function spaces that remains mathematically tractable for functional regression. To do this, we develop Neural Operator Flows (OpFlow), an infinite-dimensional extension of normalizing flows. OpFlow is an invertible operator that maps the (potentially unknown) data function space into a Gaussian process, allowing for exact likelihood estimation of functional point evaluations. OpFlow enables robust and accurate uncertainty quantification via drawing posterior samples of the Gaussian process and subsequently mapping them into the data function space. We empirically study the performance of OpFlow on regression and generation tasks with data generated from Gaussian processes with known posterior forms and non-Gaussian processes, as well as real-world earthquake seismograms with an unknown closed-form distribution.

Published in Transactions on Machine Learning Research (10/2024)
Universal Functional Flows
Regression with Neural Operator
Yaozhong Shi California Institute of Technology
yshi5@caltech. edu
Angela F. Gao California Institute of Technology
afgao@ealtech.coom edu
Zachary E. Ross
aross@caltech.edd edu
California Institute of Technology
Kamyar Azizzadenesheli
kamyora@nvidia.com
NVIDIA Corporation
Reviewed on OpenReview: https: //
// openreview. net/ forum? id- 3HL329X333
Abstract
Regression on function spaces is typically limited to models with Gaussian process priors. We introduce the notion of universal functional regression, which we aim to learn a prior distribution over non-Gaussian function spaces that remains mathematically tractable for functional regression. To do this, we develop Neural Operator Flows (OPFr.w), infiniteddimensione extension of normalizing flows. OPFLOW is an invertible operator that maps the (potentially unknown) data function space into Gaussian process, allowing for exact likelihood estimation of functional point evaluations. OrFLow enables robust and accurate uncertainty quantification via drawing posterior samples of the Gaussian process and subsequently mapping them into the data function space. We empirically study the performance of OPF1.w on regression and generation Lasks with data generated from Gaussian processes with known posterior forms and non-Gaussian processes, well as real-world carthquake scismograms with unknown closcd-form distribution.
1 Introduction
The notion inference on function spaces is essential to the physical sciences and engincering. In particular, it is often desirable to infer the values of function everywhere in a physical domain given sparse number of observation points. There are numerous types of problems in which functional regression plays important role, such as inverse problems, time series forecasting, data imputatiinnassiiiiattioo. /asssmissiiiaioo. Functional regression problems can be particularly challenging for real world datasets because the underlying stochastic process is often unknown.
Much of the work on functional regression and inference has relied on Gaussian processes (GPs) (Rasmussen & Williams 2006), & specific type of stochastic process in which any finite collection of points has multivariate Gaussian distribution. Some of the earliest applications focused on analyzing geological data, such as the locations of valuable deposits, to identify where new deposits might be found (Chiles & Delfiner 2012). GP regression (GPR) provides several advantages for functional inference including robustnoss and mathematical tractability for various problems. This has led to the use of GPR in an assortment scientific and engineering fields, where precision and reliability in predictions and inferences can significantly impact outcomes (Deringer ct al. 2021: Aigrain Foreman-Macha prior for functional inference problems can be rather limiting, particularly in scenarios where the data exhibit. heavy-tailed multimodal distributions, e.g. financial time series environmental modeling. This
(023). Despite widespread adoption, the assumption of a GP

---

Published in Transactions on Machine Learning Research (10/2024)
underscores the need for models with greater expressiveness, allowing for regression general function spaces, with data arising from potentially unknown stochastic processes and distributions (Fig. 1 We refer to such a regression problem as universal functional regression (UFR).
We hypothesize that addressing the problem of UFR requires a model with two primary components. First, the model needs to be capable of learning priors over data that live in function spaces, Indeed, there has been much recent progress in machine learning on learning priors on function spaces (Rahman et al. 2022a
Lim et al.) 2023 Seidman et al., 2023 Kerrigan et al., 2023a Pidstrigach et al. 2023 Baldassari et al. 2023
Hagemann et al. 2023 Kerrigan et al. 2023b Franzese et al., 2023). Second, the models need a framework for
performing functional regression with the learned function space priors and learned likelihood a component
that is critically missing from the previous works pure generative models described above. This includes the ability to compute the likelihood of an input data sample, which would allow for posterior estimation over the entire physical domain. Addressing these challenges not only expands the models available for functional regression but also enhances our capacily to extract meaningful insights from complex dalasets.
In this study, our contributions are as follows. We develop a viable formulation for UFR with a novel learnable bijective operator, OPFlOW. This operator extends the classical normalizing flow, which maps between finite dimensional vector spaces, to the function space setting. OFFLOW consists of an invertible neural operator, an invertible map between function spaces, trained to map the data point-process distribution a known and easy-to-sample GP distribution. Given point value samples of a data function, OrFLow allows for computing the likelihood of the observed values, a principled property priori held by GP. Using this property, we formally define the problem setting for UFR and develop approach for posterior estimation with Stochastic Gradient Langevin Dynamics (SGLD) (Weling & Tch 2011). Finally, we demonstrate UFR with OrFLow on a suite of experiments that use both Gaussian and non- Gan-Saassian processes.
In comparison to GPR, OpFLow draws on the training dataset to learn the prior over a function space. When such datasets are only minimally available, kernel tuning methods propose to hand-tune the kernel and parameters of GPR, providing considerable performance, although still with a Gaussian assumption. In such limited data scenarios, learning accurate priors with OPFLOw may be difficult and less suitable. When training data is non-existent, often neither the plain GPR nor OrFLOw can be of help; when expert knowledge can be incorporated (Aigrain & Foreman-Mackey 2023). there may be advantages to GPR. However, such method can also be considered as hand-tuning the prior used in OpFLOw by, for example, setting the invertible model to identity map, resembling strong prior over the prior learning process.
2 Related work
Neural Operators. Neural Operator is a new paradigm in deep learning for lcarning maps between infnitt-dimmensinaal function spaces. Unlike traditional neural networks that primarily work with fixed-diieenioonal vectors, neural operators are designed to operate on functions, making them inherently suitable for a wide range of scientific computing and engineering tasks involving partial differential equations (PDEs) Kovachki et al. 2023 Li et. al.] 2020). A key feature of neural operators is their discretization agnostic (resolution invariant) property (Kovachki et al. 2023), which means that a neural operator can learn from data represented on various resolutions and predict outcomes on yet new resolutions. This property is particularly valuable in applications involving the natural sciences, PDEs, and complex physical simulations where functional data. may come from variable discretizations of meshes or sensors. Within the
(a)
(b)
Figure 1: Examples of data
from non-Gaussian processes. (a) Temperature field from Rayleigh Bénard convection problem. (b) Vorticity field from Naiver-Stokes Equation.
family of neural operators, Fourier Neural Operator (FNO) (Li et al. 2021 stands out for its quasi-linear time complexity by defining the integral kernel in Fourier space and has been applied to many problems in engineering and science (Axizzaeenesheli et al. 2024).. In the conventional view of neural operators, provided discretization and point evaluation of functions resembles approximation of function for which increasing the resolution is connected to improved approximation error in the integration and differentiation operators (Liu- Schiaffii et al. (024). The proposed view in the current work provides a new perspective for which the point evaluations are instead considered as points samples of functions that are associated
2

---

Published in Transactions on Machine Learning Research (10/2024)
probably rather than finite bases and finite resolution approximation of continuous function. Therefore, increasing the resolution is seen as providing more information for regression rather than a way to reduce, e.g., the approximation error in the integral operators used in neural operations.
Function space generative modeling with Neural Operators. Several discretization invariant generative
models on function spaces have been recently developed using Neural Operators (Li et al., 2020 Kovachki
et al. 2023). These models include Generative Adversarial Neural Operator (GANO) (Rahman et al.
2022a Denoising Diffusion models (DDO) (Lim et al., 2023) and Variational Autoencoding Neural Operator
(VANO) (Seidman et al., (023). To be specific, GANO the first proposed infiniteddimensional generative
model that generalizes GAN, and learns probability measures on function spaces. However, GANO training
relies on an adversarial game, which can be slow (Goodfellow et. al., 2014; Rahman et al. 2022a DDO is insolution-invarinn
diffusion generative model for function spaces, leveraging denoising score matching within Hilbert spaces and employing ciiniteddimensionall Langevin dynamics for sampling. The inference of DDO requires a diff'usion process, as with the finite-dimensional score-based model, which can be time- consuming (Song et al. 2021 Lim et al., 2023): : VANO offers a functional variational objective (Kingma & WWelling, 2022 Seidman et al. 2023), mapping input functional data to a finite-dimensional latent space through an encoder, which may be a sub-optimal formulation due to the fixed dimensionality of the latent space being decoupled from the discretization of the input function data.
All of these aforementioned models have exhibited superior performance over their finle-dimennsioall counter- parts for functional data by directly learning mappings between function spaces, which show great potential in addressing the challenges in science and engincering problems (Shi et al. 2024a bj Yang et al.) 2021
Avizzadenesheli et al.. 2024h. Compared to these existing infinile-dimensional generalive models, OPFLoW shows advantages in fast training, fast inference, bijective architecture, and providing precise likelihood estimation as summarized in Table [ These advantages of OPFLOW can be explained by: (i) The training and inference processes of OpFLow only involve likelihood estimation, and don't require an adversarial game or a diffusion process. (ii) OPFr.w directly generalizes normalizing flows to function space without. mnite-dimensional simplicity, and enables precise likelihood estimation for functional point evaluations on irregular grids. (iii) As invertible operator, OPFLOw can be useful for tasks that require the bijectivity
between function spaces [Zhou et al.. 2023 Furnya et al. 2023
Table 1: Comparison of OrFLow with other infinite-dimensionn generative models
Models
Fast training
Fast inference Bijective architecture
Exact likelihood estimation
GANO
X
X
X
DDO
x
X
X
/
VANO
X
X
OPFLOW [Ours]
/
Neural Process (NP) (Garnelo et al. 2018 leverages neural networks address the constraints associated with GPs, particularly the computational demands and the restrictive Gaussian assumptions for priors and posteriors (Dutordoir et al. 2022). While NP aims to model functional distributions, several fundamental flaws suggest NP might not be a good candidate for learning function data (Rahman
Neural Processes.
let al.) 2022a Dupont et al.] 2022). First, NP lacks true generative capability; it focuses on maximizing the likelihood of observed data points through an amortized inference approach, neglecting the metric space of the data. This approach can misinterpret functions sampled different resolutions as distinct functions, undermining the NP's ability to learn from diverse function representations. As pointed out in prior works (Rahman et al.) 2022a), if dataset comprises mainly low-resolution functions alongside single function with a somewhat high resolution, NP only tries to maximize the likelihood for the point evaluation of the high-resolution function during training and ignores information from all other lower-resolution functions. A detailed experiment to show the failure of NP for learning simple function data can be found in Appendix A.1 of (Rahman et al., 2022a) What's more, NP relies on an encoder to map the input data pairs to fnito-dimensional latent variables, which are assumed to have Gaussian distributions (Garnelo et al. (208)), and then project the mimee-dimensional vector to an infnite-dimensional space; this approach results in diminished consistency at elevated resolutions, and thus disables NP to scale to large datasets (Dupont et al.
3

---

Published in Transactions on Machine Learning Research (10/2024)
2022). The other fundamental limitation is the Bayesian framework of NP is also defined on set of points, rather than functions themselves. This method results in the dilution of prior information as the number of evaluated points increases, further constraining NP's ability to generalize from function data.
Normalizing flows are a class of flow-based finte-dimensionall generative models, that usually composed of sequence of invertible transformations (Kingma & Dhariwal 2018 Dinh et al.) 2017 Chen et al. 2019). By gradually transforming a simple probability distribution into a more complex larget distribution with the invertible architecture, normalizing flows enable exact likelihood evaluation, direct sampling, and good interpretability (Kobyzev et al. 2021 p with applications from image and audio generations (Kingma & Dhariwal, 2018 Ping et al. 2020), reinforcement learning (Mazoure al. 20200) to
Normalizing Flows.
Bayesian inference (Gao et al. 2021 Whang et al., 2021 Despite the advantages of normalizing flows, they still face challenges with high-dimensional data due to computational complexity, memory demands, scalability, and cannot be diroctly used for zero-shot super-resolution (Lugmayr et al. 2020). Morcover, traditional normalizing flows are defined on a finiteddinensional spaces, and constrained to evaluating likelihood on fixed-size, regular grids. This inherent limits their applications in Bayesian inverse problems, where the data may not conform to fixed and regular grids (Dashti & Stuart 2017). These limitations underline the ncod for new generation of normalizing flows designed on function space, providing a solution that could potentially transcend the limitations of traditional models.
Nonparametric Bayesian and Gaussian Process Regression. Nonparametric Bayesian models are defined on an infinitedlimensional parameter space, yet they can be evaluated using only finite subset of parameter dimensions to explain observed data samples (Orbanz & Teh, 2010). These models provide a flexible regression framework by offering analytically tractable posteriors and enable adjusting their complexity by allowing the model size to grow when more observations are available. Further, nouparametric Bayesian leverages a rich theoretical foundation of parametric Bayesian modeling and can be evaluated using common inference techniques like Markov Chain Monte Carlo (MCMC) and variational inference, etc. Despite the great advantages of the nonparametric Bayesian models, such models heavily depend on the chosen prior, which could restrict their adaptability and scalability in real-world complex scenarios (Barry) 1986 Wasserman, 2006). The emergence of learning-basee Bayesian models, propelled by advancements in deep learning, offers potent ial solution by marrying the flexibility and scalability of neural networks with Bayesian principles; these modern approaches learn directly from data rather than rely on the chosen priors.
Within the family of nonparmateric Bayesian models, GPR is distinguished for its widespread applications, characterized by its analytical posteriors within the Bayesian framework (Rasmussen & Williams 2006). GPR offers a robust framework for capturing uncortainties in predictions given noisy observations. However, GPR assumes data can be modeled using a Gaussian process, both the prior and posterior should be Gaussian. Such assumption, while simplifying the analytical process, might not always align with the complex nature, thus the universal applicability of GPR is constrained by the diversity and complexity of real-world datasets.
Deep GP stacks multiple layers of Gaussian process (Damianou & Lawrence 2013), which enables modeling of highly nonlinear and potentially Gan-Gaussian data, as well as providing uncertainty quantification at each layer (Jakkala 2021 Liu et al. 2020 de Souza et al. 2024]. However, as Ustyuzhaninov et. al. (2020); Havasi et al. (2018) have pointed out, achieving exact Bayesian inference in Deep GP is challenged by its inherent intractability, necessitating the use of simplifying approximations. Unfortunately, these approximations could lead to the layers of a Deep GP collapsing to near- deterministic transformations (Ustyuzhaninov et al., 2020 significantly reducing the model's probabilistic nature and its ability to quantīfy uncertainty accurately. Separately, Warped GP presents an alternative for modeling non-Gaussian data by either transforming the likelihood of the outputs of GP with nonlinear transformations (Snelson et al.] 2003) or transforming the prior. While Warped GP has shown efficacy in handling real-world non- Gaussian data (Kou et all 2013), the selection of an appropriate warping function is critical and demands extensive domain expertisc, where an ill-suited warping function can severely degrade model performance (Rios & Tobar 2019). Besides, similar to Deep GP, Warped GP also suffers from analytical intractability, often requiring the employment of variational inference methods for practical implementation Silzaro-Gredilla 2012). Recently, inspired by the success of normalizing flows, (Maroñas) et al. 2021) proposed Transformed GP, which utilizes a marginal normalizing flow to learn the non- Gaussian prior. This approach not only matches the capability of traditional Deep GP in modeling complex data,
Deep and Warped Gaussian Processes.
4

---

Published in Transactions on Machine Learning Research (10/2024)
but is also computationally efficient. However, Transformed GP use, yet again, pointwise wrapping of GP point evaluation, resulting in marginal flows that perform diagonal mapping, limiting its applicability to learn more general stochastic process. Additionally, the inference of Transformed GP relies on sparse variational objective which is considered as a sub-optimal approach in terms of accuracy.
Regression with OFFLOw addresses the limitations of existing Gaussian process models discussed above, and offer two key improvements over the Transformed GP framework (Maroñas et al., 2021). To be specific, the OFFLOW enables modeling a more general stochastic process compared to the marginal flow which is based pointwise operator. The regression with the OPFl.ww enables true Bayesian uncertainty quantification via drawing posterior samples of the Gaussian process with SGLD and subsoquently mapping them into the data function space. Such regression formulation is valid in the sense of correctness, imposing no approximation in the formulation, which enables us to capture true Bayesian uncertainty. The mentioned conceptual advancements are ompirically shown beneficial in this paper. In contrast, variational inference formulation often relics on optimizing the Evidence Lower Bound (ELBO) and yields an approximate posterior with an approximation gap "in the formulation", rather than the true posterior. Despite the benefit, the proposed regression framework with SGLD requires significantly more time for sampling compared to variational inference, where the latter is more practical for time-sensitive application. Last, should note that the regression framework of OpFrow slightly differs from that of the generic. Deep GP. In Deep GP or Transformed GP, the model is fitted to the entire training dataset. For example, when analyzing dataset containing rainfall measurements over a ten-year period, a Deep GP or Transformed GP would be used to directly fit. all the data, enabling predictions of rainfall at any point within these ten years. In contrast, the regression with OPFLow can be analogous to image inpainting or reconstruction from pixels with generative models in a Bayesian manner (Marinescu et al. 2021p. In the OPFLOW framework, each data point in the training set represents a distinct function, polentially having its unique discretization. During training, the goal of OpFLOW is to learn bijective mapping between a Gaussian process and the data function space. Once trained, OPFLow will be frozen and serve as the prior, when provided with new observations (i.c., point evaluations of an unknown function drawn from the data function measure), the task is to ruct the entire function, aligning OpFLow closely with the traditional settings of standard GP regression. The regression setup with OPFLOW is formally defined in the subsequent section.
3 Preliminaries
Universal functional regression. Consider function space, denoted as U, such that for any E Ut u: Dy 1y Rd. Here, U represents the space of the data. Let Py be probability measure defined on U. A finite collection of observations w|D D is a set of point evaluations of function u on a few points in its domain D C Du, where D is the collection of co-location positions {ar){_1, Iy E Du- Note that in practice, u E l is usually not observed in its eutirety over Du, but instead is discretized on a mesh. In UFR, independent and identically distributed samples of {u,}}= ~ Pu are accessed on the discretization points {D,);- constituting a dataset. of (uy.,,""= The point evaluation of the function draws, i.e., u E U, can also be interpreted as a stochastic process where the process is discretely observed. The behavior of the stochastic process is characterized by the finite-dimensional distributions, such that for any ion Di, p(ulp=) denotes the probability distribution of points evaluated on a collection Di, What's more, p(u|ulpr) ) is p(u given ulD:), that is the D' collocation points. In the UFR setting, each data
posterior. Similarly, p(ul or | mt) is the posterior sample Mj is provided on a specific collection of domain points Dj, and for the sake of simplicity of notation, we consider all the collected domain points to be the same and referred to by D.
The main task in UFR is to learn the measure Py from the data (uyl 1, 1y= such that, at the inference time, for a given u of a new sample u, we aim to find its posterior and the u that maximizes the posterior,
max p(u ul)
wEl
as well as sampling from the posterior distribution. Please note that, when Py is known to be Gaussian, then CFR reduces to GPR.
Invertible operators on function spaces. Consider two Polish function spaces, denoted A and U, such that for any a E A,a: a: DA - RdA, and for any u E U,u Du - Rdd, Here, 2 represents the space of the
5

---

Published in Transactions on Machine Learning Research (10/2024)
data, whereas A represents the latent space. Let Pu and PA be probability measures defined on U and A, respectively. In this work, we take samples of functions in A, i.e., Oj E A to be drawn from a known GP, and consequently, the evaluation of ajIDa is a Gaussian random variable equipt with a Gaussian probability measure. Under these construction, and for a suitable Gaussian process on A, we may establish an invertíble (bijective) operator between A and [ We denote the forward operator as G A + U, such that for any aj: 9(a,)= = Uj and its corresponding inverse operator 9-1 as F, where F G-1 and F : 4 + A, such that for any Uj. F(u;) = aj. We aim to approximate F and correspondingly its inverse g with an invertible learnable model F8 = F, and Go Fø" from a finite collection of observations {u,lo};m Finally, to have well-defined invertible maps between the two measures, we assume absolute continuity of measures on A and U with respect to each other. Given the GP nature of the measure on A, such continuity is obtained by adding a small amount of GP noise to the measure on U. A similar procedure is envisioned and incorporated by adding Gaussian noise to images in the prior normalizing flow in fimens-iieeniioall spaces (Huang et al. Shorten & Khoshgoftaar 2019
2020
4
Neural Operator Flow
4.1
Invertible Normalized Architecture
We introduce the Neural Operator Flow (OrFlow), an innovative framework that extends the principle of the fuite-dinenniional normalizing flow into the realm of infiniteddimensional function spaces. The architecture is shown schematically in Fig. 2 OpFLOw retains the invertible structure of normalizing flows, while directly operating on function spaces. It allows for exact likelihood estimation for point estimated functions. OPFLOW is composed of sequence of layers, cach containing the following parts:
Channel-wist normalization with trainable scale and bias parameters is often effective in facilitating affine transformations in normalizing flow neural network training (Kingma & Dhariwal 2018). We implement a function space analog by representing the scale and bias as learnable constant-valuee functions. Specifically, let (v')' denote the input function to the ith actnorm layer of OPFrow and let s and in represent. the learnable constant-valued scale and bias functions, respectively, in this layer. The output of the actnorm layer is then given by v² = si O (v") + bj. where O denotes pointwise multiplication in the function space.
Actnorm.
Domain and codomain partitioning. The physical domain refers to the space in which the data resides, such as the time domain for 1D data or the spatial domain for 2D data. Conversely, the channel domain is defined as the codomain of the function data, such as the 3 channels in RGB images or Lemperature, velocity vector, pressure, and precipitation in weather forecast. (Pathak et al., 2022 Hersbach et al. 2020). In normalizing flows, a bijective structure requires dividing the input domain into two segments (Dinh et al.) 2017 Kingma & Dhariwal 2018). Based on the mothod of splitting the data domain, we propose two distinct OPFL.wW architectures.
(1) Domain partitioning: In this architecture, we apply checkerboard masks to the physical domain, a technique widely used in existing normalizing flow models (Dinh et al.) 2017 Kingma & Dhariwal 2018) which we extend to continuous domains. Our experiments in the following sections reveal that the domain decomposed OPFLOW is efficient and expressive with a minor downside in introducing checkerboard pattern artifacts in zero-shot super-resolution tasks.
(w) Codomain partitioning:
This approach partitions the data along the codomain of the input function data, while maintaining the integrity of the entire physical domain. We show that, unlike its domain decomposed counterpart, the codomain OpFLow does not produce artifacts in zero-shot super-resolution experiments. However, it exhibits a lower level of expressiveness veness compared to the domain decomposed OPFLOW.
Affine coupling. Let v' represent the input function data to the ith affine coupling layer of OPFLOw. Then, we split v² either along the physical domain or codomain, and we have two halves hi hi = split('") (1²). Let T denote the ith affine coupling layer of OrFLow, where T is a FNO layer (Li et al., 2021], which ensures the model is resolution agnostic. Then, we have log(s'), b' Tá (hi), where sf and be are scale and shift functions, respectively, output by the äth affine coupling layer. From here, we update hi through (h') s @ há b¹.
We only require homeomorphism between DA and Du where the point alignment through the underlying homcomorphic map, and use DA to represent the discretized domain of DA- For simplicity, we consider the case where Du D and the trivial identity map is the homeomorphic map in our experimental study.
6

---

Published in Transactions on Machine Learning Research (10/2024)
Training and Inference
OpFlow
A
Invorable operator
Invertble
operater
Invertitle
oparator
Inveribll Spperatto
U
Gaussian Process space
data function space
Universal Functional Regression
log Po (Mo (Dbes)
SGLD sampling
OpFlow
density function
likelihood estimation
Posterior samples in A space
(at aB ... (g))
(u ugg ... u
Inference
OpFlow
Posterior samples in 'I space
Frozen prior
Inverse/Forward
Figure 2: Model architecture of OPFLOW, OPFLOW is composed of a list of invertible operators. For the universal function regression task, OrFLow the learnt prior, which is able to provide exact likehood estimation for function point evaluation. Tobs is the noisy observation, up is the posterior function of interest. and us Go(as), where Go is the learnt forward operator.
Finally, we output the function (r')++1 for the next layer where (v/) concat(hi (N'))) The concat (concatenation) operation corresponds the reversing the split operation, and is either in the physical domain codomain.
Inverse process. The inverse process for the ithe layer of OPFLOW summarized in Algorithm First, we split the input function (e')st) with hi (N'} = (p)it(("")**'). Then the inverse process of the affine coupling layer also takes hf as input, and generates the scale and shift functions log(s'), To (hf). Thus, both forward and inverse processcs utilize the hi as input for the affine coupling layer, which implics that the scale and shift functions derived from the forward process and inverse process are identical from the affine coupling layer. Then we can reconstruct h2 and through hi = =(h')² b')/s²; v² = concat(1; At). Finally, we have the inverse of the actnorm layer with (v) = (v* 6/)/s/- Similar inverse processes of normalizing flows are
described in (Knngma & Dhariwal
2018 Dinh al. 2017
4.2 Training
Since OpFLOw is bijective operator, only need to learn the inverse mapping Fo 4 - A, the forward operator is immediately available. We train OPFrow in a similar way to other infiiit-ddimensional generative models (Rahman et al. 2022a Seidman et al. 2023 Lim et al. 2023) with special addition of domain alignment. The space of functions for A is taken to be drawn from GP, which allows for exact likelihood estimation and efficient training of OpFLow by minimizing the negative log likellhoooood Since OPFLOW discretization agnostic (Rahman et al. 2022a, Kovachki al., (002),, can be trained on various discretizations of Dy and be later applied to new set of discretizations, and in the limit, to the whole of Du Kovachki et al. 2023 We now define the training objective L for model training as follows,
min E Eu (log (ulp))) where, logpe(ulp) logp(alpa) log det( 0(FlD);) ( -)1
(1)
9E0
The inverse operator is composed of invertible operator layers with = transforms n to ut u..... --1 v where 6d u, t² and the alignment. naturally holds within the domains
and gradually

---

Published in Transactions on Machine Learning Research (10/2024)
associated with v0, v², .- as each layer of Fø is an invertible operator. V is the collection of positions for function of with D6 D, D3- = DA, and each data point has its own discretization D
Although the framework of OPFLOW is well-defined, the negative log-likelihood for high-dinensional data may suffer from instabilities during training, leading to a failure to recover the true distribution. Through our experiments, we found that relying solely on the likelihood objective in Eq L was insufficient. for training OPFLOW successfully. To address this, we introduce a regularization term, to ensure that OPFLOW learns the true probability measure by helping to stabilize the training process and leading to faster convergence. Such regularization is inspired by the fimiite-dimeniionl Wasserstein loss used in (Rahman al.) 2022a Unlike the implicit Wasserstein loss provided by the discriminator in GANO, closed-form expression of the infinite-dimensionall Wassersiein loss in the context of OPFLoW. This is due to the learning process mapping function data onto well-understood GP. The inherent properties of Gaussian Processes facilitate measuring the distance between the learned probability measure and the true probability measure, which explained into detail in the subsequent sections.
could potentially have
A common way for measuring the distance between two probability measures is square 2- 2-aassesstein Mallasto
& Feragen 2017) distance, which is defined as
FodiPPu inf sEll A
(2)a1, (21,02) E Ax
(2)
where II is a space of joint measures defined on A x A such that for any π E II, the margin measures of xt (an, 02) on the first and second arguments are A and FodPu, and d2 is a metric induced by the I,2 norm.
By construction, A ~ GP and sufficiently trained, Feill ~ GP We can thus further simplify Eq. 2 which is named as p2 score by Rahman et al. (2022b), and is defined as:
FodPP)) dB(m, m2) + Tr(Ki + K22 - -
(3)
for which PA is chosen to be GP, (mi,,, and (m2,, is the GP fit the push-forward measure FotPu The mi. K1 are the mean function and covariance operator of GP (11mmmmm)))) whereas m2, K2 are the mean function and covariance operator of GP2(m2, We note that it is equivalent to say function f f~ GP(m. k) or f GP K.), where l is the covariance function and A. is the covariance operator (Mallasto & Feragen 2017). We take Eq. 3 as the regularization term. Furthermore, we inject noise with negligible amplitude that sampled from GP1 to the training dataset {u, (y In this way, we can have the inversion map to be well-define and stable. From here, we can define a second objective function.
C - C=-ming min Eu-P, [ogpo(u)p)] + (PAl, p)) dge
(4)
where X is the weight for the regularization term. For the collections of position DA = (v:)) Vi E DA, we select a finite set of orthonormal basis {ei, .e) on the AD) space such that ||eilla = J i (1, Thus the relationship between the covariance operator and the covariance function form can be expressed as:
< <Ken,e >= / Hee (ies(i)) = = =
(5)
DA
Thus, given the chosen set of orthonormal basis functions, the covariance matrix (covariance function in matrix form) K associated with the covariance operator K. follows the relationship:
K[i,j] = < Kes, ej >= k(y., Uj)
(6)
Unlike fiite-dimensional normalizing flows on fixed-size and regular grids, OrFLow can provide likelihood estimation pog)(())p) p) arbitrary grids D. which represents potentially irregular discretization of Du. To compute pa) need the homeomorphic mapping between DA and Du to find Da, correspouding to D. The density function in Eq. [ hollds for new points of evaluation. Our implementation of OPFLOw incorporates INO that requires inputs regular grids. Consequently, although OrFLow is designed to handle functions for irregular grids, our specific implementation of OPFL.w only allows for likelihood estimation on regular girds with arbitrary resolution.
8

---

Published in Transactions on Machine Learning Research (10/2024)
Where, ][ is the (itk: Juh) component of the covariance matrix K. Detailed explanation of the chosen basis function can be found in the Appendix A.1 in (Rahman et al. 2022b). Thus the matrix form of the Eq. can be expressed as follows,
1 7 1/ - h21l + (Tr( + K2 - 2(KF k2k 7 (11 - hall |K] - K2117)
(7)
and h1, h2 are the matrix forms of means m1, my in Eq. K1, are the covariance matrix, corresponding to K1 and K2- III3 is the square of 1.2 norm and II|? is the square of Frobenius norm. For detailed proof of the simplification of the trace of covariance matrix for two Gaussian processes in Eq. 7 please refer to (Givens & Shortt 1984). Equipped with the above preliminary knowledge, we introduce a two-step strategy for training as shown in Algorithm In the first step, we use the objective defined in Eq. with > 0, whereas in the second step, we finc-tune the model by setting À = 0 and a smaller lcarning rate is preferred. For step one, even though we have the closed and simplified form of the matrix form of square 2- 2-Wasserstein distance shown in Eq. 7 calculating the differentiable square root of the matrix is still time consuming and may be numerically unstable (Pleiss et al. 2020). Here, we directly calculate || K2ll? instead of || - -K 13, utilizing W3- = } (1lh2 - h2ll2 + K| K213), and such approximation is acceptable due to the the regularization term only exists in the warmup phase and not a light preparation. In the appendix, we show an ablation study that demonstrates the need for the regularization in the warmup phase. Furthermore, Eq. 7 implics normalize the Wasserstein distance and makes sure the square of L norm and Frobenius norm remain finite the number of discretization points increases.
Algorithm OPFLow Training
Training dataset {uylD]ー= collection of positions (z,,, (.......;)) mean vector hı and covariance matrix K1; noise level 7; random noise samples {u)]== GP1; regularization weight À Initialize the reverse OpFlow Fg
Input:
3: while warmup do
4: Random batch {uilD from {u,lo
5:
Inject GP noise: (u/D ) [u + D Construct h2, Ky: lo2 = Σ Fo(u, K2 = FFo
6:
Calculate W3 approximate regularization term: W3 +(1ii
2all2 + ||| K2ll3)
Compute the raining objective L = - Σ, log (og(o)))))) + AW3
9:
Update Fp through gradient descent of L
10:
end while
11:
while finetune do
12:
Random batch {uglo}?., from {uglo}z=
13:
Inject GP noise: {udlp) = {uu + p Computing training objective L - lugpo(u:l))
14:
n Update Fo through gradient descent of L
15:
16:
end while
4.3 Universal Functional Regression with OpFlow
Bayesian functional regression treats the mapping function itself as a stochastic process, placing a prior distribution over the mapping function space. This allows for predictions with uncertainty estimates throughout D. most importantly away from the observation points. GPR places a GP prior on the mapping function space, a common choice because of mathematical tractability. However, there are many real world datasets and scenarios that are not well-represented by a GP (Kındap & Godsill 2022 To address this, we outline an algorithm for UFR with OPFLOW in Bayesian inference problems.
In the following we assume that OFFLOW has been trained previously and sufficiently approximates the true stochastic process of the data. We thus have an invertible mapping between function spaces 4 and A. We now can use the trained OrFlow as a learned prior distribution over the space of possible mapping functions, fixing the parameters of OPFl.ww for functional regression.

---

Published in Transactions on Machine Learning Research (10/2024)
Similar to GPR, suppose we are given Mobs. pointwise and potentially noisy observations of function u, where the observation points are at the discretization set D C Dus We refer to the evaluations at D) Moba; and the additive noise e is the white noise with the law ~ o Note that this is based on the conventions in GPRs; otherwise, the noise с can also be treated as a function with a GP from. One more time, similar to GPR, we aim to infer the function's values on a set of new points D' where D C D' given the observation doos- The regression domain therefore D' which we refer to the inferred function's values as us and as Fo(us). Ergo, for the likelihood of the values on points in D', i.e., the likelihood of us. we have
log pe(us (oba)) = log pe (uobe up)) + log po(up) - log po (Uobe)
(8)
log po (ooee u D) + log pe log pel (ūobe)
(9)
Mobs U6 Dol + +oggoo r log - (og(())) - -log pe (uobe),
(10)
2x
where po(u)p) denotes the probability of evaluated on the collection of points D with the learned prior (OrFLow), and r is the cardinality of set D with [D]. Maximizing the above equation results in maximum posteriori (MAP) estimate of po(ug) (obe). Since the last three terms in Eq 10 are constants, such maximization is carried using gradient descent focusing only on the first two terms. In the following, we refer to Ug as the MAP estimate given Uobs-
For sampling for the posterior in Eq. 10 we utilize the fact that OpFLow is bijective operator and the target. function us uniquely defines Og in A space, therefore, drawing posterior samples of uy is equivalent to drawing posterior samples of ag in Gaussian space where SGLD with Gaussian random field perturbation is relevant. Our algorithm for sampling from the po(ugl uobs) described in Algorithm 2 We initialize agl Malu with the MAP estimate of po po(ug) (Uglūobe); i.e., Tg and follow Langevin dynamics in A space with Gaussian structure sample Up- This choice helps to stabilize the functional regression and further reduces the number of burn-in iterations needed in SGLD. The burn-in time b is chosen to warm up the induced Markov process in SGLD and is hyperparameter to our algorithm (Ahn 2015). We initialize the SGLD with ağ = (To) and ug Go(ag) ug. Then, at each iteration, we draw latent space posterior sample a' in AA space, where a is the tth sample of function ag, and harvest the true posterior samples u' by calling Go (a Since po [obb)) ~ GP, it very efficient to sample from it, and we find that relatively little hyperparameter optimization is needed. Since the sequential samples are correlated, we only store samples every IN time step, resulting in representative samples of the posterior without the need to store all the steps of SGLD (Riabiz) et al. 2022
A Algorithm Sample from posterior using SGLD
Input and Parameters: OPFLOW models Go. Fe, Langevin dynamics temperature T, Go. MAP T.g.
2: Initialization: dg Fo(T))) and u² = ((ag),
3: for t 0, 2......... ....No do
4: Compute gradient of the posterior: Vay log Poggo Mobs)
Update a, a a' + Y v log po (uad)) + (O,11
if 2 6 then
7: Every iN iterations: obtain new sample att,, utt = 10(a
8: end if
9: end for
5 Experiments
In this section, we aim to provide comprehensive evaluation of OrFLow's capabilities for UFR, as well as generation tasks. We consider several ground truth datasets, composed of both Gaussian and non- Gaussian processes, as well as a real-world dataset of earthquake recordings with highly non-Gaussian characteristics..
10

---

Published in Transactions on Machine Learning Research (10/2024)
Cround irud
Ooftore
Unsetwiniy compptaion
613
1281
133
0.2
-.5
-1.3
4.13
4.0
-30
-332
110 10.6011
(b)
(c)
Figure 3: OPFLOW regression GP data. (a) Ground truth GP regression with observed data and predicted
samples (b) OpFlow regression with observed data and predicted samples.
(c) Uncertainty comparison
between true GP and OpFlow predictions.
5.1 Universal Functional Regression
Summary of parameters. Here, we show UFR experiments using the domain decomposed implementation of OPFLOW, and leave the codomain OPFLOW UFR experiments for the appendix. Table 2 lists the UFR hyperparameters used, as outlined in Algorithm 2 For training OrFLow, we take A ~ GP() with a Matern covariance operator parameterized by length scale and roughness v. The mean function for the GP is set to zero over D. The noise level (o²) of observations in Eq. 10 is assumed as 0.01 for all regression tasks.
Gaussian Processes. This experiment aims to duplicate the results of classical GPR, where the posterior is known in closed form, but with OPFLOw instead. We build a training dataset for U ~ GP, taking lu = 0.5, py = 1.5 and the training dataset contains 30000 realizations. For the latent space GP, we take LA = 0.1, 0.5. The resolution of is set 128 for the 1D regression experiment. Note that, the OrFLow is not explicitly notified with the mean and covariance of the data GP, and learns them using samples, while the GP ground truth is based on known parameters.
For performing the regression, we generate single new realization of Uf and select observations at 6 random positions in D. We then infer the posterior for across the 128 positions, including noise-free estimations for the observed points. Fig. 3 displays the observation points and analytical solution for GP regression for this example, along with the posterior from OPFLOW. In general, the OPFLOW solution matches the analytical solution well; the mcan predictions of OrFLow (cquivalont to the MAP estimate for GP data) and its assessed uncertainties align closely with the ground truth. This experiment demonstrates OPFLow's s ability to accurately capture the posterior distribution, for a scenario where the observations are from a Gaussian Process with an exact solution available.
Furthermore, we present the regression results of OPFr.w compared with other Deep GP models Appendix A.4. In this comparison, OPFLoW exhibits superior performance over conventional Deep GP approach. Last, explore the minimal size of the training dataset to train effective prior. In Appendix A.I1I we detail our analysis of the 1D GP regression experiment using OpFr.ow with reduced training dataset sizes. From this investigation, we conclude that the training dataset containing between 3000 and 6000 samples (10% to 20% of the original dataset size) is sufficient to train robust and effective prior.
Truncated Gaussian Processes. Next, we apply OPFLOW to a non Gaussian functional regression problem. Specifically, we use the truncated Gaussian process (TGP) (Swiler et al. 2020) to construct a 1D training dataset. The TGP is analogous to the truncated Gaussian distribution, which bounds the functions to have amplitudes lying in a specific range. For the dataset, we take l = 0.5 and 12 = 1.5, and truncate the function amplitudes to the range [-1.2, 1.2]. For the latent space GP,, we take LA = 0.1, 0.5. The resolution is set to 128. We conduct experiment with only 4 random observation points, infer the function across all 128 positions. Unlike most non- non-Gaussian processes, where the true posterior is often intractable, the TGP framework, governed by parameters lu. Vu and known bounds, allows for true postcrior estimation using rejection sampling. We consider a narrow tolerance value of +0.10 around each observation. The tolerance range can be analogous to but conceptually distinct from noise level (Robert et al. 1999 Swiler et al. 2020). For the training datasct, we use 30000 realizations of the TGP. For the posterior evaluation, we generate 1000 TGP posterior samples, and taken the calculated mean and uncertainty from the posterior samples as the ground truths.
11

---

Published in Transactions on Machine Learning Research (10/2024)
Croundl buuh
Option
14
aro
(b)
(c)
(d)
Figure 4: OPFLOW regression TGP data. (a) Ground truth TGP regression with observed data and
predicted samples (b) OFFLOw regression with observed data and predicted samples. (c) prior GP regression
with observed data and predicted samples (c) Uncertainty comparison between true TGP and OrFLow
predictions.
(b)
(c)
(d)
(e)
trith
sa
Figure 5: OPFLOW regression on 32x32 GRF data. (a) 32 random observations. (b) Predicted mean from (g)
(f)
OPFLOw. (c) Ground truth mean from GP regression. (d) Misfit the predicted mean. (e) Misfit of predicted
uncertainty. (f) Predicted samples from OPFLOW. (g) Prodicted samples from GP regression.
Fig. shows the UFR results for this dataset alongside the true TGP posterior. We see that OPFrow effectively captures the true non-Gaussian posterior, producing samples that are visually aligned with the ground truth. For comparison, we also show the results of applying GP regression to this dataset. Notice that the GP prior allows for non-zero probability outside of the truncation range, whereas the OPF1.ww prior learns these constraints, resulting in a much more accurate posterior. Fig. reveals slight deviation in OPFLow's mean prediction starting from 0.8, yet the uncertainty remains accurately represented. This discrepancy may be explained by the challenging nature of this task, which involves using only 4 observations. With more observations, OPFLow's predictions are expected to improve.
Gaussian random fields. We build 2D dataset of Gaussian random fields having bu = 0.5 and 1.5. For the latent space, we have GP with lA 0.1, AΛ = 0.5. We set the resolution of the physical domain to 32 x 32 for the training data. We generate 20000 samples in total for training. This is another experiment with an analytical expression available for the true posterior. We generate one new realization the data for functional regression, and choose observations 32 random positions, shown in Fig 5h. We then infer the possible outcomes over the entire 32 x 32 grid. Fig shows that OPFLOw reliably recovers the posterior without assuming beforehand that the true data from Gaussian process. The error is non-uniform over the domain, the largest of which occurs in the data-sparse upper right corner. Fig 5 shows several samples of OPFLow's posterior, mirroring the true posterior variability produced by GPR. We show a second scenario for this same function the appendix, but where the observations are concentrated along linear patterns (Fig Similarly, OPFLOW faithfully approximates the true posterior.
Seismic waveforms regression. In this experiment, OPFLOW regression is applied to a real-world non- Gaussian dataset consisting of earthquake ground motion time series records from Japan Kiban Kyoshhi network (KiK-net). The raw data are provided by the Japanese agency, National Research Institute for Earth Science and Disaster Prevention (NIED 2013). The KiK-net data encompasses 20,643 time series of ground velocity for earthquakes with magnitudes in the range 4.5 to 5, recorded between 1997 and 2017. All seismic
12

---

Published in Transactions on Machine Learning Research (10/2024)
7 m
b
m p
p
(a)
(b)
(c)
(d)
th pncetarry
(I)
g
Figure 6: OPFLOW regression on seismic waveform data. (a) Ground truth waveform. (b) One predicted
sample from OPFLOW. (c) One predicted sample from the best fitted GP. (d) Predicted mean and uncertainty from OrFLOw. (e) Predicted mean and uncertainty from the best fitted GP. (f) Samples from OrFLow. (g) Samples from the best fitted GP.
signals were resampled at 10Hz and have 60 second duration, resulting in a resolution of 600 for our analysis. The time series are aligned such that / coincides with the onset of shaking. A second-order Gaussian filter was used to attenuate high-frequency wiggles. We used LA 0.05, u/ 0.5 for the latent Gaussian space A. After training the OpFlow prior, we select new (unseen) time series randomly for testing (Fig. 6a), and perform UFR over the entire domain with 60 randomly selected observation points, (Fig. 60. For comparison, we show the results for GP regression with parameters optimized via likelihood maximization. Fig G shows that a sample from OFFLOW matches the actual data, with OPFLOW accurately capturing key characteristics in the data that include various types of scismic waves arriving at different times in the time scries (Shcarer 2019). Conversely, the slandard GP model struggles with the complexity of the seismic data shown in Fig. Further, Fig. Blee illustrate that OpFrow provides a reasonable estimation of the mean and uncertainty, whereas the GP model's uncertainty significantly increases after 50 seconds due to lack of observations. Additional samples from OrFLow and GP (Fig. (Gg) further highlight the superior performance of OrFlow for functional regression of seismic waveforms.
5.2 Generation Tasks
While our emphasis in this study is on functional regression, there is still value in evaluating the generative capabilities of OrFLow. Here, we take GANO as a bascline model as it is a generative neural operator with discretization invariance. The following study demonstrates that domain decomposed OpFLow provides comparable generation capabilities similar to prior works, such as GANO. The implementation of GANO used herein follows that of the paper (Rahman et al. 2022a). In the following experiments, the implementations of the GP for the Gaussian space A of GANO is adapted to optimize GANO's performance for each specific case. Unless otherwise specified, the default GP implementation utilize the Gaussian process package available in Scikit-learn (Pedregosa et al.] 2011). More generation experiments of domain decomposed OpFl.ow and codomain OPFLOW are provided in the appendix.
GP data. For both OPFLOW and GANO, set Matern kernel parameters with 4 0.5, my 1.5 for the 1.5, 1.0 observed data space. And (A 0.05, MA 0.5 for the latent Gaussian space of OpFLOW, length scale for the latent Gaussian space of GANO, where a, are the cocfficient parameter and inverse
parameter of the specific implementation of Matérn based Gaussian process used in GANO paper (Rahman et al. 2022a Nelsen & Stuart 2021). Data resolution used here 256. Fig. 7 shows that OPFLow can generate realistic samples, virtually consistent with the true data. In contrast, GANO's samples exhibit small high frequency artifacts. Both OrFLow and GANO successfully recover the statisties with respect to the autovariance function and histogram of the point-wise value (amplitude distribution). We note that there is
13

---

Published in Transactions on Machine Learning Research (10/2024)
significant difference in model complexity here, OPFLOW has 15M parameters whereas GANO has only 1.4M. The appendix shows super-resolution experiment for this dataset.
Agtoveranc
(a)
(b)
(c)
(d)
g
Figure OPFLOw generation performance on GP data. (a) Samples from the ground truth. (b) Samples
from OpFrow. (c) Samples from GANO. (d) Autocovariance of samples from OpFrow. (e) Histogram of
samples from OPFLOW. (f) Autocovariance of samples from GANO. (g) Histogram of samples from GANO.
TGP data. We maintain the same parameter settings used in the TGP functional regression experiment, with bounds set between -1.2 and 1.2, and Matern kernel parameters 4u = 0.5, vu = 1.5 for the observed space and LA- = 0.1, = 0.5 for the latent space of OrFLow, a = 1.5, T = 1.0 for the latent Gaussian space of GANO. The only difference is that we increase the resolution to 256 from 128. Fig. g demonstrates that
OPFLOW is capable of generating realistic samples that closely mimic the ground truth samples, while GANO tends to produce samples that appear rougher in texture. Details on the super-resolution experiment for this task are available in the appendix.
(a)
(b)
(c)
(d)
(e)
(f)
gg)
Figure 8: OPFLOW generation performance on TGP data. (a) Samples from the ground truth. (b) Samples
from OFFLOw. (c) Samples from GANO. (d) Autocovariance of samples from OFFLOw. (e) Histogram of samples from OPFLOW. (f) Autocovariance of samples from GANO. (g) Histogram of samples from GANO.
Seismic Data. The dataset the same as used in the functional regression section. However for this test, we no longer use a Gaussian filter to simplify the training dataset, which makes it more challenging to learn. The results are shown in Fig. These figures demonstrate that both OPFLOw and GANO can generate visually realistic seismic waveforms. For validation purposes, we examine the Fourier Amplitude Spectrum, critical engineering metric (Shi et al. 2024b), in Fig. g The analysis shows that both models accurately learn the frequency content of the scismic velocity datasct, underscoring their potential utility in scismic hazard analysis and engineering design
(Shi et al. 2024bb.
Amplitude Spectrum
Amplttuce Spectrum
10
10a.
-inegted - to
10-1_
10-1
Gruund Truth
Opfise
6.2
0.5
(c)
(d)
(0)
Figure OPFLOW generation performance on seismic velocity data from kik-net. (a) Samples from the ground truth. (b) Samples from OpFlow. (c) Samples from GANO. (d) Fourier amplitude spectrum of samples from OrFLOw. (c) Fourier amplitude spectrum of samples from GANO
6 Conclusion
In this paper, we have introduced Neural Operator Flows LOFFLow), an invertible infiniteddimensional generative model that allows for universal functional regression. OPFLow offers a baaring-bassed solution
14

---

Published in Transactions on Machine Learning Research (10/2024)
for end-to-end universal functional regression, which has the potential to outperform traditional functional regression with Gaussian processes. This is in addition to OPFLOw comparing favorably with other algorithms for generation tasks on function spaces. However, we acknowledge that our research primarily focuses on ID and 2D datasets. Learning operators for functions defined on high-dimensional domains remains challenging and is an area that has not been thoroughly developed. Looking ahead, OPFrow may allow for new avenues in analysis of functional data by learning directly from the data, rather than assuming the type of stochastic process beforehand.
Acknowledgements
The authors would like to express their gratitude to the TMLR reviewers and the action editor for their constructive comments. We also extend our thanks to Chenxi Kong for providing the data for the Rayleigh- Bénard convection problem. Z. Ross was supported by a fellowship from the David and Lucile Packard Foundation.
References
Sungjin Ahn. Stochastic gradient meme: Algorithms and applications. University of California, Irvine, 2015.
ISBN 1-339-12430-0.
Suzanne Aigrain and Daniel Foreman-Mackey. Gaussian Process Regression for Astronomical
Time Series. Annual Review of Astronomy and Astrophysics, 61(1):329-35 3293371,, 2023. doi: 10.
103508. URL https://doi.org/// http://mmmmmaaaa.aa000000000--00 _eprint: https://doi.org// 10.1110......1... http:///r.......//maaaasssstoooo (b2
1446/ammeee-astrr00522299 1555220-103508.
Kamyar Avizzadenesheli, Nikola Kovachki, Zongyi Li, Miguel Lin-Schiafini, Jean Kossaifi, and Anima Anandkumar. Neural Operators for Accelerating Scientific Simulations and Design, January 2024. URL
http://arxiv.orrg www./ogg//////.//.///...e..ee//e aaaaaabs22230 2309. 15325 arXiv:2309.155 15325 [physics].
Lorenzo Baldassari, Ali Siahkoohi, Josselin Garnier, Knut Solna, and Maarten V. de Hoop. Conditional score-based diffusion models for Bayesian inference in infinite dimensions. Advances in Neural Information Processing Systems,
December 2023. URL https::/prooceedlngs teerrpp..c/pppr 5364. URL wttpappppr//////aa//////aot/6660 Sasrreeeeeeesstaat----ooeeeeeeee
html.
Daniel Barry. Nonparametric Bayesian Regression. The Annals of Statistics, 14(3):934-955 953, 1986. ISSN 0090-
Publisher: Institute of Mathematical Statistics.
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural Ordinary Differential Equations, December 2019. URL httt:://arxw..org/allttt 1806. 07366 arXiv:1806.07366 [cs, stat).
Jean-Paul Chiles and Pierre Delfiner. Geostatisties: modeling spatial uncertaindy, volume 713. John Wiley & Sons, 2012. ISBN 0-470-18315-2.
Andreas C. Damianou and Neil D. Lawrence. Deep Gaussian Processes, March 2013. URL http://arxiv org/abs/ 1211 0358 arXiv:1211.085 [cs, math, stat).
Masoumeh Dashti and Andrew M. Stuart. The Bayesian Approach Inverse Problems. In Roger Ghanem, David Higdon, and Houman Owhadi (eds.), Handbook of Uncertainty Quantification, pp. 311- 428. Springer International Publishing, Cham, 2017. ISBN 978-3-319- 111----499-1288--4 978-3-319-12385-1 doi:
10.1007 997-33319- 1009-12285-111 URL http:// //link springer com/10. 300- 12385-1-7
Daniel Augusto de Souza, Alexander Nikitin, S. T. John, Magnus Ross, Mauricio A. Álvarez, Marc Deisenroth, João Paulo Gomes, Diego Mesquita, and César Lincoln Mattos. Thin and deep Gaussian processes. Advances in Neural Information Processing Systems, 36, 2024. URI http:::/prooeeeedngg.... neereeddddnninnnnnrippeeeeeee Aabicaa--bbbttttttt -ataaaaaaaaaaaa-CoCCeeeeeeee Conference...ccccooooooeoeeooeee html
Volker L. Deringer, Albert P. Bartók, Noam Bernstein, David M. Wilkins, Michele Ceriotti, and Gábor Csányi. Gaussian Process Regression for Materials and Molecules. Chemical Reviews, 121((66::0073-1114) August 2021. ISSN 0009-2665. doi: 10. 10.1021/acs wa..0//aasscceeree..oooo www.ehee.eee0002222. URL httt::://doi.rgee 102/10...00//.3.. chemrev 1c00022
Publisher: American Chemical Society.
15

---

Published in Transactions on Machine Learning Research (10/2024)
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using Real NVP, February 2017.
URL wttpa/wwww...g/a////...... wtw.aa///6665.8860 arXiv:160500803 [es, stat].
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, June 2021. URL
attp://arxiv.org// hbs//bb///00
11929 arXiv:2010.1 1992 (cs).
Emilien Duponi, Yee Whye Teh, and Arnaud Doucet. Generative Models as Distributions of Functions, 04776 arXiv:2102.04776 jes, stat].
February 2022. URL http:://axxxvvoor/bbbgo
Vincent Dutordoir, Alan Saul, Zoubin Ghahramani, and Fergus Simpson. Neural Diffusion Processes, June 2022. URL http:://axiv..rg///
03992 arXiv:2206.0300 [cs, stat].
Giulio Franzesc, Giulio Corallo, Simone Rossi, Markus Heinonen, Maurizio Filippone, and Pietro Michiardi. Continuous-Time Functional Diffusson Processes, December 2023. URL
00800 arXiv:2303.00800 [cs, stat].
Takashi Furuya, Michael Puthawala, Matti Lassas, and Maarten V. de Hoop. Globally injective and bijective neural operators, June 2023. URL http:://arxdv.org/bbb////2 03982 03982 arXiv:2306.0398) [cs, stat].
Angela Gao, Jorge Castellanos, Yisong Yue, Zachary Ross, and Katherine Bouman. DecpGEM: Generalized for Blind Inversion. In Advunces in Neural Information Processing Systems, volume 34, pp. 11592-11603. Curran Associates, Inc., 2021. URL https: iew.rroeeeennnnn........o.... nwwrips.ucc.cooo
Jacob R. Gardner, Geoff Pleiss, David Bindel, Kilian Q. Weinberger, and Audrew Gordon Wilson. GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration, June 2021. URL http:
Saardv..rr//ass/800111113
arXiv:1809.11163 (es, stat].
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J. Rezende, S. M. Ali Eslami, and arXiv:1807.01622
Yee Whye Teh. Neural Processes, July 2018. URL http:///axxi..orr/aa///10...... [cs, stat].
Clark Givens and Rae Michael Shortt. class of Wasserstein metrics for probability distributions. Michigan Mathematical Journal, 31(2):231 1984. ISSN 0026-2285. Publisher: University of Michigan, Department of Mathematics.
lan J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Networks, June 2014. URL httt://arxiv.orr /rrg abs/1406. 2661 arXiv:1406.200 [cs, stat]..
Paul Hagemann, Sophie Mildenberger, Lars Ruthotto, Gabriele Steidl, and Nicole Tiajjiao Yang. Multilevel Diffusion: Infinite Dimensional Score-Based Diffusion Models for Image Generation, March 2023. URL
wttp:r/aabbb23334/77/22.. 04772v3
Marton Havasi, José Miguel Hernández-Lobato and Juan José Murillo-Fuentes, Inference in Deep Gaussian Processes using Stochastic Gradient Hamiltonian Monte Carlo, November 2018. URL http:// http://axxvv..rro org/
abs/1806. 05490 arXiv:1806.0f55 05490 (cs, stat).
Hans Hersbach, Bill Bell, Paul Berrisford, Shoji Hirahara, András Horányi, Joaquín Muñoz-Sabater, Julien Nicolas, Carole Peubey, Raluca Radu, and Dinand Schepers. The ERA5 global reanalysis. Quarterly Journal of the Royal Meteorological Society, 196((30):1999-25) 2049, 2020. ISSN 0035-9009. Publisher: Wiley Online Library.
Chin-War Huang, Laurent Dinh, and Aaron Courville. Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models, February 2020. URL http:://axxi: www.arwwwww.....aa.a/r/rraaaeeee 2002.07101 arXiv:2002.0711 000000000000011 [es, stat).
16

---

Published in Transactions on Machine Learning Research (10/2024)
Kalvik Jakkala. Deep Gaussian Processes: A Survey, June 2021. URL http:://aax.....org//////////// 12135 arXiv:2106.12135 (cs, stat].
Martin Jankowiak, Geoff Pleiss, and Jacob R. Gardner. Deep Sigma Point Processes, December 2020. URL http:///awxiiv.orrab//////////// 09112 arXiv:2002001112 [cs, stat].
Gavin Kerrigan, Justin Ley, and Padhraic Smyth. Diffusion Generative Models in Infinite Dimensions, February 2023a. URL
20880 arXiv:2212.0086 [cs, stat].
Gavin Kerrigan, Giosue Migliorini, and Padhraic Smyth. Functional Flow Matching, December 2023b. URL
http://arxiv/ www.aiii...g/bbbb.aaaao abs aegg/bbs/30055 17209 arXiv:2305.1200 [cs, stat].
Diederik P. Kingma and Prafulla Dhariwal. Glow: Generative Flow with Invertible 1x1 Convolutions, July arXiv:1807.03039 [cs, stat].
2018. URL
Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes, December 2022. URL http:
//axxi.orr//bs//1333 6114 arXiv:1312.6114 [es, stat].
Ivan Kobyzev, Simon J. D. Prince, and Marcus A. Brubaker. Normalizing Flows: An Introduction and
Review of Current Methods. IEEE Transactions on Pallern Analysis and Machine Intelligence, 43(11): 3964-3979, November 2021. ISSN 0162-8828, 2160-9292, 1939-3539. doi: 10.1 202.....0000000000002.2 1990b.09999009 arXiv:1908.0922 09255 [es, stat].
URL
Andrei Nikolaevich Kolmogorov and Albert T Pharucha-Reid. Foundations of the theory of probability: Second English Edition. Courier Dover Publications, 2018. ISBN 0-486-82159-5.
Zhifeng Kong and Kamalika Chaudhuri. The Expressive Power of a Class of Normalizing Flow Models, May 2020. URL
0000/ arXiv:2006.0088 00092 [es, stat].
Peng Kou, Feng Gao, and Xiaohong Guan. Sparse online warped Gaussian process for wind power probabilistic forecasting. Ayplied eneryy, 108:410- 428, 2013. ISSN 0306-2619. Publisher: Elsevier.
Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural Operator: Learning Maps Between Function Spaces, April 2023. URL http://arxiv.orgg /21000008210 arXiv:2108.0844 [cs, math].
Yaman Kindap and Simon Godsill. Non-Gaussian Process Regression, September 2022. URL http://arxiv org/abs/2209.0311 arXiv:2209.03111 [cs, stat].
Zongyi Li, Nikola Kovachki, Kamyar Azizzadeneshelii Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkunar. Neural Operator: Graph Kernel Network for Partial Differential Equations, March 2020. URL http:://aaa....orggab//////////a 03485 arXiv:2003.03488 [es, math, stat].
Zongyi Li, Nikola Kovachki, Kamyar Azizzadeneshelli Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier Neural Operator for Parametric Partial Differential Equations, May 2021. URL http://arxiv.ort http///axivvvorrra/////////// abs/ 08895 arXiv:2010.08895 [cs, math).
Jae Hyun Lim, Nikola B. Kovachki, Ricardo Baptista, Christopher Beckham, Kamyar Azizzadenesheli, Jean Kossaifi, Vikram Voleti, Jiaming Song, Karsten Kreis, Jan Kautz, Christopher Pal, Arash Vahdat, and Anima Anandkumar. Score-based Diffusion Models in Function Space, November 2023. URL
http://arxivv http:://aardv..r//aa/////// http://b//302..7/4// 07400 arXiv:2302.07400 [cs, math, stat].
Haitao Liu, Yew-Soon Ong, Xiaobo Shen, and Jianfei Cai. When Gaussian Process Meets Big Data: A Review of Scalable GPs. IEEE Tronsactions on Neural Networks and Learning Systems, 31(11): 4405 4423, November 2020. ISSN 2162-237X, 2162-2388. doi:
http:/NNNNSSS222...9119111 URL
https///eeexlooreieee org/ .ttp///occmmttt/88115777777
Miguel Liu-Schiaffini Julius Berner, Boris Bonev, Thorsten Kurth, Kamyar Azizzadenesheli, and Anima Anandkumar. Neural Operators with Localized Integral and Differential Kernels, February 2024. URL http:///azivv.or//aa///////
arXiv:2402. 16845 [cs, math) version: 1.
17

---

Published in Transactions on Machine Learning Research (10/2024)
Andreas Lugmayr, Martin Danelljan, Luc Van Gool, and Radu Timofte. SRFlow: Learning the Super-Resolution Space with Normalizing Flow, July 2020. URL arXiv:2006.14200
14200
[cs, eess].
Miguel Lázaro-Grodilla. Bayesian Warped Gaussian Processes. In Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012. URL httppp////pooeednng..e...r.....
Anton Mallasto and Aasa Feragen. Learning from uncertain curves: The 2-Wasserstein metric for Gaussian processes. In Advances in Newral Information Processing Systems, volume 30. Cur-
ran Associates, Inc., 2017. URL https://prooceedings www.:ee.rrriicccecoo.oee cc/paper http::/wwwecco.oo......./.c///pp Absttaactttotstttccc...c.
Razvan V. Marinescu, Daniel Moyer, and Polina Golland. Bayesian Image Reconstruction using Deep Generative Models, December 2021. URL http://arxiv.orre/ wwwp/bg/gggg..o....//// 04567 arXiv:2012.04567 [cs, eess, stat].
Juan Maroñas, Oliver Hamclijnck, Jeremias Knoblauch, and Theodoros Damoulas. Transforming Gaussian Processes With Normalizing Flows. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, pp. 1081-1089. PMLR, March 2021. URL https://prooceedngss mlr. mlr.prres. v130/maronas211 html ISSN: 2640-3498.
Bogdan Mazoure, Thang Doan, Audrey Durand, Joelle Pinean, and R. Devon Hjelm. Leveraging exploration in off-policy algorithums via normalizing flows. In Proceedings of the Conference on Robot Leurning, pp. 430 444. PMLR, May 2020. URL httops://proceediings mlr ppoess//100/mmzzuur0000. html ISSN: 2640-3498.
Nicholas H. Nelsen and Andrew M. Stuart. The Random Feature Model for Input-Oulput Maps between Banach Spaces. STAM Journal on Scientific Computing, 4336)333312-33243 January 2021. ISSN 1064- 8275, 1095-7197. doi: 10.1137/20M133990 URL http:::/aaaxww.or//aaaaaaa [physics, stat)..
10224, arXiv:2005.1027
NIED. Strong-Motion Seismograph Networks (K-NET, KiK-Net). National Research Instilule for Earth Science and Disaster Prevention, 2013.
Peter Orbanz and Yee Whye Teh. Bayesian Nonparametric Models. Eneyclopedia of machine learning, 1: 81-89, 2010.
George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshmi- narayanan. Normalizing flows for probabilistic modeling and inference. J. Mach. Learn. Res., 22(1): 57:2617-57:2680, January 2021. ISSN 1532-4435.
Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay, Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Azizzadenesheli, Pedram Hassanzadeh, Karthik Kashinath, and Animashrce Anandkumar. FourCastNct: A Global Data-driven High-rcsolution Weather Model using Adaptive Fourier Neural Operators, February 2022. http:tp: http://aaxxv...o/bbbb.sss/ wwt.:/r////bb...111... arXiv:2202. 11214 [physics].
Fabian Pedregosa, Gaäl Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathicu Blondel, Peter Prettenhofer, Ron Weiss, and Vincent Dubourg. Scikit-learn: Machine learning Python. the Journal of machine Learning vesearch, 12:2825 2830, 2011. ISSN 1532-4435. Publisher:
JMLR. org.
Jakiw Pidstrigach, Youssef Marzouk, Sebastian Reich, and Sven Wang. Infinite-Dimensional Diffusion Models, February 2023. URL https://arxxv..rreg arg/abs/2302. 10130v2
Wei Ping, Kainan Peng, Kexin Zhao, and Zhao Song. WaveFlow: A Compact Flow-bascd Model for Raw Audio. In Proceedings of the 37th International Conference on Machine Learning, pp. 7706-7716. PMLR, November 2020. URL
mlr http://ngw.......eeeeess/////pea
TSSN: 2640-3498.
18

---

Published in Transactions on Machine Learning Research (10/2024)
Geoff Pleiss, Martin Jankowiak, David Eriksson, Anil Damle, and Jacob Gardner. Fast Matrix Square Roots with Applications to Gaussian Processes and Bayesian Optimization. In Advances in Neural Information Pro- cessing Systems, volume 33, pp. 22268 22281. Curran Associates, Inc., 2020. URL https: https://proceediigg https/acpppppepeeeeeeeeeeeeeeeee
files/ http:////.///oe.ccaaocoo//////// ttt.:/aaaaaoaaaeeaaattttt...... Abstract.htar
Shengju Qian, Hao Shao, Yi Zhu, Mu Li, and Jiaya Jia. Blending Anti-Aliasing into Vision Transformer. In Advances in Neural Information Processing Syslems, volume 34, pp. 5416 5429. Curran Associates, Inc., 2021. URL http://ppoooeiiigg..eeue....eoce http:///pacc//pppe////////aa//// hosa//ash//
250D7ee22270336511100caaaa6 cusee html
Md Ashiqur Rahman, Manuel A. Florez, Anima Anandkumar, Zachary E. Ross, and Kamyar Azizzaooeshhlli Generative Adversarial Neural Operators, October 2022a. URL
arXiv:2205.00017 [cs, math].
Md Ashiqur Rahman, Jasorsi Ghosh, Hrishikesh Viswanath, Kamyar Azizzadenesheli, and Aniket Bera. PaCMO: Partner Dependent Human Motion Generation in Dyadic Human Activity using Neural Operators, November 2022b. URL http:://arxxv.orggabbb//////
16210 arXiv:2211.1211 [cs].
Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian processes for machine learning. Adaptive computation and machine lcarning. MIT Press, Cambridge, Mass, 2006. ISBN 978-0-262-18253-9, OCLC:
Marina Riabiz, Wilson Ye Chen, Jon Cockayne, Pawel Swietach, Steven A. Niederer, Lester Mackey, and Chris. J. Oates. Optimal Thinning of MCMC Output. Journal of the Royal Statistical Sociely
Series B: Statistical Methodology, (((4):1059-1081 September 2022. ISSN 1369-7412, 1467-9868. doi:
oup. http:///adaaalllooo...ccoo//ataa 10.1111/rssb.1255 12503. URL http/////////////////////aaaaaaa
Gonzalo Rios and Felipe Tobar. Compositionally-Warpd Gaussian Processes. Newral Networks, 118:235-246, October 2019. ISSN 08936080. doi: www.0166..eemmet2011..0om
URL http:://arxxv..rgg/bbb////
09665 ar 106.99590600965 (cs, cess, stat].
Christian P Robert, George Casella, and George Casella. Monte Carlo statistical methods, volume 2. Springer, 1999.
Hugh Salimbeni and Marc Deisenroth. Doubly Stochastic Variational Inference for Deep Gaussian Processes, November 2017. URL
08933 arXiv:1705.0893: [stat].
Jacob H. Scidman, Georgios Kissas, George Pappas, and Paris Pordikaris. Variational Autocncoding Neural Operators, February 2023. URL http:://aaaxtv..o///////000 10351 arXiv:2302.1035) [es, stat).
Peter M. Shearer. Introduction to Seismology. Cambridge University Press, May 2019. ISBN 978-1-107-18447-33
OSaVDWAAQBAJ.
Yaozhong Shi, Grigorios Lavrentiadis, Domniki Asimaki, Zach E. Ross, and Kamyar Azizzadenesheli. De- velopment of Synthetic Ground-Motion Records through Generative Adversarial Neural Operators. pp- 105 113, February 2024a. doi: 10.1166/9780784485316..88 012. URL
1061/978078448516. 012 Publisher: American Society of Civil Engincers.
Yaozhong Shi, Grigorios Lavrentiadis, Domniki Asimaki, Zachary E. Ross, and Kamyar Azizzadenesheli. Broadband Ground-Motion Synthesis via Generative Adversarial Neural Operators: Development and Validation. Bulletin of the Seismological Society of America, March 2024b. ISSN 0037-1106. doi: 10.1785/ 0120230207. URL https: //t::::///ooooooo 10. 178...117100100000000
Connor Shorten and Taghi M. Khoshgoftaar. A survey on Image Data Augmentation for Deep Learning. Journal of Big Data 6(1):60, July 2019. ISSN 2196-1115. doi: 10.116//44053--019--999700 URL https:///oi...... org/ 10. 1001186//40037--01-0017700
Edward Snelson, Zoubin Ghahramani, and Carl Rasmussen. Warped gaussian processcs. Advances in neural information processing systems, 16, 2003.
19

---

Published in Transactions on Machine Learning Research (10/2024)
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishck Kumar, Stefano Ermon, and Ben Poole. SCORE-BASED GENERATIVE, MODELING THROUGH STOCHASTIC DIFFERENTIAL RQUATIONS. RONTTTINN 2021.
He Sun and Katherine L. Bouman. Deep Probabilistic Imaging: Uncertainty Quantification and Multi-modal Solution Characterization for Computational Imaging, December 2020. URL http://arxiv, org/ abs/ 2010. 2010.14462
arXiv:2010.14460 [astro-ph].
Laura Swiler, Mamikon Gulian, Ari Frankel, Cosmin Safta, and John Jakeman. A Survey of Constrained Gaussian Process Regression: Approaches and Implementation Challenges. Journal of Machine Learning for Modeling and Computing, 1 2020. ISSN 2689-3967. doi: 10. 11111///aachlaaaaMoooeeeee 2020035155. URL
09319 arXiv:2006.09319 [cs, math, stat].
Ivan Ustyuzhaninov, Ieva Kazlauskaite, Markus Kaiser, Erik Bodin, Neill Campbell, and Carl Henrik Ek. Compositional uncertainty in deep Gaussian processes. In Proceedings of the 36th Conference Uncertainty in Artificial Intelligence (UAI), pp. 180-489. PMLR, August 2020.
mlr.
html ISSN: 2640-3498.
Larry Wasserman. All of nonparsmetric statistics. Springer Science & Business Media, 2006. ISBN
0-387-30623-4.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient Langevin dynamics, pp. 681 688. Citeseer, 2011.
Jay Whang, Erik Lindgren, and Alex Dimakis. Composing Normalizing Flows for Inverse Problems. In Proceedings of the 38th International Conference on Machine Learning, pp. 11158-11169, PMLR, July 2021. URL https://proccedd htt.://eeenng........ee///////// hatps///13//whagg21111 html ISSN: 2640-3498.
mlr
Yan Yang, Angela F. Gao, Jorge C. Castellanos, Zachary E. Ross, Kamyar Avizzadenesheli, and Robert W. Clayton. Seismic Wave Propagation and Inversion with Neural Operators. The Seismic Record, 1(3):126 134, November 2021. ISSN 2694-4006. doi: 00.1185/0320210000 URL hattps://dot org/ 10. 1785/0320210026
Qiongyi Zhou, Changde Du, Dan Li, Haibao Wang, Jian K. Liu, and Huiguang He. Neural Encoding and Decoding With Flow-Based Invertible Generative Model. IEEE Transactions on Cogniilive and Developmental Systems, 15(2):724 736, June 2023. ISSN 2379-8920, 2379-8939, doi: 10.1109/TCDS.2022. 3176977. URL https://aeeexppooee ieee org/bbcummnt/970026//
20

---

Published in Transactions on Machine Learning Research (10/2024)
A Appendix
A.1 Forward and Inverse processes of OpFlow
A Algorithm Forward and Inverse processes of OPFLOW
1: Input and Parameters: Input function (u') to the ith layer of OPFLow; ith actnorm layer parameterized by sg. bg: ith affine coupling layer To Discretized domain Dg for sg. by and DN for the output functions of the affine coupling layer
Forward process:
Input function: (v')²
Actnorm layer: 8² s/ (1')' b/g
5: Affine coupling layer: hi split(u log(s"), Tg(()) (10) = st
6:
Output function: (u') concat(A) (hi, (1(2)
7: Inverse process:
Input function: (0')
8:
Partition: hf, (h)} = split((((((t
10:
Inverse of the afline coupling layer: log(s"), bog(s)),b==T((ke (h())); N² ((h)) -b)/88; v² concat((,,
11:
Inverse of the actnorm layer: (u/)² (u' - bb)/
12:
Output function:(1')"
13:
dog-dettermnnnt:
14:
Actnorm layer: sum(log(s)ln.) Affine coupling layer: sum (.))
15:
A.2 Discretization agnostic property of OpFlow
In this part, we first expand the definition of neural operators and elaborate on their discrvelizatioon-ivvvraalll properties, then explain why OrFlow is also discretization invariant.
Given two separable Banach spaces A and U, a neural operator (denoted as Go) Iearns mappings between two function spaces. Go : AA- + Lt, involving imput and output functions on bounded domain real-valued spaces. Thus for an input function E A, we have Ga(a) u, where u E U. The architecture of neural operators is usually composed of three parts (Kovachki al. 2023) (1) Lifting: Converts input functions into higher-dimensionn
- Rdd space W using local operator, with d., da. (2) Kernel Integration: For = 0 to T - 1, map hidden state to the next 1i+1 by a combination of local linear, non-local integral operator and a bias function. (3) Projection: The final hidden state is projected to the output space ddd through a local operator, where dey > du.
The lift and projection parts in neural operators are usually shallow neural networks, such as Multilayer Perceptrons. For ith kernel integration layer, we have the integral kernel operator KG, with corresponding kernel function h; C(D++1 x D;; IR d xdv.), where 0:41 (y), (x(x) are defined domains D,+1, D,, respectively. Denote μ4 the measure D,, W; as the local linear opeartor, b, the bias function. Then for the ith kernel integration layer, we have
/ ki(I, ((z + ())(()) + b,()), Vy E Di+1
U (v) =
(11)
Fourier Neural Operator Li et al. 2021 simplifies Egq II using Fourier transform and achieves quasi-lines time complexity, which can be expressed
0 = (R,(F)))) + Wi2; + b
(12)
where R; are learnable parameters defined in Fourier space associated with user-defined Fourier modes, F and F-1 are Fourier transformation and inverse Fourier transformation respectively. The calculation of Eq 12 for data with arbitrary discretization involves Riemann summation. For more details of calculation of the kernel integration, please refer to Kovachki et al. 2023 Li et al. (2021): Rahman et al. (022a))
Furthermore, the formulation of OFFLOW is for function spaces, all components and operations invovled in OpFrow directly defined on function space. The likelihood computation in Eq. 1 can take place
21

---

Published in Transactions on Machine Learning Research (10/2024)
for any discretization, and morcover, the 2- Wasserstein distance in Eq. 2 is derived for function spaces, which we show how to compute given discretizations in Eq. Therefore OPFrow is discretization agnostic. We further use a series of zero-shot super-resolution experiments in Appendix A.7 to demonstrate feature of OrFLOw.
the discretizationn-nvarin
A.3 Parameters used in regression tasks
Table shows the regression parameters (explained in Algorithm 2 for all regression experiments used in this paper. The learning rate is decayed exponentially, which starts from {T. = 0}, ends with {n, = N}
Table 2:
Datasets and regression parameters
Datasets
total
burn-in itera-
sample itera- temperature of noise (T)
initial
learn-
end learning rate
iterations
lions (b)
tions (Iv)
ing
rate
(M), N)
(N)
(n, 0)
GP
4e4
2e3
10
5e-3
4e-3
TGP
4c4
2c3
10
1
5c-3
4c-3
GRF
2e4
2e3
10
1
5e-3
4e-3
Seismic
4c4
2c3
10
1
le-3
8e-4
Codomain GP
4e4
2e3
10
5e-5
4e-5
A.4 Comparison against Deep GPs on Gaussian Process example
In this experiment, we evaluate the regression performance OpFLOw against that of variational Deep GP [Salimbeni & Deisenroth, 2017) and Deep Sigma Point Processes (DSSP) (Jankowiak et al., 2020). We maintain the same dataset and hyperparameter scttings for OpFLow as those used in the Gaussian process regression example discussed in the main text. The only difference in this experiment is that it includes 10 observations. For the baseline models, use official implementations from GPyTorch (Gardner et al. 2021 and present the best available results.
For this comparison, we employ the following metries: (1) Standardized Mean Squared Error (SMSE), which
normalizes the mean squared error by the variance of the ground truth; and (2) Mean Standardized Log Loss (MSLL), first proposed in Equation 2.34 of Rassussen & Williams (006). defined as:
- logr(o) = 2 log ( + (y 0)²
(13)
282
where D represents the observation pairs (Zobes Vobe), and 2. y, 8² denote the inquired new positions, test data, predicted mean and predicted variance, respectively. We collect test data contains 3000 true posterior samples for averaging out SMSE the MSLL. As depicted in Fig and Table 3 OrFlow generates realistic posterior samples and achieves lower SMSE and MSLL, demonstrating superior performance over the current. -tate-of-the-art Deep GPs.
Table 3: Comparison with benchmark GPs
Models
SMSE
MSLL
OPFLow
0.083
-0.879
Deep GP
0.084
-0.442
DSPP
0.094
-0.673
A.5 Valid stochastic processes induced by OpFlow via Kolmogorov extension theorem
In finite dimensional spaces, normalizing flow is provided to have universal representation for any data distribution under some mild conditions (Kong & Chaudhuri 2020 Papamakarios et al. 2021). Following the fact that universal approximation of normalizing flow, we expect such universal approximation also be generalizable to OPFlow, similar to its fnite-dlimensional case. However, providing rigorous proof of
22

---

Published in Transactions on Machine Learning Research (10/2024)
Ground trush
Ooftore
Uncorianty conperiin
-1.9
-121
-115
-100
-150
-110
4.0
-22
-55
-250
-230
-231
-15
(b)
Cooo
Uncerialrry Coep
Uhcuriality
6.9
0.9
1.2-
+1.5
4.70
-115
-179
4.5
4.00
-221
(d)
(e)
(f)
g
Figure 10: OPFLOw regression on GP data. (a) Ground truth GP regression with observed data and predicted samples (b) OPFLOW regression with observed data and predicted samples. (c) Uncertainty comparison between true GP and OPFrow predictions. (d) Deep GP regression with observed data and predicted samples. (e) Uncertainty comparison between true GP and Deep GP predictions. (f) DSSP regression with observed data and predicted samples. (g) Uncertainty comparison between true GP and DSSP predictions.
universal approximation of OrFlow remains beyond the scope of this paper and we leave for future work. In this paper, we focus on demonstrating that OPFLow can induce valid stochastic processes and able to learn more general stochastic processos compared to the marginal flow described in Transformed GP. The marginal flow is based on pointwise transformation of GP values, ergo, pointwise operator. It induces valid stochastic processes and since it is limited to only pointwise transformation of GP values, it produces transformations with diagonal Jacobians (Maroñas et al. 2021]. In contrast, OPFLOW acts as a more general bijective operator that results in transformations with triangular Jacobian matrix (non-diagonal) and jointly transforms the GP values. We next formally define the problem seiting and provide proof of our statements.
Consider stochastic process Fo : Xy + Vo. For a finite sequence ab x = (xh, -/. .x) with zá E No we define the marginal joint distribution over the function values yd V0 (F)(=)), Fo(zz)). Within the framework of OPFrow, Fo is Gaussian Process and the joint distribution p(vó v8) = Pat=ng (vd v6') is a multivariate Gaussian. With a bijective operator Ge, we have Gø(b vū) - = y and Gp (n² :²) (vd : v6 ) where y y is defined over x² x=. The point alignment between _² x and ot x is determined through an underlying homeomorphic map. According to the Kolmogorov Extension Theorem (Kolmogorov
& Sharucha-Reid 2018). to establish that valid stochastic process F has p(y' y ) its finite dimensional
distributions it is essential to demonstrate that such joint distribution satisfies the following two consistency properties:
Permutation invariance. Given a permutation æ of [1....,m] the joint distribution should remain invariant when elements of {at,.... ... ",") are permuted, such that
Parga (y : v²) = Pa (y= (1) (())
(14)
Consistency in marginalization. This property states that if part of the sequence is marginalized out, the marginal distribution remains consistent with the distribution defined the original sequence, such that for m 2 1
plu' : = / plu' w mm "dy" + +1m+m lan+m
(15)
Let's first check the permutation invariance condition. The joint distribution can be expanded as follows:
ayh vn ply': :z") p(u. vt det
(16)
01y ym)
23

---

Published in Transactions on Machine Learning Research (10/2024)
Given that Go consists of t invertible layers, have Go = 9 o... Gj G. that gradually transforms (vb 96)
to (vl )² (v2 12 12) ... (v² y*). Since OPFLoW generalize RealNVP structure, the Jacobian matrix for transformations between any two adjacent layers is triangular, which implies the determination of the
Jacobian matrix is the product of its diagonal components (Eq.
8. of (Kingma & Dhariwal 2018 then
(
((vloy")) -1--11=11
for any i E {0,-... 1 1,, and det D0y 20 -2=1 1
dy
IT - dy
Thus Eq. 16 can
be simplified as:
dy dys
ply' v²) plub 16 )T
(17)
For this analysis, we adopt the torm "volume" to refer to the determinant of Jacobian matrix discussed in Eq. 16 or 17 align with the terminology used in RealNVP (Dinh et al. 2017 From Eq we have the joint
distribution under permutation as p(y"()) y p(30 (1)) (())
yo Irr
dy T6)
Given that p(u) vt) is a
dy
dug (2)
multivariate Gaussian, which is inherently permutation invariant, the volume II the order of variables, which is also permutation invariant. Thus, have:
y=
is independent of dy
dyo +()) d
dyd dy
ply : (()) = p(y0"
Mo
m(h) 11
= p(ub 36 16)
ply' yz")
(18)
In this way, we verify the permutation invariant property. Intuitively, this can be explained by the fact that the joint distribution ply' : yy) is fully determined by multivariate Gaussian and the volume. Under permutation of variables, the both the multivariate Gaussian and volume remain unchanged.
Now, let's verify the consistency in the marginalization. According to Eq the product IT dad is dy separable and since p(ut = is a multivariate Gaussian determined from Gaussian Process, the marginalization as specified in Eq naturally holds, which verifies valid stochastic process can be induced from OpFl.w. Last, the Jacobian matrix involved in OpFlow is triangular matrix, which allows OPFL.Ow to model more general stochastic processes compared to the marginal flow used in (Maroñas et al. the Jacobian of transformation is diagonal matrix.
2021 where
A.6 OpFlow regression on Gaussian random fields example
This is the second Gaussian random field regression example mentioned in Section 5.1. As depicted in Fig. OPFLOW faithfully approximates the true posterior given the observations concentrated along linear patterns. The relative error is the error normalized by the absolute max value of ground truth mean from GP regression.
(a)
(b)
(c)
(d)
(e)
mrth
Oppiow
(I)
Figure 11: OPFLOw regression on 32x32 GRF data. (a) 4 strips observations. (b) Predicted mean from OpFLow. (c) Ground truth mean from GP regression. (d) Misfit of the predicted mean. (c) Misfit of predicted
uncertainty. (f) Predicted samples from OPFLOw. (g) Predicted samples from GP regression.
24

---

Published in Transactions on Machine Learning Research (10/2024)
A.7 More generation experiments with domain decomposed OpFlow
GRF data. For both OpFLow and GANO, take the hyperparameters to be by 0.5, 1.5 for the observed space and LA V. = 0.5 for the GP defined latent Gaussian space A. A resolution of 64 x 64 is chosen. Fig. 12 illustrates that OrFlow successfully generates realistic samples, closcly resembling the ground truth in both autocovariance and amplitude distribution.
(a)
(b)
(c)
Autocovatuurce
Histogram
1.7
Gours Tuch
Coued Pugh
08
0.6-
0.3
0.5
50-
0.0
-4
03 04
2.3
0.6
07
Poodon
Vddx
(d)
(e)
(f)
(g)
Figure 12: OPFLOW generation performance on GRF data. (a) Samples from the ground truth. (b) Samples from OPFLOW. (c) Samples from GANO. (d) Autocovariance of samples from OPFLow. (c) Histogram of samples from OpFLow. (I) Aulocovariance of samples from GANO. (g) Histogram of samples from GANO.
TGRF data. We take the hyperparameters 4 0.5, m 1.;; lA 0.1, MA- = 0.5 for generating truncated 2D Gaussian random fields. The distribution is truncated between [_ with resolution of 64 64. From Figure 13 both OpFrow and GANO generale high quality samples. In Fig. 13 we also include the histogram of the GP training data before truncation, which shows that the TGRF data strongly deviates from Gaussian. While GANO cxhibits some artifacts in the generated samples, it is worth noting the covariance function and amplitude histogram are learned reasonable well, indicating that the artifacts are only present at very high
frequency.
(a)
(b)
(c)
Autotovatiarce
Hittogram
L.Aocoveriantee
Histogram
0.5-
Opf
0.5
2.5
Geffee
0.4
5.5-
GAS0
0.2
03
0.0
03
0.2
-2
0.0
5.1
322
03
114
0.5
06
(d)
(e)
(f)
(g)
Figure 13: OrFLOw generation performance on TGRF data. (a) Samples from the ground truth. (b) Samples from OpFr.w.. (c) Samples from GANO. (d) Autocovariance of samples from OPFr.w. (e) Histogram of samples from OPFLOw. (f) Autocovariance of samples from GANO. (g) Histogram of samples from GANO.
A.8 Generation with codomain OpFlow
For the codomain implementation OrFLOw, we will focus on super-resolution tasks, which do not suffer from the masking artifacts that occur with the domain decomposod setting. It is often the case that the function of interest is multivariate, i.c. the codomain spans more than one dimension. For instance, in the study of fluid dynamics, velocity fields often linked with pressure fields. Tn these cases, there exists
25

---

Published in Transactions on Machine Learning Research (10/2024)
natural partitioning among the co-domain for the affine coupling layers. However, in order to make codomain OPFr.w applicable to univariate functions, we first randomly shuffle the training dataset (1ylp) [ j==1 and partition the data into two equal parts {uilD)= and {uild)= We then concatenate these parts along the channel dimension to create a paired bivariate dataset for training, {(u, 1/)l
Codomain GP data. We found that the codomain OPFLOW requires different hyperparameters from the domain decomposed setting. Here we set 4 - 0.5, Myx = 0.5; A = 0,1, 0.5, with a resolution of 256. In particular, these values suggest that codomain OrFLOw has less expressivity than domain decomposed OPFLOW, which we attribute to the need for codomain OPFLOW to learn the joint probability measures, which is hard by nature. Given that both channels convey identical information, we present results from only the first channel for samples generated by codomain OrFLow. Fig. [4| demonstrates that both codomain OPFLOW and GANO effectively produce realistic samples, capturing both the autocovariance function and the histogram distribution accurately. Contrastingly, in the appendix, codomain OpFlOw shows great performance in super-resolution task, potentially due to the channel-split operalion preserving the physical domain's 's integrity. To the best of our knowledge, codomain OFFLOW the inaugural model within the normalizing flow framework to achieve true resolution invariance.
16
(b)
(c)
Autocovariance
Histogram
Histogram
Ground Irass
Ground
Ground hun
Opt en csamee
G480
0.3
0.3
0.3
5.2
0.2
0.-
5.0
6.1
0.0
0.1
-5.
0.0
0.0
0.2
Poston
Value
Peatton
Value
(d)
(e)
(f)
(g)
Figure 14: Codomain OPFLOW generation performance on GP data. (a) Samples from the ground truth. (b) Samples from OrFLow. (c) Samples from GANO. (d) Antocovariance of samples from OrFLow. (c) Histogram of samples from OpFrow. (f) Autocovariance of samples from GANO. (g) Histogram of samples from GANO.
1.5 and L1 0.2, 1.5, Codmain TGRF data. We set the hyperparameters to bu 0.5, vu
applying bounds of -2 and 2 to the data, with resolution of 64 x 64. Fig. 15 demonstrates that, under these parameters, both OPFLOW and GANO accurately capture the autocovariance function and histogram distribution. This experiment underscores codomain OrFLow's capability in effectively handling non- Gaussian 2D data. Additionally, super-resolutinn results included in the appendix reveal that codomain OPFLoW is free from artifacts during super-resolution tasks, showcasing its robustness.
A.9 Zero-shot super-resolution generation experiments of OpFlow
Here we present zero-shot super-resolution results with domainndeecoppoeed OPFLow and codomain OPFLow, corresponding to the experiments detailed in the Generation Tasks section. All statistics, such the autocovariance function and the histogram of point-wise values, were calculated from 5,000 realizations. Fig. OrFlow, which was trained at resolution of 256 and then evaluated at a
16 depicts the domain-decompose resolution of 512. In Fig. 1 and (b), the super-resolution samples from domain-decomposed OPFLow display a jagged pattern and show high discontinuities, Additionally, Fig. [6(c) and (d) show that both the autocovariance function and the histogram function evaluated from the domain-decomposed OrFlow diverge from the ground truths. We attribute these artifacts the domain decomposition operation, which splits the domain into smaller patches and can result in loss of derivative continuity at the edges of these patches.
26

---

Published in Transactions on Machine Learning Research (10/2024)
(a)
(b)
(c)
Hstogram
Autocovarancs
Oftoe
5.5
DANG
2.5
Drand
02-
0.2
0.2
8.1
Wa
Pastio
Vakez
(d)
(e)
(f)
(g)
Figure 15: Codomain OPFLOw generation performance TGRE data. (a) Samples from the ground truth.
(b) Samples from OrFLow. (c) Samples from GANO. (d) Antocovariance of samples from OrFLow. (e) Histogram of samples from OpFrow. (f) Autocovariance of samples from GANO. (g) Histogram of samples
from GANO
This artifact analogous to the jagged artifacts in vision transformers (ViTs), caused by the discontinuous patch-wise tokenization process Dosovitskiy et al.
2021
Qian
2021
truth
Opficm
Autacova/lance
0.3
0.5
Grondd Tuth
Spfooe
0.5
Oofion
6.1
5.0-
23
0.
61
02
0.3
0.4
0.5
3
Pooilos
10
(a)
(b)
(c)
(d)
Figure 16: Domain decomposed OPFLoW super-resolution experiment on generating GP data with resolution
of 512. (a) Samples from the ground truth. (b) Samples from OPFLOw. (c) Autocovariance of samples from
OrFLow. (d) Histogram of samples from OpFLOw
The reppriiesolution study was extended to non-Gaussian data, with results illustrated Fig. 17 Analysis of Fig. 17 leads us to similar conclusion that the fomain-dccompositionn operation introduces artifacts into the super-resolution outputs of OPFLow. This artifact is also evident in the GRF super-resolution experiment shown in Fig. 18 Here, the Nomain-decomposee OrFLow at 128 x 128, resulting in observable checkerboard patterns in the 2D super-resolution outcomes. Furthermore, the statistical analysis of the samples generated by the domain-decompest OrFLow, as depicted in Fig. 18 reveals a significant divergence from the ground truth.
trained at resolution of 64 x 64 and evaluated
However, in the super-rosolution tasks using codomain OPFLOW for both GP and non-GP experiments, as shown in Fig. 19 and 20 the jagged or checkerboard artifacts of zero-shol super-resolution performance OPFLOW are gone. This improvement attributed to codomain OrFLow maintaining the integrity of the entire input domain. Yet, this advantage does come with its own set of challenges, notably an increased difficulty in the learning process, which requires more intensive tuning efforts and may yield results that less expressive when compared to domain-decomposet OpFr.ww. Consequently, there exists trade-off between stronger lcarning capability and achieving true resolution invariance when we use different partitioning methods of OPFLow.
A.10 Ablation study
In this ablation study, we investigate the role of Wasserstein (W2) regularization on the performance of the OPFr.ww, as detailed in Eq. 4 We set the hyperparameters to lu 0.5, vy 1.5 for the observed space and
27

---

Published in Transactions on Machine Learning Research (10/2024)
Ground truth
pFlow
Autocova fiance
Histogram
1.
0.25
Smord
drand
6.
6.30
30
-15
0.00
50
02
6.1
0.6
0.6
34
63
04
36
0.
14
(c)
(d)
Figure 17: Domain decomposed OrFLOW super-resolution experiment on generating TGP data with resolution
of 512. (a) Samples from the ground truth. (b) Samples from OPFLOW. (c) Autocovariance of samples from
OFFLOW. (d) Histogram of samples from OpFLOw
Crourd truth
(a)
(b)
Histogram
1.0
0.5
Diflow
Optice
6.
0.7
64
0.
04
04
(c)
(d)
Figure 18: Domain decomposed OPFLOw super-rosolution experiment on generating GRF data with resolution of 128x 128. (b) Samples from groud truth. (c) Samples from OpFrow. (d) Autocovariance of samples from
OPFlow. (c) Histogram of samples from OPFLow
L 0.1,VA= 0.5 for the Gaussian Process (GP) defined in the latent space A, with a chosen resolution of x 64. The domain-docomposed OPFLow model was trainod under two conditions: with and without the incorporation of W22 regularization during the warmup phase. According to the results depicted in Fig, [2] incorporating W2 regularization significantly enhanced OPFLow's ability to accurately learn the statistical properties of the input function data. In contrast, the model trained without W2 regularization demonstrated notable deficiency in capturing the same statistics, underscoring the critical role of W2 regularization in the model's performance
A.11 Size of training dataset for learning prior
In this section, we revisit the 1D GP regression experiment with OrFLow to assess the minimal dataset size required to train an effective prior. We reduced the size of the training datasets to be substantially fewer than the original count of 30000 samples. Specifically, conducted tests using datasets of 3000, 6000, and 9000 samples to examine their impact on the regression performance.
As depicted in Fig. 22 the prior trained with 3000 samples showed reasonable performance. Although there observable increase in the error of the mean prediction, the uncertainty correctly captured. This demonstrates that even substantially reduced dataset size retains predictive utility. For the dataset of 6000 samples, we observed that the regression performance closely aligned with the ground truth and matched the performance of the prior trained with 30000 samples shown in the main text. This finding highlights that a training dataset size of 6000 is nearly as effective as one five times larger, significantly reducing data-collection demand while maintaining high fidelity in the prodictions. Morcover, incrcasing the dataset 9000 samples further enhanced both the accuracy and the precision of uncertainty estimation, closely approximating the performance achieved with the original dataset size of 30000.
28

---

Published in Transactions on Machine Learning Research (10/2024)
Ground truth
Opfiow
Autocovaraance
Histogram
0.4
Smord
Druad
0.2
0.2
6.8
0.4
6.
1/
06
5/2
15
(b)
(c)
(d)
Figure 19: Codomain OpFLOW super-resolution experiment on generating GP data with resolution of 512. (a) Samples from the ground truth. (b) Samples from OPFLOW. (c) Autocovariance of samples from OPFLOW. (d) Histogram of samples from OPFLOw
Gound
(a)
Autocovariance
Histogram
Osflow
6.3
0.2
33
0.4
0.5
Valua
(c)
(d)
Figure 20: Codomain OPFLOW super-resohttin experiment on generating TGRF data with resolution of 128x 128. (b) Samples from groud truth. (c) Samples from OPFrow. (d) Autocovariance of samples from
OrFlow. (c) Histogram of samples from OpFlOw
These results underscore the point that while larger datasets can enhance OPFLow's performance, comparable performances are achievable with much smaller datasets. the context of GP regression with OPFLow, training datasets ranging from 3000 to 6000 samples, which is about 10% 20% of the original training dataset size, should suffice to train robust and effective prior. Last, it's worth exploring using an untrained OPFLOW prior for estimating the posterior distribution of specific observations, where no training data is required. This approach is known as Deep Probabilistic Imaging (Sun & Bouman] 2020). Utilizing OPFr.ww in this manner may significantly broaden its applicability, particularly in scenarios where collecting training data is challenging (Aigrain & Foreman-Mackey 2023)
A.12 Regression with codomain OpFlow
For the training of the codomain OPFLOW prior, we selected hyperparameters 4, 0.5, pu = 1.5 for the observed space and (Λ = 0A = 1.5 for the Gaussian Process (GP) in the latent space. In our regression task, a new realization of u E 24 was generated, with observations taken six randomly chosen positions within the domain D. The goal was infer the posterior distribution of u across 128 positions. Further details regression parameters available in Table 2 Additionally, from the first channel of OPFLOw generated samples. As depicted in Fig. 23 codomain OPFLOw accurately captures both the mean and uncortainty of the true posterior, well as generating rcalistic posterior samples that resemble true posterior samples from GPR. However, as discussed in the previous sections, codomain OPFLow is not as effective as domain- decomposed OPFLOw for learning the prior, which highly affects the regression performance and may account for the underestimated uncertainty after 0.5 on the x-axis. This experiment confirms codomain OrFLOw can also be used for URF, which demonstrates the robustness of UFR performance across different OpFrow architectures.
only show the regression results
29

---

Published in Transactions on Machine Learning Research (10/2024)
Crourd
(a)
(b)
Ostbe Aakntto
(c)
Autecovarrance
Histogram
Autocovatinne fance
Histogram
Coard 0.4
Cound Truhh
1.0
Ocent
0.3
0.5
3.-
0.4
0.6
0.1
32-
0.2
Podzen
Vaka:
Vakas
(d)
(e)
(g)
Figure 21: Ablation study of OPFLow generation performance GRF data (a) samples from the ground truth. (b) Samples from OPFLow. (c) Samples from OpFLow-ablation (OPFLOW trained without regularization). (d) Antocovariance of samples from OrFLow. (e) Histogram of samples from OrFLow. (f) Autocovariance of samples from Ortoow-ablation. (g) Histogram of samples from LPFLow-bblation
Crounel mnth
opTiow #o0-
Dofi Nuwe
Coffees Mous 1000
023
-100
-1.00
-113
-1.0
-116
-.7
1.20
1.20
230
4.9
74
(a)
(b)
(c)
(d)
-3.5
400
210
-15
-180
-17
1.3
-12
-200
-35
210
05
M
60
66
12
(e)
(f)
(g)
OPFLOW regression on GP data with varying sizes of training dataset. (a) Ground truth GP regression with observed data and predicted samples (b) OpFrow regression with observed data and predicted samples based on the prior trained with 9000 samples. (c) OrFlow regression with observed data and predicted samples based on the prior trained with 6000 samples. (d) OPFLow regression with observed data and predicted samples based the prior trained with 3000 samples. (e) Uncertainty comparison between true GP and OrFLow predictions based on the prior trained with 9000 samples. (f) Uncertainty comparison between true GP and OPFLOW predictions based on the prior trained with 6000 samples. (g) Uncertainty comparison between true GP and OPFLow predictions based on the prior trained with 3000 samples.
Figure 22:
30

---

Published in Transactions on Machine Learning Research (10/2024)
comparison
6.2
6.5-
6.2-
-92
-6.4
-24
-0.1
-18
-38
+3
(a)
(b)
(c)
Figure 23: Codomain OPFLOW regression on GP data. (a) Ground truth GP regression with observed data
and predicted samples. (b) OpFlow regression with observed data and predicted samples. (c) Uncertainty comparison between true GP and OpFlow predictions.
31


Reviews:
[{"review_id": "AkAiQYOYqg", "reviewer": "Reviewer_7nyT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The work builds on broad current literature, providing almost review-like coverage on recent advances in several topics in Section 2. The mathematical development appears sound and the empirical experiments demonstrate the model achieves its goals, in effect confirming the methodological development to a sufficient degree. The method works well in the experiments considered and the experiments are informative and clear, considering also the generative side is a nice addition, and the Supplement provides extensive technical details that help understanding the details. From these perspectives, the paper clearly satisfies the evaluation criteria and I would recommend acceptance.\n\nHowever, I have one concern about the positioning and narrative for readers who are not inherently familiar with the concept of learnable priors and who are looking for practical tools to address the same kind of tasks they today use GPs for. I would like to see the authors make it more transparent that with the added flexibility comes quite substantial additional requirement of having relevant training data. Note that this is not a problem as such since the task is clearly interesting in the current form, I am merely asking for communicating it better. I feel the paper slightly too strongly suggests, even though implicitly, the proposed method as a better plug-in replacement for GPs and in doing so can mislead some readers, and this is getting close to the spirit of the TMLR evaluation criterion about validity of the claims. I marked that the explicit claims are supported by evidence, but from this perspective I consider it a borderline case.\n\nIn brief, your model appears to require a training data set of substantial size; for the synthetic experiments you use 30,000 examples even for the simple 1D GP and TGP cases and 20,000 for the 2D random field, and in the seismic waveform problem you assume access to 20,643 time series. While these are not massively large numbers, they are still way above what would be available in many of the use-cases where nonparametric regression building on GPs (the starting point of your story) are used. In fact, in many concrete cases there are no training instances of this kind, but rather the models need to be fit directly on the data you consider as test cases (e.g. the 60 randomly selected observation points for the new time series in the seismology case). I fully agree there are use cases where such training data can be available (like your particular historical seismic data collection), but they are not the most common ones especially in empirical sciences where the measurements are done in a laboratory.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors introduce a new form of nonparametric regression where a specific form of learnable and invertible prior over functions is used for addressing functional regression. The method builds extensively on modern literature that is covered well, and the approach is demonstrated clearly in (mostly synthetic) examples tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Practical suggestions on how to address my concern with minimal effort:\n- You could add one synthetic data experiment that explicitly quantifies how much data is needed, so that you can make a clear conclusion on when the approach can be used. For example, just run the GP example with smaller and smaller training sets and show what happens. By explicitly showing that you need a certain amount of training samples to learn a useful prior you would clearly communicate the property. Alternatively, if already a few tens/hundreds of samples would turn out to be enough (for the 1D example), you would get to state that as an additional result.\n- Explicitly mention in both Introduction and Conclusion that compared to the traditional GP regression you require additional data, and in case this data is not available the method is not applicable and GP -- despite its other simplifying limitations -- remains a practical tool of choice. Emphasise that assuming a specific type of stochastic process beforehand has its own merits, in particular processes like GP with very few learnable parameters and strong inductive biases (e.g. Aigrain&Foreman-Mackey explicitly discuss how physical prior knowledge is encoded into kernels). Check which of your motivational examples actually fit within your scope; I am not sure myself whether in the astronomical applications we would have the suitable training samples or not.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Irx4XH8Ho3", "reviewer": "Reviewer_z1Tq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "S: The idea of normalizing flow GP is very strong, and this model family has great potential. \n\nS: The writing is good and flows well. With below changes and polishing this has potential to become a nice paper.\n\nW: I’m not sure if this idea is novel\n\nW: The presentation is incomplete and insufficiently rigorous, and seems to detach itself from the conventional GP notation and community making it hard to understand how this positions in the GP literature, or where should it position itself.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes normalizing flows for random functions, which I think reduces to warping GPs with normalizing flows.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "I see multiple issues with the paper.\n\n1. The presentation is abstract and large part of the method is undefined. The paper needs to be revised extensively to make the method description self-contained and rigorous and reproducible.\n2. The paper needs to more clearly position itself to the wider GP and regression literature, and demonstrate what do we gain by framing things in the (rather elegant) functional perspective wrt more traditional ML/GP style, and discuss and acknowledge deep GP literature  \n3. The paper needs to make its task, open problem, novelties and claims explicit and precise\n4. Depending on what the claims are (not sure at the moment), the experiments need to convincingly demonstrate those. This warrants comparing against other flow/deep GPs and likely including standard GP benchmarks, and comparing against spatial GPs as well.\n5. The paper needs to argue explicitly why the contributions are interesting and novel for the research community. Currently these are vague (both empirical and theoretical)\n\nBelow are more precise change requests. I’m looking forward to seeing a major revision based on these.\n\n- The problem, task or setting of this paper is poorly described, and needs to be made explicit. The paper seems to coin a new concept of UFR, but doesn’t define what it is, and only vaguely describes it. I’m quite confused by this: I’ve not heard of UFR before and I don’t see how this differs from just regression. It’s not clear what is the task in this paper (GP regression?), what is the open problem (non-shallow GPs?).\n- The paper presents its material from a functional viewpoint, which is elegant but it’s not made clear how does this differ from the traditional ML approach. Regular GPs (and even neural networks) are all functions as well, or are they not? What do we gain from framing everything in terms of function spaces? The introduction claims that earlier methods are lacking function space priors or likelihoods. Err… what? This is obviously false. Please contextualise and contrast the paper’s ideas with more conventional ML approaches. The paper needs to clearly explain why it chooses the functional perspective, and what do you gain by doing so, and how it differs from conventional perspective.\n- The claims of this work are ambiguous and need to be made precise. The intro says that they develop a “viable” formulation. What does this mean? Didn’t we have any “viable” regression functions before? We certainly do! Again, I don’t understand where this paper is coming with statements like this. The paper needs to make its claims precise and contextualise them to the earlier works. The open problem, contributions, claims all need to be rigourous and contextualised to earlier works.\n- I think ultimately the contribution is a GP warped by a normalizing flow. This is fantastic idea! However, it already exists by Maronas et al (2020) and Klein et al (2022) and also arguably by Hegde et al (2019) and Ustyuzhaninov et al (2020). The paper needs to cite, discuss, contrast, and compare to these methods. It also needs to contrast and compare to deep GPs. There are also deep kernel methods (DKM, DKP), and spatial GPs (eg. GPRN) that seem to tackle the same problems (and thus warrant comparison or discussion).\n- Overall this paper is not self-standing: the task and model have not been defined, and perhaps these definitions are in some other papers (but none are cited for the “UFR”...). The Sec 3 does not define the task, or relate the math to GP regression. I don’t understand what the {u,a,F,G} are. I don’t understand why don’t we want to find a=u, but instead we want to find some other function a and then make it only match data through some “forward” (not explained either). The paper then says that F=G=I so nothing happens. Err… what? Section 4 seems misplaced, since it starts with some informal and vague description of the implemention details before the model has been defined in any way! Please define the task, the model and the inference all in rigorous math that can be understood without reading other papers first.\n- The eq 1 describes the normalizing flow, but I’m confused where does this come from or what is it. We already know normalizing flows, so is this just a standard NF or something different? This needs to be exposed better. The eq 1 is also quite casually presented, and needs more rigor on its derivation and showing its a valid object. It would also be good to somehow illustrate what happens here, and what each layer is doing to the base GP. \n- I can’t identity the motivation or justification for the Wasserstein stuff. The Wasserstein seems to do the same thing as the flow, so now we have two losses that both match densities through a transport. Isn't this redundant. The Wasserstein is also poorly motivated, since it’s not clear what problem it fixes (apart from normalizing flow being “insufficient”).\n- In eq 3 it is assumed that FU is a GP. This is a drastic assumption, and this is surely almost surely never true. If the F can map U to a GP, it probably already perfectly matches to A as well, and then we have already succeeded. Generally the FU is not a GP, and you can’t do this. Can you comment on this, and discuss what repercussions this has? I also don’t understand where the kernels K_1 and K_2 come from. Surely you can’t know the kernel of FU in any way.\n- I can’t follow eq 5. We have in left (i,j), in middle the j disappears, in right the j appears again. Can you explain or fix?\n- I don’t understand where the m’s and K’s are coming from.\n- Alg 1 does a bunch fo stuff that is not described in the paper. Everything that happens in alg1 needs to be properly explained in main text.\n- I don’t understand the phi stuff. Not sure what is u_phi, or what is a_phi or F here. I can’t map this notation to the regression problem or to inputs or outputs. I don’t understand where the F appears here, or what does it have to do with regression. Didn’t we assume F=I? Didn’t the eq 1 already solve the regression problem? Why are we doing some extra stuff in addition to solving eq 1? I’m quite confused of the wider setting or scheme that is implicitly assumed in this paper. Can you help me understand? Usually regression approach start by defining y = f + eps, and then state likelihood p(y|f) and prior (f) and state the problem as finding p(f|y). Can you start from the beginning and explain what you do in this paper step by step?\n- Eq 8-10 seems to be some kind fo redefinition of basic GP equations. Can you relate them back to the conventional equations, and describe how this is different (or beneficial). Why not go for the conventional notation?\n- I’m confused of alg2: isn’t this just standard NF? Are you describing something novel or known?\n- How do you compute the determinant? What kind of NF is this (discrete, continuous, planar, sylvester, etc)?\n- At experimental section I’m still not sure what UFR means, so I’m not sure what claims are you trying to demonstrate. To me this just seems GP regression and GP density fitting using output-warped GPs. In that case you need to compare to competing GP methods that do similar (ie. warped GPs, deep GPs, flow GPs). There’s also deep kernel learning and deep kernel processes that should probably be acknowledged or compared against.\n- If the paper claims to develop more flexible deep GPs, then it needs to demonstrate these on standard GP benchmarks.\n- Throughout the paper it's unclear what parts are novel and what known. Please clarify throughout.", "broader_impact_concerns": "No issues", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FZDDhj3i4W", "reviewer": "Reviewer_4VjP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "In general, the paper seems relevant, correct, and a good fit for TMLR. Below, there are several points that I would like to see improved or explained.\n\nStrengths\n\n- The idea is interesting.\n- The paper is mostly well-written, with some exceptions below.\n- The stochastic approaches are reasonable choices, despite my criticism in a few lines.\n- The experiments are well chosen, from simple experiments giving an intuition, to real world experiments, both in regression and generative tasks.\n- There are informative additional experiments and ablations in the appendix.\n\nWeaknesses\n\n- When you introduce your architechture in Section 4.1, I find the text somewhat vague. This is probably due to two reasons: (1) not all ingredients are my strength and (2) the connection between the ingredients is described without formulas. This prevents me from reimplementing your method. I understand that this might be too dry and long, but perhaps put this into the appendices and set a reference from 4.1 to the appendices? Similarly, in Section 4.2, I do not clearly understand how to get the inverse (but this is due to the missing details in Section 4.1).\n- The loss function in (4) seems somewhat ad-hoc. I would have prefered something coming from first causes.\n- The figures seem misleading, see requested changes.\n- The improvements over the state of the art seem at best incremental. (This is not a problem when submitting to TMLR.)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a non-Gaussian model for stochastic processes. This model a combination of both normalizing flows, which transform probability distributions in an invertible way, and neural operators, which allow to learn operators between function spaces. The theory is explained, the state of the art is explained, and various comparions to the state of the art is given in multiple experiments.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Why do you talk about PDEs in first paragraph of introduction? I get that you are influenced by neural operators, but this mislead me when starting to read the paper.\n- What does \"genetic\" in the footnote of page 4 mean?\n- Typesetting is suboptimal. Just in equation (1), you should apply the following suggestions:\n  - typeset $det$ as $\\operatorname{det}$.\n  - \"where\" in math mode can be read as the product w*h*e*r*e\n  - ocassionally use \\left and \\right on brackets\n  - Similar problems appear in many formulas\n- On page 4, I do not understand the sentence \"Finally, to have well-defined invertible maps between the two measures, we assume absolute continuity of measure on A and U.\" I think this sentence is not well-defined. Note that measures are not \"absolute continuous\", but \"absolute continuous w.r.t. some other measure\".\n- Why is OpFlow discretization agnostic? I find it strange that you argue that your method is this way by just giving citations.\n- Please unify the scaling of the y-axes in Figure 3.\n- The colors in Figure 5 (b)&(c) are different, but seem to be the same. It is not clear to me, what colors are being used in (f) and (g). The same holds for Figure 10.\n- The colors in Figure 5 (d)&(e) are very missleading. It seems that there is a high misfit, but this is hard to see. Please make the colors very clear and comment on the misfit. The same holds for Figure 10.\n- Capitalization in the caption of Figure 7 (c).", "broader_impact_concerns": "No concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors have made a great effort to address the comments of the reviewers and the editor. The paper has improved and now contains better-substantiated claims and further experimental results that support the claims. As a minor revision, the authors need to include the limitations of their work prominently, perhaps as part of the Conclusion. As a minimum, the authors need to say the current model only allows 1-D or 2D input dimensional datasets, as was confirmed by the authors in the discussion.
Decision: Accept (poster)