Generative Adversarial Neural Operators
Authors: Md Ashiqur Rahman
Manuel A Florez
Anima Anandkumar
Zachary E Ross
Kamyar Azizzadenesheli
Venue: Accepted by TMLR ()

Abstract: We propose the generative adversarial neural operator (GANO), a generative model paradigm for learning probabilities on infinite-dimensional function spaces. The natural sciences and engineering are known to have many types of data that are sampled from infinite- dimensional function spaces, where classical finite-dimensional deep generative adversarial networks (GANs) may not be directly applicable. GANO generalizes the GAN framework and allows for the sampling of functions by learning push-forward operator maps in infinite-dimensional spaces. GANO consists of two main components, a generator neural operator and a discriminator neural functional. The inputs to the generator are samples of functions from a user-specified probability measure, e.g., Gaussian random field (GRF), and the generator outputs are synthetic data functions. The input to the discriminator is either a real or synthetic data function. In this work, we instantiate GANO using the Wasserstein criterion and show how the Wasserstein loss can be computed in infinite-dimensional spaces. We empirically study GANO in controlled cases where both input and output functions are samples from GRFs and compare its performance to the finite-dimensional counterpart GAN. We empirically study the efficacy of GANO on real-world function data of volcanic activities and show its superior performance over GAN.

Published in Transactions on Machine Learning Research (10/2022)
Generative Adversarial Neural Operators
Md Ashiqur Rahman
rahman 70lmarn790 eduu
Deportment of Computer Science Purdue University
Manuel A. Florez
mfloreat/Baltecch.omm edu
Seismological Laboratory
California Institute of Technology
Anima Anandkumar
anéma@caltech.com edu
Computing + Mathematical Sciences California Instiuute of Technology
Zachary E. Ross
aross@callechh.c edu
Seismological Laboratory
California Institute of Technology
Kamyar Azizzadenesheli
kamyora@nvidia.com com
NVIDIA Corporation
Reviewed on OpenReview: https: //tp://ppppe openreview. net/ forum? id= X=XX12bbUUa2
Abstract
We propose the generative adversarial neural operator (GANO), generative model paradigm for learning probabilities on anfiniteddimessione function spaces. The natural sciences and engineering are known have many types of data that are sampled from infinite- dimensional function spaces, where classical fiite-diimensional deep generative adversarial nelworks (GANs) may not be directly applicable. GANO generalizes the GAN framework and allows for the sampling of functions by learning pusw-forward operator maps in infinite- dimensional spaces. GANO consists two main components, a generator noural operator and a discriminator neural functional. The inputs to the generator are samples of functions from a user-specified probability measure, e.g., Gaussian random field (GRF), and the generator outputs are synthetic data functions. The input to the discriminator is either real or synthetic data function. In this work, we instantiate GANO using the Wasserstein criterion and show how the Wasserstein loss can be computed in infinite-dimensional spaces. We empirically study GANO in controlled cases where both input and output functions are samples from GRFs and compare its performance to the finite-dimensional counterpart GAN. We empirically study the efficacy GANO on real-world function data of volcanic activities and show superior performance over GAN.
1 Introduction
Generative models are of the most prominent paradigms in machine learning for analyzing unsupervised data. To date, there has been considerable success in developing deep generative models for finite-dimensional data (Goodfellow et al. 2014 Kingma & Welling 2013 Dinh et al. 2014 Radford et al. 2015). Generative adversarial networks (GANs) are among the most successful generalive models with rich theoretical and empirical developments (Arjovsky et al., 2017 Liu et al.) 2017). The empirical success of GANs has been mainly within fniteedimennionnal data regimes; there has been relatively little progress on developing generative models for dffnitedimmnnsinall spaces-and cooortantly-fueettt spaces. This is the case despite the fact that many fields of science and engineering, including seismology, computational fluid dynamics, aerodynamics,

---

Published in Transactions on Machine Learning Research (10/2022)
G
G
G,,
G.
(a) Generator. a ((a). The input a first gets passed to a pointwise lifting operator parameterized with P. Then multiple layers of global integral operators Gi's applied which are accompanied by a few skip connections. At last, the output u is generated using final pointwise projection layer paramelerized with Q.
Cpeonrp layer
Newnl Functional Layas
G
Functional Layer
G
G
(b) Discriminator. d(u). The input first gets passed to pointwise lifting operator parameterized with P. Then multiple layers of global integral operators Gi's are applied which accompanied few skip connections. At last, the output is generated using final pointwise projection layer parameterized with Q followed by a linear integral functional layer.
Figure 1: Generative adversarial neural operator (GANO)
physics, and atmospheric sciences, work primarily with data that live in function spaces. In these settings, the observation data is mainly irregular and changing points in both space and time.
Applications of functional data are abundant in scismology. For example, for a specific region on Earth, the base stations record lataaseismoogann on the surface of Earth. These receiver centers are located on irregular grids, e.g., point clouds (e.g., more stations closer faults). The point cloud configuration also is different from region to region (Tokyo and Osaka). Moreover, due measurement and local noise, some of and off in time. It mcans that
the receivers irregular grids both in time and space. Moreover, when studying these functions, query them at any spatiotemporal point. Since the governing equations are partial differential wave equations, access to the temporal and spatial derivatives reveals information about the dynamics of physical phenomena. Generative models for the mentioned wave functions allow for sampling many potential seismic behaviors of each region of Earth, facilitating the hazard study. Similarly, weather forecasts, many recording stations the surface of Earth located on irregular grids (fewer stations on oceans than on lands) with different fidelity and frequency of observation. For these applications, scientists represent the weather condition function 2D sphere. This allows for evaluating weather conditions at any point the surface of Earth and computing the gradient and momentum of the fluid dynamics. The function representation with a learned generative model allows for accurale sampling of weather forecasts and future events.
dealing with functional data that are observed
aim to evaluate and
In this paper, we study the problem of generative models in function spaces. We propose generative adversarial neural operator (GANO), deep learning-based approach that enables the learning of probabilities function
2

---

Published in Transactions on Machine Learning Research (10/2022)
spaces, and allows for efficient sampling from such learned models. GANOs generalize the GAN paradigm to function spaces, and in particular, to separable Polish and Banach spaces. GANO, unlike traditional kernel density estimation methods, is computationally tractable, works on general spaces, and does not require the existence of a density nor the assumption of defined underlying measures for density (Rosenblatt,
1956 Parzen 1962 Craswell [965] Another line of work proposes to use neural stochastic dilferential equation (SDE) solver (Teen& & Raginsky, 2019) to generate temporal signal function with finite-dimensional co-domain (Kidger et al. 2021). ,. However, while the generated signals are infinite dimensional objects, the loss construction in the mentioned work is still for fiite-dimennsonal spaces, for grid evaluation points, therefore, making the learned generative model implicitly yet for fintt-ddimensional domains, and ergo, a special case of GAN setting. Such generative models require an underlying SDE solver to solve the temporal equation and are only designed for temporal data. The same SDE struct is also used for the discriminator models, resulting in causal discriminators, for which the optimality even for GAN setting is an open problem.
GANO consists of two main components, & generator neural operator and a discriminator neural functional. GANO architecture is empowered by neural operators, which are maps between function spaces (Li et al., 2020b). The generator neural operator receives a function sampled from a Gaussian random field (GRF) and outputs a function sample. This is in contrast to GAN, where the input is sample from finite-dimensional multivariate random variable and the output is finitedlimensional object. The efficiency of traditional sampling methods from GRF's enables GANO to be considered as computationally efficient generative model. The discriminator neural functional consists of a neural operator followed by an integral function. The discriminator receives either synthetic or real data as input and outputs a scalar. For the architecture choices in the generator, we use the efficient implementation of U-shaped neural operators (U-NO) (Rahman) et al. 2022) and use Fourier integration layers, termed Fourier neural operator (FNO) (Li al. 2020a] layers to construct push-forward maps from GRFs to the desired probability over function data. We use similar architecture for the discriminator neural functional and use three-layered neural network to implement the integral functional layer. For the adversarial min-max game, in particular, we instantiate the GANO framework by generalizing Wasserstein GAN (Arjovsky et al. 2017) setting to infinite dimension space. For the Wasserstein formulation, the discriminator neural functional is constrained to have a bounded norm in the infiniteddimensional space in terms of the Fréchet derivative operator. We propose how to impose this constraint in infinite dimensional space, which is invariant the discretization. The discretization invariance property introduces one of the main differences to GAN setting where imposing the norm raint. requires hyperparameter tuning for each resolution and discretization.
The generator in GANO is neural operator, type of deep learning model that is resolution and discretization invariant Kovachki et al. (2021). It means that, the input function to the generator can be expressed with an arbitrary discretization or basis representation, yet the generated output is a function, which can be queried at any resolution or point. Similarly, the discriminator is a neural functional with input functions that can be expressed in any resolution or basis representation. These properties follow the recent advancements in operator learning that generalize neural networks that only operate on a fixed resolution (Li ct al. 2020b Kovachki et al. 2021).
The effective dimension of the output function space can be controlled by restricting the effective dimension of the GRF, c.g. by increasing the length scale of the defining covariance function. This is in contrast to GANs where the dimension of the input space controls the dimension of the output manifold. Table I compares the settings of GANOs and GANs, Since diite-dimensional spaces are special cases of infinite- dimensionnnnii spaces, and multi-variate Gaussian is a reduction of GRFs, then, GAN is a special case of GANO when applied on fixed grids.
We construct a series of controlled empirical study to assess the performance of GANO. To maintain full control of the data characteristics and complexity of the task at hand, we generate the data itself using GRF's of varying complexities. We show that GANO can learn probability measures on function spaces. One important example is when the data is generated from mixture of GRFs; GANO reliably recovers the measure, while GAN collapses to mode. In this work, we use the Wasserstein version of GAN for
In finite dimensional spaces, it is conventional and staudard to define density with respect to Lebesgue mcasures. However, in the infinite dimensional cases considered in this paper, Lehesgue measures do not exists and a density, exists, needs to be defined with respect to a user-defined measure that the users need argue for its relevance.
3

---

Published in Transactions on Machine Learning Research (10/2022)
Table 1: GANOs and GANs
Models
GANO
GAN
Input/oute spaces
Function Spaces Gaussian Random Fields length scale, variance, energy, etc.
Euclidean spaces
Input measure Controls
Multivariate random variables dimension, variance, etc.
compassion. We show that as the bugghesss/nosinnes of the input GRF is increased, GANO properly learns to generate functions from the underlying data probability, while if the input GRF generates smooth or nearly fixed-value functions, the trained models lose the ability to properly capture the data measure.
We extend our empirical study to satellite remote sensing observations of an active volcano, where each data point is the phase of complex-valuse function defined on a 2D domain (Rosen et al., 2012). This is a real world function dataset which each data point represents ~ millimeter-scale changes in the surface of a volcano a spatial resolution of 70 meters, measured every 12 days. This dataset constitutes a noisy and challenging function dataset for GANO and GAN training. We show that GANO learns generate functions par with the real dataset while GAN fails in generating these volcanic phase functions.
We release the code to generate the data sets in the first part of the empirical study. For the purpose of bench-marking, we also release the processed volcano dataset, which is ready to be deployed in future studies. We also release the implementation code along with the training procodure.
2 Related Works
The original GAN formulation can be interpreted as an adversarial game procedure in which the Jensen-Shanmn Shannon divergence between a synthetic distribution, implicitly defined by generator model, and a real data distribution is minimized (Goodfellow et al.) 2014). However, models trained with a Jensen- Shannon objective function require substantial tuning, suffer from stability issucs, and notoriously difficult to scale (Radford et al. 2015). Considerable work has therefore been devoted to developing novel architectures, improving the formulation, and enhancing the theoretical understanding. In particular, the Wasserstein version of GAN allows for a more stable training scheme, is less sensitive to hyperparameter and architectural choices, and provides a loss function that correlates with output quality (Arjovsky et al. 2017). The Wasserstein formulation often understood as an attempt to minimize the Wasserstein or Farth Mover's 's distance between the synthetic and real data distributions. In Adler & Lunz (2018), rigorous thcoretical extension of WGANs along with theoretically grounded choices of hyperparameterss are presented, which the present paper follows. For the comparison study, choose the Wasserstein version of GAN.
There has been limited previous work on learning densities over function spaces. These works have mainly focused on non- non- parametric density estimation with 8-sequences separable Banach spaces and topological groups RRao 2010 Craswell 1965.. Heuristic kernel density estimation for infiniteddimensional spaces was also developed (Dabo-Niang) 2004). Such methods assume the existence of a density with respect to (sometimes unspecified) base measures (Lebesgue measures are undefined for dimensialmm spaces) and impose strong assumptions on the metric and similarity of the output spaces. Morcover, learning the density does not provide matching algorithmic sampling methods from such infinite-dimensional spaces. Since pure memorization using d-sequences does not exploit. the data suctuue and does not constitute particularly appealing approach, we do not consider it appropriate baseline for this study. For this study, we choose the GAN framework mainly due to its proximity to GANO, its vast success in many machine learning domains, and the lack of suitable methods for learning generative models in infinite dimensionnl spaces.
Pioncering work by [Li ct al. and introduced the concept of neural operators, novel composable architecture that is able to learn mappings between functions spaces. (Li et al. 2020a) showed that neural operators could be efficiently implemented as a scries of convolutions performed in the Fourier domain of the input function. It has also been shown that any complex operator can be approximated by neural operators, which are compositions of linear integral operators and non-linear activation functions (Kovachki et al. 2021 Neural operators have been successfully
2020b) generalized the notion of neural networks to infiniteddimensional spaces
4

---

Published in Transactions on Machine Learning Research (10/2022)
used for learning the solution spaces of Partial Differential Equations (PDE). FNOs have been used to learn
the solutions to the Accustic Wave-equation in two spatial dimensions YYang et al. 2021 Operator learning has transformed the field of physics-informet machine learning. {Li et al. 2021 2020a and improvements in the underlying architecture have allowed neural operators to learn complex solutions to multiphase flow problems (Wen et al.; 2
A set of earlier attempls are made to develop learning methods to generate function samples using point cloud and point-wise evaluation of sampled data functions. Along these, neural process (Garnelo et al. 2018 is motivated as a Bayesian framework to generate function samples. While motivated as generative model for underlying function distributions, the proposed amortized variational method aims at generating values of point sets, rather than functions. Therefore, it may come with a few limitations to be considered as a generative model for the underlying function distribution. This approach consists of an encoder model that given the point evaluation data, generates a fiite-dimennsionll vector of noise which is aimed to be close to a prior multivariate Gaussian random variable. The noise = is used as an input to an implicit neural net work, i.c., the decoder. For any 2, the implicit neural network represents a function sample that can be queried at any point on the domain. However, this early attempt does not learn the data distribution over functions and comes with a few limitations.
The proposed neural process method has limitation in the way the data is perceived, lacks expressively, and may not learn the underlying function distribution, As pointed out in the prior works (Dupont et al. (021), the proposed neural process approach perceives the point cloud data as a set of values (no metric between points), therefore it ignores the presence of the metric space which is noted as a crucial limitation in prior works (Dupont et al. 2021). Furthermore, the proposed model maps a finiteddimensional vector to an infinite dimensional space of functions. Duc to this limited input dimension, the generated functions can only cover a finite-dimensional manifold in function spaces, ergo, this method may lack the required approximation theoretic expressively to learn generating data functions. The last limitation is the fundamental issue with the formalism of the proposod neural process that prevents this method from being a generative model for the data function distribution. The proposed approach learns model to maximize the probability of observing the values on the set of points rather than learning to generate function data. This is major limitation of the proposed method and therefore, undesirable for the setting of learning function distribution. For example, consider dataset consisting of many functions with a very low resolution (a few point evaluations) and only one function in the dalaset with super high resolution (orders of magnitude more point evaluations, e.g., infinity). The objective of the neural process ignores the presence of all the function samples except the high-resolution one since it aims at maximizing the probability of point evaluation rather than the function samples. Moreover, since the formulation is Bayesian for points rather than functions, for a fixed number of function data samples, as increase the number of point evaluations (e-g.. to infinity), the prior will be ignored, and at the inference time, since the is drawn from the prior, even the generated points sample would not match the data. In the appendix, we show these limitations in a set of empirical tests, appendix A.1
Another line of an attempt. to learn generative models in function spaces proposes to accomplish the learning task in the space of implicit neural network parameterize of the given function space (Dupont et al. 2021 p. This approach proposes to train implicit neural networks to fit each data point in the data set. Ergo, for each data point, there will be a trained implicit neural network approximating it. Then a GAN model is trained to map input random vector 2, e-g., drawn from a multi-variate Gaussian to the parameters of the implicit neural network. Ergo, for each draw of z, this approach computes the parameters of an implicit neural network, resulting in function that can be queried at any point on the domain. This method requires extensive computation due to fitting many implicit neural networks and needs extensive memory to store these models. Furthermore, this model in the end is map from finitedimeensional 2 to infinite dimensional space, limiting its cover lo the space of functions. In general, the proposed approach is a generic idea that has many favorable points as opposed to the prior works and does not have the fundamental and mathematical limitation of point samplings in prior works of noural processes (Garnelo et al., 2018). However, the current setting proposed in the prior work (Dupont et al., (021) comes with a few limitations that prevent this approach to be considered as generative models of underlying function distribution.
5

---

Published in Transactions on Machine Learning Research (10/2022)
The definition of the discriminator and the gradient penalty introduces fundamental mathematical limitations that prevent the model from learning the data distribution on function space. Given the construction of the discriminator, as the function resolution increases, e.g., the resolution goes to infinity, the proposed discriminator roduces to trivial maps, outputs a function instead of a number, and lacks the discrimination power desired for the learning lask. The discriminator is implemented such that for an input function u, it computes Σ; (x:; - 2))((;)) where the summation is on the nearest neighbors and W is a learnable multi layered neural network. As the resolution goes to infinity, i.e, on a uniform grid, E; W(x; - x))(2;) + (z)))((z). Therefore, repeating such pointwise operating layers many times results in function with values at any equal to (xr)u(x), here W is the multiplication of W's at all the layers. This pointwise architecture lacks expressive discriminating power. Moreover, the output of the discriminator is a function instead of a single number, which is undesirable since the discriminator is expected to output a number. The second issue is that, as the resolution of the function increases, e.g., goes to infinity, the gradient penalty merges to infinity and the training process missos learning the data distribution. A similar limitation is also observed in period works that use the stochastic differential equations (Kidger al. 2021 to generale causal in-time data. This limitation makes the resulting models to be generative models for
spaces.
3 Generative Models in Function Spaces
One of the requirements to develop a stable model that maps an input probability measure to a general probability measure defined infinite dimensional spaces is to have infinite-dimensional input space. In this section, describe the setting of such maps and propose GANO, deep learning approach for learning generalive models in infiniteddimensionall function spaces. We propose GANO by exlending the Wasserstein GAN formulation (Guraaaaii et al., 2017) with gradient penalty term applied to an infinite-dimensionn sctting.
3.1 GANO
Let A and U denote Polish function spaces, such that for any a E A, a DA + Rda, and for E U, u : Du + Bdd, Let G denote a space of operators and for any operator G E G, we have g A- + U an operator map from A to U. Let L denote a space of functionals such that for any functional d E L, we have d:u u + IR, a functional map from U to R.
Let (A, PA) denote a probability space induced by a GRF on the function space A. Following the construction of GRF, PA is a probability measure such that for any sample PA we have that for any finite collection of points { in the domain DA, the joint probability of collection {a(x;)} follows Gaussian probability. Furthermore, let (21 (l1), Pu) denote the probability space on the function spaces U that the real data is generated from. For a given function space U, let Ut denote the dual space of U. When U is also a Banach space, and G is Fréchet differentiable, we define 0g as the Fréchet derivative of G. For the measure Fur and the pushforward measure PA under map 9, i.., GiP,, we define the Wasserstein distance as follows,
W (Puu,9PP
= sup dip(
Brg [d] - Rgg [d]
(1)
For the dual space U", we have that Lip(d) < 1 + [[ōd(u)]l- < 1, Vu E U. Therefore, write the constraint in the form of an extra penalty part in the objective function, i.c.,
inf sup Ep. [d(w)] Eg [d(w)] + AlEe (lodd|u-- 1)²
(2)
geG deL
This relaxation is similar to the relaxation proposed in improved Wasserstein GAN Gulrajani et al. (017 for finite dimensional spaces which recently have been shown to be equivalent to congestion transport MMilne & Nachman 2022). In this objective, the constraint is induced as a soft penalty. The P is an uniform mixture of the data and generated data measures, i.e., Iy == 79:P + (1 - P) for 7 ~ U[0 1], where U(0, 1] is the uniform distribution in the interval [0, 1] . Note that, while the cost functional in Eq. 2 is well defined, showing that the learned measure is indeed an approximation of Tr remains an open problem. We address this issue empirically and perform a set of experiments that demonstrate that GANO produces diverse outputs from the data probability measure. To this end, algorithm. 14 summarizes the GANO training procedure. Tn
6

---

Published in Transactions on Machine Learning Research (10/2022)
algorithm. 14 we first initialize the parameters of the generator 0g and the discriminator 0g. The learning process, at for each iteration, updates 0g and 0g for nig and na times, respectively, each with samples to approximation the cost in Eq. 2
3.2 GANO Architecture
We explain neural operators architecture maps between function spaces. We describe the input and output of the generator which itself is neural operator. We propose neural functionals that are maps from infinite dimensional spaces, e.g., function spaces, to finite-dimensional spaces, e.g., R. Neural functionals are neural operators that are followed by kernel integral functional. We deploy neural functionals to implement discriminators.
Neural operators Neural Operators are deep learning models that are the building blocks of the generator and discriminator architectures in GANO to learn maps between function spaces, and the space of reals. Given input function to neural operator 9, we first apply a pointwise operator P, parameterized with a neural network P, to compute Vo, i.e., (o(r)= P(a(z)) Vr E D. Let Do denote the domain functions for which vo is defined in. Given the application of P, we have Do = D. This point-wise operator layer is followed by integral layers. For any layer i, we have,
Vi+1 (v) =
= -/ ni (ee,))))(())daae + W.ns(v) + b.(w), Vy E Di+1
where ny is the kernel function, duc the measure in the i'th layer, W; is a pointwise operator, and the bias function. This operation followed by a pointwise non-linearity. The role of the pointwise operator W;, aside from decomposing the linear operator to local and global terms, is similar to the residual connection in residual neural networks (He et al. 2016). We deploy convolution theorem to compute this integral proposed in Fourier neural operator layer Li et al. (2020a). In particular, we write the k, y) and compute the first part of the integral operation in the Fourier domain. Let F denote the Fourier transform and F the inverse Fourier transform operations. Given a periodic function f4 (periodicity can be achieved by padding, a common practice in convolutional neural networks), the output of the layer is as follows,
Pi+1 = =F (Foi)) + Win + bb
where R is the Fourier transform of K, and for each Fourier mode k (k) is matrix of learnable parameters. To improve computation complexity, after the Fourier transforms at each layer i, we keep Fourier modes up to k-max, This allows for an elficient implementation of the layer and the presence of the residual connection W, makes sure all the Fourier components are passed to the next layer. This step, along with the residual connection W,, allows the resulting Fourier neural operators to take into account all the Fourier components al each layer.
After L above mentioned integral layers and computing vr. defined on the domain D,, DD, we apply the final pointwise operator Q, parameterized with neural network Q. It is such that for any E D, we have u(x)= = (z). When the input function is provided on a discretized domain, c.g., on grid, we use the Riemannian approximation of the Fourier transform to compute the Fourier modes at each layer. When the input is provided on a regular and uniform grid, this operation can be accomplished using fast. and mefory-efficient methods such as fast Fourier transform, resulting in efficient implementation of the corresponding neural operators.
Neural operators output functions that can be queried at any point. Furthermore, they can be applied on input functions presented in many forms, e.g,, presented as weighted sum of basis functions, or presented on a discrete set of points that includes regular and irregular grids. This property of neural operators is known as discrctization invariance (Kovachki et al. 2021). In the following, we provide the definition of discrctization invariance. Let D, denote a discretization (e.g., point cloud) of size j in DA. We call a sequence of nested
sets D2 D2 C ... C DA a discrete refinement of D and each Dj discretization of D if for any e 0,
7

---

Published in Transactions on Machine Learning Research (10/2022)
there exists number j E N such that,
D fUE æ1|2 <c).
aED;
Definition 3.1 (Discretization Insurance) For an operator G - U, where A a set of m-valued functions, let Dr Rd be L-point discretization of DA, and for any 0 E O fonile dimensional parameter space, 9 : R Lm A - U some map. We define the discrelized uniform risk as,
=sup ( - G(a)llu aEA
Given a discrele refinement (Dz) of the domain D suy G is diserelizalion-inaariic if there erists sequence of maps G2....... where G2 IRLd x IR Lm x θ + such that, for any 0 E O
lim R(g(; R) 9ul,, Dz) 0. L-000
This definition implies that, as the discretization used to present the input function becomes finer, the approximate error in the approximate operator vanishes. Tt has been proven that neural operators are discretization invariant deep learning models and traditional neural networks fall short in this desirable property.
Generator We implement the generator operator using an eight-layered neural operator model. The P point-wise operator consists of a one-layered neural network. The point- wise operator consists of a two-layer neural network. The parameter vector of the generator model is denoted by Og. The inputs to the model are samples generated from GRF defined on the 2D domain of D [0, 1Ja. The output of G, and u's are sample functions that are defined on 2D domain. In this work, we utilize the U-NO architecture (Rahman et al. 2022) for its efficiency, stability, and robustness to the choice of hyperparameters. This architecture uses skip connections between layers and increases the dimensionality the co-dimensions of the functions in the intermediate layers. Moreover, U-NO allows for highly parameterized models, a favorable property missing in the earlier Fourier neural operator models. GANO, the generator neural operator model G outputs function u given sample function a, i.e., u = G(a). Therefore, pushes the GRF measure to a measure on the data space.
Discriminator The discriminator is a neural functional that consists of an cight-layer neural operator followed by an integral functional that maps the out function of the neural operator to number in R. other words, we feed an input function u E U to the neural operator part of the discriminator compute the intermediate function h and the output of the discriminator E IR is computed as,
:= d(u) 1 ma(x))(()ddz
(3)
where the function Ka parameterized as a 3-layered fully connected neural network. Note that h is the output of the inner neural operator with as an input, therefore, h directly depends on u. The parameter vector of the discriminator model denoted by 0a. The function ka constitutes the integral functional f ka(a)) its input function. This linear integral functional the last layer is the direct
which acts point-wise generalization of the last layer of discriminators in GAN models to map function to number. In many GAN models, the last layer maps high dimensional vector to number. This step is accomplished by a or-vectr inner product. Such product, in continuum, resembles the act of linear integral functional. This also directly follows the Riesz representation theorem Walter [1974] stating that, under suitable construction, lincar functional (map from infinite dimension the space of reals) can be written as a linear integral functional. Fig. demonstrate the architecture of the generator G and the discriminator d.
inner product, i.e.,
8

---

Published in Transactions on Machine Learning Research (10/2022)
Gradient penalty In this paper, we consider the case where Du is a subset of Euclidean space and to define the function space U, we consider measure μ Du- We often use Lebesgue measure for μ in this paper. The construction of the dual space of U and the computation of the Fréchet derivative used in the penalty term Eq. 2 follow after defining jr. We represent the input function on a grid of m, x m2 (in general, it can be on any point cloud or basis function representation and following derivation follows). It allows us to use aulo-grad to compute the gradient penalty for the Wasserstein loss. Following the function space definitions, the gradient penalty using the auto-grad call of Vd(u) is implemented as Ex, (I/Vd(u)ll (a(x))))) - 1/ / /mmmm222 which is different than the finite-dimennional view in GAN. The choice of ymimg arises from the fact that we use the Lebesgue measure on Du to define the space U. This ratio resembles that the basis functions deployed represent the function on the grid of my X m2 need to be chosen and scaled according to m, due to the fact that the basis functions are functions with unit norms in the metric space U with L as the metric. It is important to note that since we compute the gradient with the consideration of the underlying metric space, the gradient computation using auto-grad is resolution invariant. Ergo, any resolution of choice can be used to train GANO models, fulfilling the premise of learning in infnite-dimensional spaces. Note that, for irregular grids where measures other than the deployed Lebesgue measures are used, this calculation should be adapted properly.
A Algorithm GANO
1: Input:
Gradient penalty weight A, number of discriminator updates per iteration ni, number of generator updates per iteration ng. number of samples per update m.
2: Init: Initialize generator parameters 0g. discriminator parameters 0a, and optimizers Opla: Opli
for each iteration t 1, ... do
4: for == .... na do
Sample {ar from PA. {u,), 1m from Pu, and {n)) from 1[
6: Compute loss L = 1 Σ, (d(u;) - dd(((a)) + X((dd(()) (1-A),,, Iue 1)2)
7: Update Oa via call to Opta(L,0a)
8:
end for
9:
for T= ng do
10:
Sample {a:}." tm from PA
11: Compute loss L 1 Σ." -d((aa))
m
12: Update 0g via a call to Optg(L,0g)
13:
end for
14:
end for
4 Experiments
In this section, we study the performance of GANO when the data is generated from a GRF. We compare the performance of GANO against GAN in this setting. The models in GANO consist of eight-layer neural operators following the architecture in (Rahman et al. 2022). The initial lifting dimension, i.e., co-dimension is set to 16 and the number of modes is set to 20. To implement the GAN bascline model, deploy convolutional neural networks, consisting of ten layers for the generator and half the size discriminator, and use Wasserstein loss for the training. For both models, we kept the number of parameters of the generative models roughly the same (20M). To train GAN models, we use GANO loss with the gradient penalty provided in the prior section. This choice is made to avoid otherwise required parameter turning for GAN loss for any resolution. We use the same grid representation of the input and output functions for the GAN and GANO studics. For training, we use Adam optimizer (Kingma & Ba 2014 and choose a 2D domain of [o, 1]2 to be the domain where both input and output functions are defined on. For the empirical study, GAN trained, optimized, and tested on a given discretization. Despite the fact that GANO can be trained and tested any discretization, to make the comparison on par with GAN, we limit GANO experiment to the same discretization as GAN. It is worth noting that, GANO generates samples of functions that can be queried at any point and the GAN setting does not allow for it. Since GAN is not resolution invariant and does not. generate function samples, it fails to be applicable to the general setting of function spaces.
We then study the effect of the roughness and smoothness of the input GRF on the quality of learning probability measures on function spaces for which use the resolution of 64 for each dimension. Tastly,
9

---

Published in Transactions on Machine Learning Research (10/2022)
(a)
Data GRF samples
(b) GAN generated samples
(c) GANO generated samples
CX
43
(d)
(f)
(g)
Figure 2: The input function sample is CRF and the data generated from another GRF. (a) The samples of data GRF. (b) The samples of generated data from GAN model.
(c) The samples of generated data from
GANO model. (d) GAN Auto correlation. (e) GAN histogram.
(f) GANO auto correlation. (g) GANO
histogram
we study the performance of GANO on a real-world remote sensing datasct of an active volcano for which we use the resolution of ~22 for each dimension. This is a challenging dataset with often times very low signal-to-noise noise ratio. For the choice of GRF, choose the efficient implementation of Matérn based Gaussian process (Nelsen& & Stuart 2021 parameterized with the inverse length scale.
GRF data. For the setting where data is generated by sampling from GRF, we use a dataset of random functions drawn from GRF with length scale
(somewhat smooth functions). We use GAN and GANO approaches to learn the data GRF. We train the generative models using the inputs sampled from the same GRF Fig. 2 Fig. 2a demonstrate the sample data. Subsequently, Figs. 2b and [c] demonstrate the generated samples of GAN and GANO models respectively. To analyze the quality of the generated functions, we compare the auto-correlation and histogram of point-wise function values of the generated data and the true data, Fig 2 The r-axis in the histogram plots denote the values the functions take. The x-axis in the auto-correlation plots denote the positional distance of the points on the domain Der for which we compute the auto-correlation. We observe that GANO properly recovers the statistics of the data GRF in terms of anto-correlation, Fig. 2d 21 and the histogram of the generated function values, Figs. 2e and We observe that, while the GAN approach provides smoother-looking functions, the functional statistics fail to be exact.
Mixture of GRFs data. For this experiment, aim to learn to generate data from mixture of GRFs. The training data is generated with equal chance from either GRF with fixed mean function of 1 or -1, and both with == 1. We use GAN and GANO approaches to learn the data probability measure, where the input functions are sampled from a mean zero GRF with T = 1, Fig. 3 Fig. 2a demonstrate the sample data. Subsequently, Figs. 26 and 2c demonstrate the gencrated samples GAN and GANO models respectivoly. The aulo-correlation and histogram point-wise function values generated data and the true data are 2 2e and 2g] As we observe, GANO properly recovers the statistics of the data GRF in terms of functional statistics of auto-correlation and histogram. Similar the previous experiment, we observe that the GAN approach provides mootherr-ookingg functions, but in terms of the functional statistics, it drastically underperforms GANO.
provided Figs, 2d
In the previous two experiments, we observed that GANO enables us to learn measures on function spaces and generate samples that match the functional statistics of the underlying data. In the following, examine the importance of the choice of input GRF on the performance of GANO.
GANO and the length scale of the input GRF. In GANO, when the GRF input to the generative model is very smooth (compared with the output GRF), we expect the generator to fail to learn a proper map. We expect this to be the case because the input lacks sufficient. high-frequency components, and this smoothness
10

---

Published in Transactions on Machine Learning Research (10/2022)
(a) Data GRF samples
(b) GAN generated samples
(c) GANO generated samples
GM
10-
125
3.50
022
C54
64
02
04
05
622
(d)
(e)
(f)
(g)
Figure The input function sample GRF and the data generated from a mixture of CRFs, (a) The samples of data from mixture of GRFs. (b) The samples of generated data from GAN model. (c) The samples of generated data from GANO model. (d) GAN Auto correlation.
(e) GAN histogram. (f) GANO
Auto correlation. (g) GANO histogram
(a) Imput GRF samples
(b) Data GRF samples
(c) GANO samples
GALD
SALD
92
32
-9.3
-25
30
(d) GANO Histogram
(e) GANO Auto Correlation
Figure 4: GANO trained on smooth data (r = 5) with rougher input GRF (r = 7)
prevents the generator from generating high-frequency and rough output functions. On the contrary, we expect that when the input GRF is much rougher than the data GRF and contains many high-frequency components, the generator would have an easier task to generate output functions. Therefore, the length scale and smoothness of the input GRF can play a role in regularizing GANO model, a very similar role that the dimension of the input multivariate Gaussian plays in the GAN approach.
We first show that when the input GRF is rougher (r 7) and contains more high frequency components than the output GRF (7 5), GANO successfully learns to generate samples with similar statistic of data GRF, Fig. 4
When the output and input GRF are identical measures 57 GANO still successfully learns to generate samples with similar statistics of the data GRF, Fig. 5 However, this setting requires more delicate hyper parameter tuning and requires more training epochs to converge. It is worth noting that, with proper choices of the spaces, an identity map may also be a solution.
Lastly, when the imput GRF is smoother (r = 3) than the functions samples in the output data GRF (T = 5), the generative model fails to recover higher order statistics, including the auto correlation Fig In this experiment the input function much simpler than the output functions. This study suggest that, when the real function data is very complex, very noisy, contains varying high frequency components, and poses high
11

---

Published in Transactions on Machine Learning Research (10/2022)
(a) Inpul GRF samples
(b) Data GRF samples
(c) GANO samples
1996
(d) GANO Histogram
(e) GANO Auto Correlation
Figure GANO trained on same GRF as input and data (T = 5)
(a) Input GRF samples
(b) Data GRF samples
(c) GANO samples
35
05
0.4
1936
Poxttin
(d) GANO Histogram
(e) GANO Auto Correlation
Figure 6: GANO trained on rougher data (T = 5) with smoother input GRF (r (r=3) 3)
entropy, it is crucial to provide the generator with on par GRF, On the contrary, when the function data at hand poses smoother behavior, smooth GRF suffices for training generator.
Inputs and outputs of the generator in GANO are functions The GANO framework is based on neural operators that are discretization invariant maps between functions spaces 3.1 The inputs to the generator neural operator model in GANO are functions and following the discretization invariance property of such models, these input functions can be provided to the generator in any discretization, and in particular in any mesh, grid, and resolution. In addition, the generator outputs functions, therefore, by definition, the outputs can be queried at any point in the domain. In the following empirical study, demonstrate these properties of neural operators. We train the GANO models one resolution and test the trained generator in another resolution. In particular, we consider a setting where for the training, the input GRF samples (with - = 5) are presented on a 64 x 64 grid on the two-dimensional domain. Moreover, the training data functions are draws from a GRF, with the same parameter as the input GRF, and samples are represented on the same 64 x 64 grid.
After training, assess the above-mentioned properties of neural operators. We double the resolution of the input GRF samples and present them in 128 x 128 grid. We provide these higher-resolution inputs to the generator lo generate output functions. We evaluate the generated functions on a finer grid of 128 x 128. Figure 7 demonstrates the result of the study. Figure (c) represents high-resolution data, and figure (c) represents the generated samples on the higluroresolution input and query points. Figures (g) and (i) demonstrate the histogram and auto-correlation of the higher-resolutie data and higher-resolution generated samples. This study expresses that neural operators take inputs at any resolution and the
12

---

Published in Transactions on Machine Learning Research (10/2022)
(a) Inpul GRF samples
(b) Data GRF samples at 64 x 64 (c) Data GRF samples at 128 x 128
(d) GANO generated samples on 64 x 64
(e) GANO generated samples on 128 x 128
021
42-
09
33
(f) GANO Histogram on 64 x 64
(g) GANO Histogram on 128 x 128
0.33
4.5
0.
0.
0.4
(1)) GANO Auto Correlation on 64 x 64
(i) GANO Auto Correlation on 128 x 128
Figure 7: We train GANO on a function data set of resolution 64 x 64. The data samples are generated using a GRF (T= 5). The input GRF (r= = 5) sample functions are also represented on a 64 64 grid. The generator neural operator takes a function as an input and outputs a function. To demonstrate this fact, we test the trained generative model on a different resolution. We change the resolution of the input function to higher resolution of 128 x 128 and query the generated function samples 128 x 128. Figure (c) represents high-resolution data, and figure (c) represents the generated samples on the higher-resolution input and query points. Figures (g) and (i) demonstrate the histogram and auto-correlation of the higher-resolution data and higher-resolution generated samples. This study expresses that neural operators can take inputs at any resolution and the output function can be queried at any point in the domain. Furthermore, despite the fact that the model has never seen high-resolution data during the training, it can generate statistically matching samples of high resolution.
higher resolution of
output function can be queried at any point the domain. Furthermore, despite the fact that the model has never seen high-resolution data during the training, can generate statistically matching samples of high resolution. These desirable properties of the GANO framework, as the first generative model on function spaces. Please note that, for this empirical study, we use smaller models in GANO in order to fit the high-resolution data to the present GPU machines. In particular, we reduce the number of layers to 5, the co- co-dimension to 8, and the number of modes to 10. These choices for the smaller model did not alter the performance of the trained generator.
Volcano deformation signals in InSAR data. Interfcrometric Synthetic Aperture Radar (InSAR) is a remote sensing technology used to measure deformation of Earth's surface, often in response to volcanic eruptions, earthquakes, or subsidence due to excessive groundwater extraction. In InSAR, a radar signal is emitted from satellites or various types of aircraft and echocs are recordod. Changes these echoes over time (as measured by repeat flyovers) can be used to preciscly measure the amount that a point on the surface moves between repeats. The most common form of InSAR data is the interferogram, which
13

---

Published in Transactions on Machine Learning Research (10/2022)
(a) Real samples
(b) GAN samples
(c) GANO samples
GAN
GAN
GAND
2,0
GAND
Hittogram
Histrgram
1.5
1.0
0.5
0.5
0.0
0.8
value
(d) Circular variance
(e) Circular skewness
Figure GANO and GAN samples of InSAR data for Long Valley Caldera
is an angular-valued spatial field u E U, with u(x) E [- and x E D. Interferograms are known to be highly-complex functions because they exhibit many modalities, types of noises, and patterns that depend strongly on local atmospheric and topographic conditions. Additionally, since the values are angles on [-T,, T]. if the change between two echoes is large enough, the angles can wrap around.
We produce dalaset of 4096 data points from raw interferograms, each in a grid of 128 x 128, from the Sentinel-1 satellites covering the Long Valley Caldera, which is an active volcano near Mammoth Lakes, California. We processed the InSAR dunctions/imagee from 2014-Nov to 2022-Mar, covering an area around Long Valley Caldera (approximately 250 by 160 km wide) using the InSAR Scientific Computing Environment (Rosen al. 2012 The stack of SAR functions is co-registered with pure geometry (precise orbits and digital clevation model) and the network-basod enhancod spectral diversity approach. Then, we pair each function (277 in total) with its three nearest neighbors in time to form 783 initial interferograms with pixel spacing of 70 m. Finally, we subset each interferogram into six windows non-overlapping windows of 128 x 128 grid. Examples of real samples are shown in Fig.
publicly provided by the European Space Agency,
We train GANO on the intire datasel of 4096 inteferograms. Generated samples are shown in Fig, Sc where it is clear that many the complexities of this dataset have been learned. One of the types of noise in interferograms results from decorrclation of the radar signal between repeat flyovers, and the most extreme case, can lead a stochastic process that random uniform on [- that covers part all of the image. GANO is able to learn an effective operator that approximates this complex behavior. We quantitatively
14

---

Published in Transactions on Machine Learning Research (10/2022)
evaluate the quality of the learned samples using circular statistics, which is necessary since these functions are angular-valued. Analogously to traditional random variables, there are moments of circular random variables. For a collection of N random angular variables, {0.]/=1 define
Then, Rp = l and Pp = arg(2p). The circular variance is then given by a Σn e where i = v-I. R1, and the circular skewness is given by, R2 sin()2-222) Figs. Sd and 8c show the performance of GANO w.r.t circular (1-R: )3/2
variance and circular skewness. These results demonstrate the suitability of GANO framework on learning complex probabilities on function spaces and emphasizes the data efficiency of this framework.
For the comparison study, train GAN model on the same data set. Despite extensive hyperparameter tuning, the GAN model fails to learn to generate proper samples of functions. Fig. [bb demonstrates samples generated using a trained GAN model. The generated samples do not resemble the true samples, neither perceptually nor with respect to the circular variance and skewness Fig. [dd Se This study establishes the importance of learning the generative model directly in function spaces using global kernel integration instead of local kernels.
5 Conclusion
We propose GANO, generative adversarial learning approach for learning probabilities on function spaces and generating samples of functions. GANO generalizes GAN, an established and powerful method for learning generative models on fnito-dimensional samples. GANO framework consists of two models, a generator operator and a discriminator functional. We use the neural operator framework directly model the generator and deploy the ideas from neural operators, and propose new deep lcarning paradigm, namely neural functional, for the discriminator. We empirically show that the GANO framework is suitable for dealing with function spaces. We show that the imput to the generative model can be chosen to be GRF for which the length scale controls the diversity of the pushed mcasure. We release the code, package, datasets, and the results of this study for future reproducibility.
Acknowledgments
The authors would like to thank the TMLR reviewers and the action editor for their constructive comments. Part of this research is developed when K. Avizzadenesheli was with the Purduc University. A. Anandkumar is supported in part by Bren endowed chair.
15

---

Published in Transactions on Machine Learning Research (10/2022)
References
Jonas Adler and Scbastian Lunz. Banach wasserstein gan. Aduances in Neural Information Processing Systems, 31, 2018.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pp. 214 223. PMLR, 2017.
KJ Craswell. Density estimation in a topological group. The Annals of Mathematical Slatistics, 36(3): 1047-1048, 1965.
Sophie Dabo-Niang. Kernel density estimator in infinite-dimensionall space with rate of convergence in the case of diffusion process. Applied mathematics letters, 17(4):381-38) 386, 2004.
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint 14 10.8516, 2014.
Emilien Dupont, Yee Whye Teh, and Arnaud Doucet. Generative models as distributions of functions. arXiv preprint arXiv:2102.04 776, 2021.
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo Rezende, SM Eslami, and Yee Whye Teh. Neural processes. ar preprint orXi::1807.01 0122,, 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley Farle, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in newral information processing systems, 27, 2014.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vineett Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. Advances in neural information processing systems, 30, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Patrick Kidger, James Foster, Xuechen Li, and Terry J Lyons. Neural sdes as infinile-dimensionnal gans. In International Conference on Machine Learning, pp. 5453 5463. PMLR, 2021.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6988 2014.
Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 2013.
Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Arizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces. arXiv preprint NVi::2108.08481 2021.
Zongyi Li, Nikola Kovachki, Kamyar Aviwadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. arXic preprint Xix::22010. 08895, 2020a.
Zongyi Li, Nikola Kovachki, Kamyar Azizzadeneshelli Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Graph kernel network for partial differential equations. arXio preprint arXiv:2003.08 03485, 2020b.
Zongyi Li, Hongkai Zheng, Nikola Kovachki, David Jin, Haoxuan Chen, Burigede Liu, Kamyar Azizzadenesheli, and Anima Anandkumar. Physics-informed neural operator for learning partial differential equations. arXiv preprinl XrXiv:21 11. 1. .3799,, 2021.
Shuang Liu, Olivier Bousquet, and Kamalika Chaudhuri. Approximation and convergence properties of generative adversarial learning. Advances in Neural Information Processing Systems, 30, 2017.
16

---

Published in Transactions on Machine Learning Research (10/2022)
Tristan Milne and Adrian I Nachman. Wasserstein gans with gradient penalty compute congested transport. In Conference on Learning Theory, pp. 103-129. PML.R, 2022.
Nicholas H Nelsen and Andrew M Stuart. The random feature model for input-output maps between banach spaces. SLAM Journal Scientific Computing, (000))3222-A324435 2021.
Emanuel Parzen. On estimation of probability density function and mode. The annals of mathematical statistics, 33(3):1065-1076 1962.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised Representation Learning with Deep Con- volutional Generative Adversarial Networks. arXiv preprint arXiu:1511.0644 064444 November 2015. doi:
11.45550/arXiv.11111.
URL htttps//aaraa...ogggaa//////////
Md Ashiqur Rahman, Zachary E Ross, and Kamyar Azizzadeneshelii U-no: U-shaped neural operators. arXie preprint arXiv22044.11197 2022.
BLS Prakasa Rao. Nonparametric density estimation for functional data by delta sequences. Brazilion Journal of Probubility and Statistics, 24(3):468 478, 2010.
Paul A Rosen, Eric Gurrola, Gian Franco Sacco, and Howard Zebker. The insar scientific computing environment. In EUSAR 2012; 9th European conference on synthelic operture rador, pp. 330-733. VDE, 2012.
Murray Rosenblatt. Remarks on Some Nonparametric Estimates of Density Function. The Annals of Mathematical Statistics, 27(3):832 - 837, 1956. doi: 10.1214/20ms/11777/ 10. 25/14/aams/17777211111
URL httptttt/t.i...ooo
Belinda Tzen and Maxim Raginsky, Neural stochastic differential equations: Deep latent gaussian models in the diffusion limit. orXiv preprint Xrxiv:1905. (0t:099900988,,, 2019.
Rudin Walter. Real and complex analysis, 1974.
Gege Wen, Zongyi Li, Kamyar Azizzadenesheli, Anima Anandkumar, and Sally M Benson. U-fno an enhanced fourier neural operator based-deep learning model for multiphase flow. arXiv preprint orXiv:2109 2rX.0:2109.03697, 2021.
Yan Yang, Angela F' Gao, Jorge C Castellanos, Zachary Ross, Kamyar Arizzadenesheli, and Robert W Clayton. Scismic wave propagation and inversion with neural operators. The Seismic Record, 1(3):126-134, 2021.
A Appendix
A.1
Neural Process and generative functions
In this section, expand the discussion neural process approach (Garnelo et al 2018) and show few problems it may have in learning generative models for data function distribution. For given function sample u, let (x,, v:)) denote its point evaluation representation, where for any 4;, a collocation point, yi is the point evaluation of the function at point z; For this construction, the following is the evidence lower-bound objective function proposed in neural process,
p(2) q( (ra, v:
logo ((ul(((((()))) 2 (o
Σ log (2)l2,))) + log
where the method learns the encoder q and a decoder map from (z, to mean and variance of p ()) This objective function maximize the probability ye8, and does not give a formulation to learn the data function distribution.
Let's consider a trivial setting where the function distribution is a Dirac, meaning that, the data consists of many repetitions of single function. For example, consider the function u(x) 0.5 the interval
17

---

Published in Transactions on Machine Learning Research (10/2022)
[T] The dataset consists of many sample functions, all are the mentioned Following the Bayesian and variation form of this objective, when is small, e-g., n = 100 Fig. Da training this model results in a function distribution around the input function but does not collapse on the data function, Figure 9b
For reasonable generative model, we expect that, we increase the function resolution, e.g., taking it infinity, the function distribution learning approach to get better at learning a sensible generative model. However, in the hcuristic neural process approach, as increase the function resolution, the first term in the objective function dominates (goes to negalive infinily); and ( we))") no longer can be replaced by p(z) in the inference time. Figure 9c shows data function with 10000 point evaluations, and when neural process is trained on 10000 resolution input function, Figure [d] shows that the generated sample become more off and do not capture much about the function distribution.
One function: points
One function: 100
0.7
0.8-
0.5
Vilue
0.5
9.5
Function 8.5
Function 0.5
Rondom droms
0.4
94-
0.2
Domain
Domann
(a) Function data with 100 point evaluations
(b) Generated samples using input data of 100 point evaluations.
One function: 10000 points
One function: 10000 points
0.7
0.5
0.5
0.5
Fanction
0.3
0.3
0.2
02
Dombin
(c) Function data with 10000 point evaluations (d) Generated samples using input data of 10000 point evaluations.
Figure 9: The sample generation in the neural process does not collapse to the data samples. As a Bayesian approach for point ovaluations rather functions, as the number of point evaluations increases, the training process ignores the prior, resulting in an inconsistent model in the inference time.
To elaborate more on the inconsistency of the proposed heuristic neural processes model and its lack of foundations on leaning function data distribution, we construct the following additional toy example. Consider a similar setting as previous example with function distribution as mixture of two Diracs on u(z) = 0.5 and u(z)= 0.7. In this setting, the data set consists of repetitions of u(x) = 0.5 and u(z) 0.7 functions. A sensible function distribution learning method should be able to learn this mixture. Let us consider the setting where the resolution of u(x) = 0.7 is 2 (2 point evaluations). and the resolution of u(x)) = 0.5 is 100. Per our above discussion on the lack of motivation behind the objection function proposed in neural process approach and the fact that this approach aims to capture point evaluation distribution, training on such mixture of data results in model that totally ignores the data samples of u(x)) 0.7. Figure 10a shows the data set and Figure [0b] shows the learned model totally ignores the function samples u(x)) = 0.7, only because they contain fewer point evaluations. We expect that, as the resolution of u(x) 0.5 increases, neural process
18

---

Published in Transactions on Machine Learning Research (10/2022)
approach even misses learning u(z) 0.5. Figure shows the data sets where u(z) 0.5 has resolution of 10000 and Figure T0d shows training on such data set does not learn the function distribution.
To this end, we concluded that the heuristic approach proposed in neural process does not learn function distribution, rather attempts to learn point evaluation, and is not clear how this approach can be helpful learn distribution of function data
Two fun ctions: 2 vs. 100 points
Two tu nctionn:: 2 vs. 100 points
0.7
0.7
0.8
Function Function
Function D.r
Function 6.5
0.4
0.4
0.3
0.3
Domain
Doman
(a) Function data with and 100 point evalua-(b) Generated samples using input data of 2 and 100 point evaluations.
tions
functions: 10000
vs. points
0.5
Fueciton 5.5
Function 5.7
0.5
Function 8.7
Random droms
Tunchio 3.5
0.4
0.3
0.2
3
0
Domamn
Doman
(c) Function data with 2 and 10000 point evalu-(d) Generated samples using input data of 2 and 10000 point evaluations.
ations
Figure 10: Since the neural process approach aims to follow point evaluation distribution, the sample generated using the neural process ignores the low resolution data. Also, the number of point evaluations in one function u(x;) = 0.5 increases, the training process ignores the prior more, resulting in an inconsistent model in the inference time.
For the empirical study neural httpps: hwt.ghhthhhc.m.m/. Bwt.e//oommmmmieeeeeeeeeeeee DuponttnDnuonnn/nuuuee processss
processes, we
used the
implementation provided in
19


Reviews:
[{"review_id": "LDfQ8arXbE", "reviewer": "Reviewer_DpXW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n\n- Learning generative models in function spaces is interesting, and the combination of neural operators with GANs looks promising.\n- The volcano deformation data demonstrates the potential practical impact of this work.\n\nWeakness\n\nThe paper pitches GANO as a generalization of GANs, and claims its advantage over GANs. However, I think many of the claims are unfounded or based on unfair comparison.\n\n- More serious baselines are needed to assess the proposed model. \n    * The toy tasks are illustrative, but unfair for GANs as the data are generated from GRFs which were inputs into GANOs. As the GANO is proposed as a generalization of GANs, at the very least, similar toy tasks commonly used for GANs such as mixture of Gaussians, would help to level the playing field.\n    * Given the existence of other function space models (such as [2]), additional baselines should be used to ground the performance of GANO in this domain. I am not convinced by argument in “Changes Since Last Submission” that GAN is the *only* meaningful baseline, given that it is by-design ill-suited for the tasks examined in the paper.\n\n- The paper emphasizes at various places the advantage of function space modeling, and its accompanying “resolution and discretization invariance”. However, since any Fourier modes beyond the chosen max frequency are discarded, I think the model can be queried only at any *spatial* resolution, as the frequency resolution is limited. It is nice to provide this trade-off between spatial and frequency resolution, but I think the paper should be clear about this trade-off, instead of advertising it as an absolute advantage.\n\n- A few other seemingly unfair comparison:\n    * The paper claims “GANOs are more stable to train than GANs and require less hyperparameter optimization”. It is unclear to me as the effort of tuning GANO vs. the baseline GAN is not described in the paper.  On the other hand, it seems that GANO needs complex enough input GRF to match complex enough data (Fig. 4), while such input tuning is not required for GANs (which simply takes Gaussian noise in most cases).\n    * The criticism of GANs: “The dimension of the input space controls the dimension of the output manifolds” is not true. See e.g., Appendix E of [2] for the robustness in the choice of input noise.\n\nFurther several experimental details are missing:\n- I think the paper can benefit from a self-contained short introduction of the U-NO architecture on which the model based\n- The implementation of gradient penalty on grids is unclear to me.\n- How the architecture is chosen (e.g.What are the non-linearity and type of convolution)? In particular, contrary to the common practice of balanced generator and discriminator, why does the discriminator only have half the size of the generator?\n- What is the “given discretization” used for all the experiments?\n\n[2] Kidger et al. 2021. Neural SDEs as Infinite-Dimensional GANs\n[3] Brock et al. 2019. Large Scale GAN Training for High Fidelity Natural Image Synthesis", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This submission proposes the GANO, as a function space generalization GANs. It places neural operators [1] in an adversarial framework and trains parametrised function-space mappings similar to GANs. Empirically, toy examples from Gaussian Random Fields and volcano deformation data (InSAR) are used to demonstrate the capacity of the new model, and its advantages over a GAN baseline.\n\n[1] Li et al. 2018. Neural Operator: Graph Kernel Network for Partial Differential Equations", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Address the weakness mentioned above.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "di54MNdjAB", "reviewer": "Reviewer_cT3N", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n(1) It is interesting to see GAN's applied to learning distributions over functions.\\\n(2) Results in Figure 3 are a nice demonstration comparing GANs to GANO.\\\n(3) Authors show results on a complex, real world dataset.\n\nWeaknesses:\n(1) The implementation of the discriminator and gradient penalty is not clear - this could be improved by including an algorithm in the paper.\\\n(2) The experiments are very limited and the authors only compare to GANs (see more below).\\\n(3) They authors do not show any results on a down-steam task.\\\n(4) Many experimental details are missing.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors extend Neural Operators, NO, (Li et al. 2020) to the GAN framework. Instantiating both the generator and the discriminators as NOs. While GANs generators typically map from one vector to another, in the paper the authors learn to map from one function to another.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Please add a reference for this: “These properties follow the recent advancements in operator learning that generalize neural networks that only operate on a fixed resolution”\n\nIf “R is the Fourier transform if k. Then why is k an input to R in R_i(k)?\n\nTo clarify, is the gradient penalty computed on samples from the function? \n\n[CRITICAL] It would be helpful to show the final losses used for training in an algorithm similar to Alg 1. in the WGAN and WGAN-GP paper.\n\n[CRITICAL] Clarify the description of the discriminator implementation. Why is the integral function applied point-wise? Is r a logit?\n\nIs the GAN baseline trained on samples from functions? It would be helpful to clarify this.\n\nResults in Figure 2 are not convincing. There is only a small difference between (d) & (e) and (f) & (g). Does this hold over several run? I suggest that the authors repeat experiments several times to demonstrate that these effects are not the result of noise.\n\nFigure 2(c) why do the GANO samples contain high frequency noise while the GAN samples do not? Is this an artefact of the Fourier transform?\n\n[CRITICAL] GRF Data: It appears that both the input and target distributions are a set of random functions from a GRF is this is the case then the generator could just learn the identity? Especially given that they appear to be defined on the same 2D domain? Please give more details of the input and target distributions.\n\nGRF data: please clarify the domain of the target function distribution.\n\nFigure 2(a): Please clarify wether these are samples from the target (or input) distribution?\n\nTypo: Pg 2. the lost construction\n\nPlease clarify if the baseline is GAN or WGAN. Since you use the WGAN loss it may be better to label baselines as WGAN.\n\n[CRITICAL] GANO and the length scale of the input GRF: How are the input and target distributions defined? For Figure 4 demonstrate that GANO fails when the input distribution is more smooth that the target one, but performs well when the input distribution is more rough that the target distribution,  the authors should compare results with both smooth and rough input distributions, otherwise Figure 4 does not support this argument.\n\nHow does the experiments in Figure 5 differ for Figure 2? And if GANO’s input and target distributions are the same, then surely the model can just learn the identity?\n\nFigure 6 is not referenced in the text. I believe is should be referenced in the last paragraph on page 8. \n\nFigure 6: please also give details of the input and target distributions.\n\nI would suggest combining Figure 6 and 4. I would suggest using the same target distribution and using different input distributions and showing GANO results on the same plot with each type of input distributions.\n\nAs a general comment, it would be helpful to see multiple runs to see the variance over seeds.\n\nThe authors do not provide key training details, what is the learning rate and the optimiser?\n\n[CRITICAL] The authors claim that the model can be evaluated at different granularities and scales. The authors should demonstrate this on a toy task.\n\n[CRITICAL] GANs should not be the only baseline in this section. I would suggest comparing to another suitable baseline. For example:\n- Neural Processes (Garnelo et al.)\n- Generative Models as Distributions of Functions, GASP (Dupont et al.)\n\nVolcano deformation signals in InSAR data: The authors should combine GAN and GANO results on the same plot to make comparison easier. The authors should also compare to at least one other method that learns distributions over functions.\n\nGANO allows one to sample functions from a target distribution, but its not clear how this is useful in a down-steam task; what does such a model enable?\n\nRelated Work:\nThe authors should also compare their works to Neural Processes (Garnelo et al), Generative Models as Distributions of Functions  (Dupont et al.), which claim similar benefits and show results on many more datasets.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "KqKPIfHI3w", "reviewer": "Reviewer_rbkE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths \n\nThis work deals with an important problem of dealing with generating infinite-dimensional data, which can have important applications. \n\n## Weaknesses\n\nIt seems that this work does not extend that easily (as we may think with a quick read) to non-uniform grids (see my questions).\n\nThe experiments could be more convincing. More especially, I think that providing a qualitative metric of performance would strengthen the results. \n\nI feel that these two limitations should at least be transparently exposed in the paper. \n\n\n## Questions: \n- In equation (2) What is $\\mathbb{P}’_{A}$ ? I guess it is a mixture between the data distribution and the generated distribution.\n- It seems to me that there is some typos in the neural operator equation. First, it seems to me that $\\mathcal D_i = \\mathcal D$ (which is the case in [Li et al. 2020a] equation 2). Second, the integration should be with respect to $x$, i.e., \n$$ \\nu_{i+1}(y) = \\int_{\\mathcal D} k_i(x,y)\\nu_i(x) d \\mu(x) + M \\nu_i(y) + b_i(y)$$\nThus does it mean that we have to have $D_U = D_A$?\n- I have an important question regarding approximating integrals by sums. Since you have only access to the evaluation of the real data samples $u_j$ at points $x_j$, i.e., $\\{u_j(x_i)\\}$. Thus, it means that when considering samples $u$ from $P_U$, you have to consider that the measure on $D_U$ is $\\mu$ such that $x_j \\sim \\mu$ because the sum of the form $\\sum_i f(x_i) u(x_i)$ only approximate $\\int_{\\mathcal{D}} f(x) u(x) d\\mu(x)$. Thus, as mentioned, “Note that, for irregular grids where measures other than the deployed Lebesgue measures are used, this calculation should be adapted properly.” though I then think that implying that this work has application to irregular grids is a bit of an overclaim since, in my opinion, it is not clear that the framework extends and that the algorithm scales in that situation. I detail  my concern here:\n  - In the case of irregular grid, it seems to me that the metric on the function space has to be the $L_2$ with respect to $\\mu$ (the distribution of inputs) thus it mean that the target of the learning algorithm is to learn a distribution of functions that the discriminator cannot distinguish on that irregular grid. Thus, it seems very speculative to claim that “Similarly, in weather forecasts, many recording stations on the surface of Earth are located on irregular grids( fewer stations on oceans than on lands) with different fidelity and frequency of observation. [...] This allows for evaluating weather conditions at any point on the surface of Earth and computing the gradient and momentum of the fluid dynamics.” Overall I do not think that it is an argument for rejection since it does not seem that the previously addressed this concern, but I think it should be clarified in the paper's contribution that this work is on regular grid.\n  - To the extent of my knowledge (and as mentioned in [Li et al,. 2020a] after (5) The FFT can only be computed when the grid is uniform) Thus, it is not clear whether the algorithm will scale with a non-uniform grid. \n- In the experiments: \n  - It is interesting that GANO fails at producing smooth functions. What is your interpretation of it? Do you think it is because you remove some modes in the FFT? Do you have experiments about the impact of this removal on the results (e.g. how did the tuning of $k_{\\max}$ impact the results?\n  - I am not 100% convinced by the experiments on real data:\n  - Looking at the samples and the circular variance, GANO seems better\n  - However, It is not clear that GANO is better looking at the statistics, More specifically for the Circular skewness.\n  - The distribution of the circular skewness of the real data seems slightly different between Figure 7 and Figure 8, the max is a different value), is it because you used a different bin size? Thus I feel like the statement “The generated samples do not resemble the true samples, neither perceptually nor with respect to the circular variance and skewness” is an overstatement (more especially regarding the skewness). \n  - To me, it seems like it lacks a more quantitative metric such as for instance a notion of distance between the generated distribution and the data distribution (this could be computed on the discretization which would end up being a distribution on “images”) \n\n\n\n\n## Minor remarks:\n\n“Note that, while the cost functional in Eq. 2 is well defined, showing that the learned measure is indeed an approximation of P U remains an open problem.” Note that https://arxiv.org/abs/2109.00528 shows that WGAN-GP computes a congested transport problem. (most likely not in the infinite-dimensional case though) \n\nSome statements about GANs that I found inaccurate:\n- “However, models trained with a JensenShannon objective function require substantial tuning, suffer from stability issues, and are notoriously difficult to scale”. Most of SOTA GANs use the original (non-saturating) formulation. For instance, [Broke et al. 2019], [Karras et al. 2021] [Sauer et al 2022] \n- “One of the requirements to develop a stable model that maps an input probability measure to a general probability measure defined on infinite dimensional spaces is to have an infinite-dimensional input space.” For example, you could generate the parameter of a function thus pushing a measure from a finite-dimensional space to a measure on a space of functions. \n\n\n## Typos: \n“grids( fewer”\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a framework to extend the GAN framework [Goodfellow et al, 2014] to generate infinite dimensional objects such as functions using the architectures developed by [Li et al. 2020ab]. This work has application to modeling the distribution of some physical phenomena that are usually the solutions of some highly non-linear PDEs. The authors also show how to extend gradient penalty to function space with grid discretization and propose experiments on simulated and real physical data. \n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Answering the questions mentioned in the previous questions is critical to securing my recommendation for acceptance. \n\nMainly the following ones:\n- The questions around the discretization and approximating integral by sums. \n- The questions about the experiments. \n\nthank you in advance.", "broader_impact_concerns": "No concerns. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "DkQbGTGwcQ", "reviewer": "Reviewer_RQnM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The problem of generative modeling for functional space is interesting and relatively less explored.\n\n2. Strong empirical results of comparison with GAN.\n\n3. Code release for potential reproduction.\n\nWeakness:\n\n1. Not sufficient motivation and application. While it is interesting to learn generative models in functional space, I think the task can be better motivated. For example, why is it important to train generative models for functions? What are some potential practical applications? What are the practical cases where traditional models on discretized grid does not work well?\n\n2. Not sufficient background. Understanding the paper requires good familiarity with recent advances in deep learning in functional space, such as neural operators. To make the paper self contained, it would be better to briefly introduce the formulation of neural operators.\n\n3. As pointed out in multiple previous research, to convincingly show the advantage over GAN, more choices of hyper-parameter setting need to be explored.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes the GANO model, which is a generative model trained by a loss similar to WGAN on functional space. The model leverage previous neural operator model, and use it as the building block of the generator and discriminator. It is an interesting application of neural operator structure to generative modeling, and to my knowledge it is the first generative model on function spaces.\n\nThe paper studies GANO against GANs, the authors demonstrate the superior performance of GANO in modeling functions (with a discretization). \n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "It would be good to address my concern in the weakness part, namely motivation for the approach, introduction to the details of neural operators, and more comprehensive choices of hyper-parameters.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a new type of generative model for learning probabilities on infinite functional spaces. The method is novel and interesting. As stated in the introduction, there are real uses for this, and there is a convincing application of the proposed method to volcanic activities using real data. The paper has gone through a long list of improvements from reviews and the current version is significantly stronger than the first version I read. 

Reviewers remain concerned of the lack of sufficient baselines, which in my opinion is a valid criticism especially given the narrative of the paper (e.g. constant comparison to GANs). The paper could benefit from another read through to reduce the tone on this front, given the lack of sufficient baselines of comparison. Nonetheless, as this is a new method, there is no truly proper baseline for comparison making it difficult to make said comparison. Reviews also remain concerned that it unclear how GANO can be used for a down-stream task or application. 

Despite these outstanding criticisms, the paper is still worth publishing due to its novelty, convincing empirical study on real-world data, and fit to the journal audience. 

Minor note that some mark-up needs to be removed for camera-ready copy (i.e. assuming the red color emphasis in the algorithm pseudo-code is not intended for the final version).
Decision: Accept (poster)