Distribution Matching in Deep Generative Models with Kernel Transfer Operators
Authors: Zhichun Huang
Rudrasis Chakraborty
Vikas Singh
Venue: ICLR 2022 Submitted (2022)

Abstract: Generative models which use explicit density modeling (e.g., variational autoencoders, flow-based generative models) involve finding a mapping from a known distribution, e.g. Gaussian, to the unknown input distribution. This often requires searching over a class of non-linear functions (e.g., representable by a deep neural network). While effective in practice, the associated runtime/memory costs can increase rapidly, usually as a function of the performance desired in an application. We propose a substantially cheaper (and simpler) distribution matching strategy based on adapting known results on kernel transfer operators. We show that our formulation enables highly efficient distribution approximation and sampling, and offers surprisingly good empirical performance that compares favorably with powerful baselines, but with significant runtime savings. We show that the algorithm also performs well in small sample size settings (in brain imaging). 

Under review as a conference paper at ICLR 2022
FORWARD OPERATOR ESTIMATION IN GENERATIVE MODELS WITH KERNEL TRANSFER OPERATORS
Anonymous authors Paper under double-blind review
A ABSTRACT
Generative models which use explicit density modeling (e.g., variational autoencoders, flow-based generative models) involve finding a mapping from a known distribution, e-g. Gaussian, to the unknown input distribu- tion. This often requires searching over a class of non-linear functions (e.g., representable by deep neural network). While effective in practice, the associated runtime/ memory costs can increase rapidly, usually as function of the performance desired in an application. We propose much cheaper (and simpler) strategy to estimate this mapping based on adapting known results in kernel transfer operators. We show that our formulation enables highly efficient distribution approximation and sampling, and offers surpris- ingly good empirical performance that compares favorably with powerful basclines, but with significant runtime savings. We show that the algorithm also performs well in small sample size settings (in brain imaging).
INTRODUCTION
Generative modeling, in its unconditional form, refers to the problem of estimating the data generating distribution: given i.idd samples X with an unknown distribution Px, generative model seeks to find a parametric distribution that closely resembles Px. In modern deep generative models, we often approach this problem via a latent variable i.e., we assume that there is some variable Z associated with the observed data E X that follows a known distribution Pz (also referred to as the prior in generative models). Thus, we can learn a mapping f > X' such that the distribution after transformation, denoted by Pς(z), aligns well with the data generating distribution Px. Therefore, sampling from Px becomes convenient since Pz can be efficiently sampled. Frequently, f parameterized by deep neural networks and optimized with stochastic gradient descent (SGD).
Existing generative modeling methods variously optimize the transformation f, most com- monly modeling it as Maximun Likelihood Estimation (MLE) or distribution matching problem. For instance, given data X = {xI....,X., X,), a variational autoencoder (VAE) (Kingma & Welling, 2013) first constructs Z through the approximate posterior Q2) and maximizes a. lower bound of likelihood (X). Generative adversarial networks (GANs) Goodffllow et al. 2014) relics on simultancously learned discriminator such that samples of Pς(z) are indistinguishable from X. Results in (Arjovsky al. 2017 Li al.) 2017) suggest that GANs minimize the distributional discrepancies between P((2) and Px. Flow-based generative models optimize (X) explicitly through the change of variable rule and efficiently calculating the Jacobian determinant of the inverse mapping f-!
In all examples above, the architecture or objective notwithstanding, the common goal is to find a suitable function f that reduces the difference between Pςız) and Px. Thus, key component in many deep generative models is learn a forward operator as defined below. Definition 1.1 (Forward operator). A forward operator f" E C: Z + X is defined to be mapping associated with some latent variable ~ Pz such that f = arg minsec ((P)(2))) Px) for some function class C and distance measure d(...).
Motivation: The specifics of the forward operator may differ from case to case. But its properties and how it is estimated numerically greatly influences the empirical performance

---

Under review as a conference paper at ICLR 2022
of the model. For instance, mode collapse issues in GANs are well known and solutions contimue to emerge (Srivastava et al. 2017). To learn the forward operator. VAEs use an approximate posterior x that may sometimes fail to align with the prior (Kingma et al. 2016 Dai & Wipf 2019). Flow-based generative models enable direct access to the posterior likelioood yet in order to tractably evaluate the Jacobian of the transformation during training, one must either restrict the expressiveness at cach layer (Dinh et al. 2017 Kingma & Dhariwal 2018) or use more involved solutions (Chen et al. 2018). Of course, solutions to mitigate these weaknesses (Ho al.) 2019) remains an aclive area of research.
The starting point of our work is to evaluate the extent to which we can radically simplify the forward operator in deep generative models. Consider some desirable properties of hypothetical forward operator (in Def. (I.1b) D): (a) Upon convergence, the learned operator f minimizes the distance between Px and Pr(z) over all possible operators of certain class. (b) The training directly learns the mapping from the prior distribution Pz, rather than a variational approximation. (c) The forward operator /* can be efficiently learned and sample generation is also efficient. It would appear that these criteria violate the "no free lunch rule", and some compromise must be involved. Our goal is to investigate this trade-off: which design choices can make this approach work? Specifically, a well studied construct in dynamical systems, namely the Perron-Frohbe Frobenius operator (Lemmens & Nussbaum 2012). suggests an alternative linear route to model the forward operator. Here, we show that if we are willing to give up on few features in existing models this may be acceptable depending on the downstream use case then, the forward operator in generative models can be efficiently approximated reproducing kernel Hilbert space (RKHS). With simple adjustments of existing results, we identify novel way to replace the expensive training for generative tasks with simple principled kernel approach.
the estimation of a closed-form linear operalor in the
Contributions. Our results are largely based results in kernel methods and dynamical systems, but we demonstrate their relevance in generative modeling and complement recent ideas that emphasize links between deep generative models and dynamical systems. Our contributions are (a) We propose a non-porametric method for transferring a known prior density linearly in RKHS to an unknown dała density equivalent to learning nonlinear forward operator in the input space, When compared to its functionally analogous module used in other deep generative methods, our method avoids multiple expensive training steps yielding significant efficiency gains; (b) We evaluate this idea in multiple scenarios and show competitive generation performance and efficiency benefits with pre-trained autoencoders on popular image datasets including MNIST, CIFAR-10, CelebA and FFHQ; (c) As a special use case, we demonstrate the advantages over other methods in limited data settings.
aaaationlly--nalooous
PRELIMINARIES
We briefly introduce reproducing kernel Hilbert space (RKHS) and kernel embedding of probabil- ity distributions, concepts we will use frequently.
Notations
Meaning
Z
X
Random variable
Z
Data samples
Z
X
Domain
Definition 2.1 (RKHS (Aronszajn 1950 ). For set X, let H be a set of functions g: x R. Then, H is a reproducing kernel Hilbert space (RKHS) with product (r if there erists a function : X x X + R (called a reproducing kernel) such that (i) Vx X.g E H.)(z) (g, k(x, )H;; (ūi H H=ca cl cs(ppnn(( (k(x, ),, E X))),
Pz
Px
Distribution
pz
Px
Density function
K
Kernel function
H.
G
RKHS
((-)
wD(-)
Feature mapping
EK
&1
Mean embedding operator
JX
Kernel mean embedding
where el(.) is the set closure.
Table 1: Notations used in this paper.
:x X is referred to as the feature mapping of the induced RKHS H. A useful identity derived from feature mappings is the kernel mean embedding: it defines a mapping from a probablity measure in X' to an element. in the RKHS.
The function σ(x) = k(xx :::
Definition 2.2 (Kernel Mean Embedding (Smola et al., 2007)). Given probability mea- sure p on X' with an associated RKHS H. equapped with a reproducing kernel k such that supzex a)xx,)) < 00, the kernel mean embedding of p in RKHS H. denoted by Pp E H. is
2

---

Under review as a conference paper at ICLR 2022
defined as Mp Epl[[z]] (p(()dd and the mean embedding operator & : (Z) - H defined as H4 = Ep.
Remark 1. For charucleristic kernels, the operalor & is injective. Thas, two distri- butions (p. in X' are identical iff E
This property allows using of Marimum Mean Discrepancy (MMD) for distribution match- ing (Gretton et al,, 2012; Li et al., 2017) and is common, see (Muandet et al., 2017 Zhou et al. 2018). For finite number of samples {x,) drawn from the probabil- ity measure p, unbiased empirical estimate of pnu is PH 1. -.=,,((x such that lim,-roo >> 1(x;,,) paz-
Next, we review the covariance/ cross covariance operators, two widely-used identities in kernel methods (Fukumizu et al. 2013 Song al. 2013) and building blocks of our approach.
Definition 2.3 (Covariance/ / Covasacov Operator). Let X, Z be random variables defined on X x Z with joint distribution Px.z and marginal distributions Px, Pz. Let (l, σb H) and (ki, 12, G) be two sets of (a) bounded kernel, (b) their corresponding feature map, and (c) their induced RKHS, respectively- The (uncentered) covariance operator Czz H - H and cross-covor covoriccee operator Cxz H + are defined as
Czz - BemP2(0(2) @ d(2)]
Cxz &
0(()l
(1)
where @ is the outer product operator.
3 SIMPLIFYING THE ESTIMATION OF THE FORWARD OPERATOR
Forward operator as a dynamical system: The dynamical system view of generative models has been described by others (Chen et al. 2018 Grathwohl et al. 2019 Behrmann
et al. 2019 These strategies model the evolution of latent variables in a residual neural network in terms of its dynamics over continuous or discrete time i, and consider the output function f as the evaluation function at a predetermined boundary condition t = t Specifically, given an input (i.e., initial condition) =(to), f is defined as
+ 1/' Ac(=(1))dt
= ((z(t)))=(()
where A, is timo-dependent neural network function and 2(t) is the intermediate solution at 1. This view of generative models is not limited to specific methods or model archetypes, but generally useful, for example, by viewing the outputs of each hidden layer evaluations in discrete-time dynamics. After applying of on random variable Z E Z, the marginal density of the output over any subspace A X can be expressed as
f P)12)
JEE Pz(2)dz
If there exists some neural network instance A; such that the corresponding output func- tion f satisfies Px Pf.(2), , by Def. I.1 fr is forward operator. Let X be set of ii.d. samples drawn from Px. In typical generative learning, either maximizing the likeli- hood [x| Exex (x) or minimizing the distributional divergence d((r(z), Px) requires evaluating and differentiating through ʃ or f-1 many times.
Towards a one-step estimation of forward operator: Since f and f in 3 will be highly nonlinear in practice, evaluating and computing the gradients be expensive. Nevertheless, the dynamical systems literature suggests a linear extension of f* namely the Perron-Frobenie operator transfer operator, that conveniently transfers pz to px-
Definition 3.1 (Perron-Probeniuss operator [Mayer 19801). Given dynamical system f X + X, the Perron-Frobenius (PF) operator P L (X) + (x) is an infinite- dimensional linear operator defined Sn (pPz))(z))r Jaer *(A) Pz(=)dz for all A C X.
3

---

Under review as a conference paper at ICLR 2022
Pre Pre-tnined Autoenccdar (For High-dimeniioall lpputs nrly)
Pe
As
Inpur
f
8g(+)
60
Encoddr
sample
1 |ontorms to
Decoder
P
Samples
J2 E (g(2)
Figure 1: Summary of our framework. The proposed operator is estimated to transfer RKHS embedded densities from one anothter (blue arrows). Generating new samples (red arrows) involves embedding the prior samples z" Pz; applying the operator V =P<((=)), and finding preimages (VV). The pre-trained autoencoder projects the data onto smooth latent space and is only required when generating high- nigh-dimensional data such as images.
Although in Def. 3.1 the PF operator P is defined for self-maps it is trivial to extend P to mappings f Z X by restricting the RHS integral JaEs-1A)) Pz (z)dz to Z.
It can be seen that, for the forward operator f", the corresponding PF operator P satisfies px Ppz
(4)
If P can be efficiently computed, transferring the tractable density pz to the target density Px can be accomplished simply by applying P. However, since P is an infinito-dimensional
operator on L' (x)), is impractical to instantiate it explicitly and exactly. Nonetheless, there exist several methods for estimating the Perron-Frobente
operator, including Ulam's method U1am 1960) and the Extended Dynamical Mode Decomposition (EDMD) (Williams) let al. 2015a). Both strategies project P onto finite number of hand-crafted basis functions this may suffice in many settings but may fall short in modeling highly complex dynamics.
Kernel-embedded form of PF operator: A natural extension of PF operator is to rep- resent P by an infinite set functions (Klus et al. 2020). e.g., projecting it onto the bases of an RKHS via the kernel trick. There, for characteristic kernel 1, the kernel mean embedding uniquely identifies an element μx = Eppx E g for any Px E L (X). Thus, to approximate P, may alternatively solve for the dynamics from pz to px in their embedded form. Using Tab. I notations, we have the following linear operator that defines the dynamics between two embedded densities.
Definition 3.2 "Kernel-embeddee Perron-Frood Frobenius operator (Klus et al. 2020 Given pz E L' (X) and px L' L')). Denote k as the input kernel and 1 the output kernel. Let Px = Epx and fzz = ExpZ be their corresponding mean kernel embeddings. The kernel-embedded Perron- Froonnnns (k:PF) operator, denoted by Pe H 9, defined as
Pe CxzCzz
(5)
Proposition 3.1 (Sog et al. 2013 ). With the above definition, Pe satisfies
px = Peuz
(6)
under the conditions: (i) Czz is injective (ii) μe range(Czz) (ii) B H. for any g E G.
The last two assumptions sometimes be difficult to satisfy for certain RKHS (see Theorem 2 of Fukumizu et al. (2013)) ). In such cases, a relaxed solution can constructed by replacing Cz by regularized inverse (Czz + AL) or Moore-Penroee pseudoinverse Ct
The following proposition shows commutativity between the (kernel-embedded)) PF operator and the mean embedding operator, showing its equivalence to P when / is characteristic.
Proposition 3.2 (Klus et al. 20201). With the above notations, El P Pe Ec-
4

---

Under review as a conference paper at ICLR 2022
Transferring embedded densities with the kPF operator: The kPF operator is powerful tool that allows transferring embedded densities in RKHS. The main steps are:
Use mean embedding operalor & on pz- Let denote by uzz. (2) Transfer μzz using kPF operator Pg to get the mean embedded px, given by μax
Of course, in practice with finite data, (xiheenl ~ Px and {ziheem ~ Px, Pe must be estimated empirically (see Klus et al. (2020) for an error analysis).
Pe = Cx2(C22)) = V( + AnI) 1 ()"
where & [k(2),)) :), k(za,') -)), [((x1,). ... ((xx,,)) -] are simply the corresponding feature matrices for samples of Px and Pz, and x is a small penalty term.
Learning kPF for unconditional generative modeling: Some generative modeling
methods such as VA and flow-based formulations explicitly model the latent variable Z as conditionally dependent on the data variable X. This allows deriving/ deriving/optimien /optimiziiiig the likelihood (X)). This is desirable but may not be essential in all applications. To learn kPF, however, and Z can be independent RVs. While it may not be immediately obvious why could assume this independence, empirical kPF operator, assuming that the empirical covariance operator Czz is non-singular:
can observe the following property for the
PEAz CxzCzziiz -D (4
II ( - Vl,, = Åx
(7)
Exz Czz
Suppose that (xiheen)) and {2ileen)) are independently sampled from the marginals Px and Pz. It. is easy to verify that (7) holds for any pairing {(x, (.))xl,)e However, instantiating the RVs in this way rules out the use of kPF for certain downst tasks such as controlled generation or mode detection, since Z does not contain information regarding X. Nevertheless, if sampling is our only goal, then this instantiation of kPF will suffice.
Mapping Z to G: Now, since Pg is a deterministic linear operator, we can easily set up scheme to map samples of Z elements of G where the expectation of the mapped samples equals μxx
Define dbz) = .(2,,) and (bx) 1(x, :) as feature maps of kernels and We can rewrite Hx as
Px = Pe = P3E2 2) = E2(Pe(((()) = Ezlω (v (P&o (2)))))))
(8)
Here 6-1 is the inverse or the preimage map of v. Such an inverse, in general, may not exist (Kwok Tsang 2004 Honeine & Richard (011). We will discuss a procodure to approximate 2 in $4.1 In what tollows, we will temporarily assume that an exact preimage map exists and is tractable to compute.
Define V = (Z) as the transferred sample in 9 using the empirical embeddod PF operator Pg. Then the next result shows that asymptotically the transferred samples converge in distribution to the target distribution.
Proposition 3.3. As n 0o, きレ (V²) 3 Px. That is, the preimage of the transferred sample upprozimately conforms to Px under previous assumptions when n is large.
Proof. Since Pg asymp. P, the proof immodiately follows from
SAMPLE GENERATION USING THE KERNEL TRANSFER OPERATOR
At this point, the transferred sample V obtained by the kPF operator, remains an element of RKHS G. To translate the samples back to the input space, we must find the preimage x² such that (xxx)) V².
5

---

Under review as a conference paper at ICLR 2022
4.1 SOLVING FOR AN APPROXIMATE PREIMAGE
Solving the preimage in kernel-based methods is known to be ill-posed (Mika el al. 1999) because the mapping w(.) not necessarily surjective, i.e., a unique preimage x' x """"), V H may not exist. Often, approximate preimage (X,, V* e 1 (v²)
is constructed instead based on relational properties among the training data in the RKHS.
We consider two options in our framework (1) MDS-based method (Kwok & Tsang 2004
Honcine & Richard 2011)
(X, V²' = 2 (X'X' " X" x') - d,), where Vi E [n], d; [1(x4,) - v'
(9)
which optimally preserves the distances in RKHS to the preimages in the input space, and (2) weighted Fréchet mean (Friedman et al. 2001 which in Euclidean space takes the form
WFFM(X, (X, V- = WWFM (X: = X's s/ |||||| where Vi E h1 (1x4), -), w²)
(10)
where X' a neighborhood of training sam- ples based on pairwise distance or similarity in RKHS, following (Kwok & Tsang 2004).
Algorithm 1 Sample Generation from kPF
1: Input: Training data X = (xi.... ............ Op- lional autoencoder (E, D), input/oulput ker- nels (k, neighborhood size 7
The weighted Fréchet mean preimage uses the inner product weights (VL", (xx)) as measures of similarities to interpolate train-
2: Training
ing samples. On the toy data (as in Fig, 2 weighted Fréchet mcan producos fewer sam- plcs that deviate from the true distribution
3: X (E (x1)....; ....; E.....,.,,... (x.)) if is provided
4: Sample {z, he(m)) ~ Pz independently
5: Construct L, K E R**= s.t.
Lig = ((x,(,())) Kij = l: (2),2;)
and is easier to compute. Based on this ob- servation, we use the weighted Fréchet mean
Kinv (K + Xmn)
7:
Inference
as the preimage module for all experiments
8: Generate new prior sample z" Pz
that requires samples, while acknowledging that other preimage methods can also be substituted in.
9:
s L. Kine [k(z), "=)...")(e z²)
10:
ind = arg)ort(s){
11:
x" = wht WTrM (X X(ind| s(ind)) ).
12: Output D(x") if D is provided else x"
With all the ingredients in hand, we now present an algorithm for sample gencration
using the kPF operator in Alg. [ The idea simple yet powerful: at training time, we construct the empirical kPF operator Pe using the training data {xi) and samples of the known prior [z, hejn)- At test time, we will transfer new points sampled from Pz to feature maps in H. and construct their preimages as the generated output samples.
4.2 IMAGE GENERATION
Image generation is a common application for generative models (Goodfellow et al. 2014 Dinh et al.) 2017). While our proposal is not image specific, constructing sample preimages in a high dimensional space with limited training samples can be challenging, since the space of images is usually not dense in a reasonably sized neighborhood. However, empirically images often lie near a low dimensional manifold in the ambient space (Seung & Leel 2000), and one may utilize an autoencoder (AE) (E, D) to embed the images onto latent space that represents coordinates on a learned manifold. If the learned manifold lies close to the true manifold, we can learn densities on the manifold directly (Dai & Wipf 2019).
Therefore, for image generation tasks, the training data is first projected onto the latent space of pretrained AE. Then, the operator will be constructed using the projected latent representations, and samples will be mapped back to image space with the decoder of AE, Our setup can be viewed analogously to other generative methods based on so called "ex-post" density estimation of latent variables (Ghosh et al. 2020). We also restrict the AE latent space to a hypersphere S to ensure that (a) k(. ) and I(:,-) are bounded and (b) the space is geodesically convex and complete, which is required by the preimage computation.
To compute the weighted Fréchet mean on a hypersphere, we adopt the recursive algorithm in Chakraborty & Vemuri (2015) (see appendix D for details).
6

---

Under review as a conference paper at ICLR 2022
Data
Data
GMM
MDS
Glow
wFM
kPF
Figure 2:
Left figure: Density estimation on 2D toy data. Top to bottom: (1) Training
data samples, and learned densities of (2) GMM (3) Glow (4) Proposed kPF operator. More details in appendix. Right figure: Sample generation results. Top: Data samples Middle:
MDS-baased preimage samples Bottom: Weighted Fréchet mean samples.
5 EXPERIMENTAL RESULTS
Goals. In our experiments, seek to answer three questions: (a) With sufficient data, can our method generate new data with comparable performance with other state-of-the-ars generative models? (b) If only limited data samples
given, can method still estimate the density with reasonable accuracy? (c) What are the runtime benefits, if any?
Datasets/setupp To answer the first question, we evaluate method standard vision datasets, including MNIST, CIFAR10, and CelebA, where the number of data samples is much larger than the latent dimension. We compare results with other VAE variants
Two-stage VAE Dai & Wipf (2019), WAE Arjovsky et al. (2017), CV-VAE Ghosh et al. (2020 and flow-based generative models (Glow Kingma & Dhariwal (2018), CAGlow Liu et al. (209 The second question is due to the broad use of kernel methods in small sample size settings. For this more challenging case, we randomly choose 100 training samples (< 1% of the full dataset) from CelebA and evaluate the quality of generation compared to other density approximation schemes. We also use dataset of T1 Magnetic Resonance (MR) images from the Alzheimer's 's Disease Neuromaging Initiative (ADNI) study.
Distribution transfer with many data samples. We evaluate the quality by calculating the Fréchet Inception Distance (FID) (Heusel et al. 2017) with 10K generated images from each model. Here, we use pretained regularized autoencoder (Ghosh el al. 2020) with latent space restricted to the hypersphere (denoted by SRAE) to obtain smooth latent representations. We compare our kPF to competitive end-to-end deep generative baselines (L.e. flow and VAE variants) well other density estimation models over the SRAE latent space. For the latent space models, we experimented with Glow (Kingma &
S
6
5 8
4
4
/
8
6
7
8
8
8 1
3
7
6
9
7 8
2
8
3
4
0
7
9
4
3
3
3
7 4
O /
2
0
6
4
Two-stage VAE
SRAE
SRAEGMM
SRAENTK<PPF
Figure 3: Comparison of different sampling techniques using AE trained on CelebA 64x64 Left to right: samples of (1) Two-Stage VAE (Dai & Wipf 2019) (2) SRAEGlow (Kingma &
Dhariwal 2018 (3) SRAEGMM (4) SRAE NTK-&PF using 10k latent points.
7

---

Under review as a conference paper at ICLR 2022
Figure 4: Representative samples from learned kPF on pre-trained NVAE latent space
Dhariwal 2018), VAE, Gaussian mixture model (GMM), and two proposed kPF operators with Gaussian kernel (RBF-kPF) and NTK (NTK-kPF) as the input kernel. The use of NTK molivated by promising results at the interface of kernel methods and neural nelworks Jacot et al. 2018 Arora et al. 2020 Implementation details are included in the appendix.
Comparative results are shown Table 2 We see that for images with structured feature spaces, e.g., MNIST and CelebA, our method matches other non-adversarial generative models, which provides evidence in support the premise that the forward operator can be simplified. Further, present qualitative results on all datasets (in Fig. 3 where compare our kPF operator based model with other density estimation techniques on the latent space. Observe that model generates comparable visual results SRAEGlow.
Since kPF learns the distribution on a pre-trained AE latent space for image generation, using more powerful AE offer improvements in generation quality. In Fig. present representative images by learning our kPF on NVAE Vahdat. & Kautz (2020) latent space, pre-trained on the FFHQ dataset. NVAE builds hierarchical prior and achieves state-of-the-art generation quality among VAEs. We see that kPF can indeed generate high-quality and diverse samples with the help of NVAE encoder encoder/decodes /ecooder. In fact, any AE/VAE may be substituted in, assuming that the latent space is smooth.
When sufficient number of samples are available, our algorithm performs as well as the alternatives, which is attractive given the efficient training. In Fig. 6 we present comparative result FIDs with respect to the training time. Since kPF can computed in closed-form, it achieves significant training efficiency gain compared to other deep generative methods while delivering competitive generative quality.
Summary:
Distribution transfer with limited data samples. Next, present evaluations when only a limited number of samples are available. Here, each of the density estimators was trained on latent representations of the same set of 100 randomly sampled CelebA images, and 10K images were generated to evaluate FID (see Table 30. Our method outperforms Glow and VAE, while offering competitive performance with GMM. Surprisingly. GMM remains a strong baseline for both lasks, which agrees with results in Ghosh el. al. (2020). However, note that GMM restricted by its parametric form and is less flexible than our method (as shown in Fig 2).
MNIST
CIFAR
CelebA
Glow
25.8
103.7
VAE
!AGlow
26.3
104.9
Glew
Vanilla VAE
36.5
111.0
52.1
RBF-UPF
CV-VAE!
33.8
91.8
48.9
NTK-KPF
WAE!
20.4
117.4
53.7
Two-stage VAE
16.5
110.3
44.7
50
SRAEcIow
15.5
85.9
35.0
SRAEvAE
17.2
198.0
48.9
45
SRAEGMM
16.7
79.2
42.0
40
SRAERBE-RR (owr))
19.7
77.9
41.9
35
SRAENTR (ours)
19.5
77.5
41.0
100
200
300
400
500
600
700
wall clock time (sec)
Table 2: Comparative FID values. SRAE indicates an autoencoder with hyperspherical latent space and spectral regularization following Ghosh et al. (2020). Results reported from f Liu et al. (2019).
Figure 5: FID versus training time for latent space models. All models are learned on the latent representa- tions encoded by the same pro-trained SRAE
Ghosh al. (2020).
8

---

Under review as a conference paper at ICLR 2022
Learning from few samples is common in biomedical applications where acquisition
VAE
Glow
GMM
RBF--PF
NTK-KPF
59.3
77.0
39.6
40.6
40.9
is costly. Motivated by interest in mak-
ing synthetic but statistic preserving data
Table 3: FID values for few samples setting
(rather than the real patient records) pub-
density approximation on CclebA
licly available to researchers (see NIH N3C Data Overview), we present results on gen- erating high-resolution (160 196 160) brain images: 183 samples from group AD (di- agnosed as Alzhcimer's discase) and 291 samples from group CN (control normals). For n 474 < d 5017600, using our kernel operator, we can generate high-quality samples that are in-distributtin. We present comparalive results with VAEs. The generaled samples in Fig. G clearly show that our method generates sharper images. To check if the results are also scientifically meaningful, we test consistency between group difference testing (i.e., cases versus controls differential tests on each voxel) on the real images (groups were AD and CN) and the same test was performed on the generated samples (AD) and CN groups), using a FWER corrected two-sample t-test Ashburner & Friston (2000). The results (sce Fig 6 show that while there is deterioration in regions identified to be affected by discasc (different across groups), many statistically salisticaall-siiificaaattt regions from tests on the real images are preserved in voxel-wise lests on the generated images.
Summary: We achieve improvements in the small sample size setting compared to other generative methods. This is useful in many data-poor settings. For larger datasets, our method still compares competitively with alternatives, but with a smaller resource footprint.
6 LIMITATIONS
Our proposed simplifications can be variously useful, but deriving the density of the posterior given mean embedding or providing an exact preimage for the generated sample in RKHS is unresolved at this time. While density estimation from kPF has been partially addressed in Schuster et al. (2020b), finding the pre-image is often ill-posed. The weighted Fréchet mean preimage only provides approximate solution and evaluated empirically, and due to the interpolation-oet based sampling strategy, samples cannot be obtained beyond the convex hull of training examples. Making Z and X independent RVs also limits its use for certain reams lask such as representation learning or semantic clustering. Finally, like other kernel-based methods, the sup-quadratie memoryy/omm /compule cost. can be bottleneck on large datasets and kernel approximation (e.g. (Rahimi & Recht 2009) may have to be applied; we discuss this in appendix F
CONCLUSIONS
We show that using recent developments in regularized autoencoders, a linear kernel transfer operator can potentially be an efficient substitute for the forward operator in some generative models, if some compromise in capabilttissppeffrmnnee performance is acceptable. Our proposal, despite its simplicity, shows comparable empirical results to other generative models, while offering efficiency benefits. Results on brain images also show promise for applications to high- resolution 3D imaging data generation, which is being pursued actively in the community.
Figure 6: Left. Top: data, generated samples of Middle: VAE Bottom: kPF with SRAE. Right. Statistically significant regions Top: vorel-wise tests on real data, Bottom: vozel-wise tests on generated samples are shown in negative log value thresholded at = 0.01.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein gan. arXiv preprint (rXic:1711007877 2017.
Nachman Aronszajn.
Theory of reproducing kernels. Transactions of the American mathe- matical sociely, 68(3):337 404, 1950.
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R. Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' dlcch-BBue, E. Fox, and R. Garnett (ods.), Advances in Neural Information Processing Systems 32, pp. 8141-8150. Curran Associates, Inc., 2019.
Sanjeev Arora, Simon S. Du, Zhiyuan Li, Ruslan Salakhutdinov, Ruosong Wang, and Dingli Yu. Harnessing the power of infinitely wide deep nets on small-data tasks. In International Conference on Learning Hepresentations, 2020.
John Ashburner and Karl J Friston. Voxel-based morphometry the methods. Neuroimage, (6)))880 821, 2000.
Jens Behrmann, Will Grathwohl, Ricky TQ Chen, David Duvenaud, and Jörn-Henrik Jacobsen. Invertible residual networks. In International Conference on Machine Learning, pp. 573-582. PMLR, 2019.
Steven L. Brunton, Joshua L. Proctor, and J. Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the National Academy of Sciences, (13(11):33323300)) 2016. ISSN 0027-8424. doi: 10.1073/
Rudrasis Chakraborty and Baba C Vemuri. Recursive frechet mean computation on the grassmannian and its applications to computer vision. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4229 4229-4233 4237, 2015.
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. BiancBiachhi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 6571-6583 6583. Curran Associates, Inc., 2018.
Bin Dai and David Wipf. Diagnosing and enhancing VAE models. In International Conference on Learning Representations, 2019.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. In International Conference on Learning Representations, 2017.
Paul F. Evangelista, Mark J. Embrechts, and Boleslaw K. Szymanski. Taming the curse of dimensionality in kernels and novelty detection. In Ajith Abraham, Bernard de Baets, Mario Köppen, and Bertram Nickolay (eds.), Ayplied Soft Compuling Technologies: The Challenge of Complezity, pp. 425-435 Berlin, Heidelberg, 2006. Springer Berlin Heidelberg. ISBN 278-3-540-31662--
Jerome Friedman, Trevor Hastic, Robert Tibshirani, et al. The elements of statistical learning. volume 1. Springer series in statisties New York, 2001.
Kenji Fukumizu, Le Song, and Arthur Gretton. Kernel bayes' rule: Bayesian inference with positive definite kernels. The Journal of Machine Learning Research, 14(1):3753 3783, 2013.
Partha Ghosh, Mehdi S. M. Sajjadi, Antonio Vergari, Michael Black, and Bernhard Scholkopf. From variational to deterministic autoencoders. In International Conference Learning Representations, 2020.
Ian Goodfellow, Jean Pouget-Abadee Mchdi Mirza, Bing Xu, David Warde-Farloy Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 27, pp. 2672 2680. Curran Associates, Inc., 2014.
10

---

Under review as a conference paper at ICLR 2022
Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, and David Duvenand. Scalable re- versible generative models with free-form continuous dynamics. In International Conference Learning Representations, 2019.
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research, 13(1): 723-77 773, 2012.
Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning. Springer Series in Statistics. Springer New York Inc., New York, NY, USA, 2001.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp- 770-778, 2016. doi: D0.119/CCVPR.21166
Martin Housel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochre- iter. Gans trained by a two time-scale update rulc converge to local nash cquilibrium. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and
In I. Guyon, U. V. Luxburg, R. Garuell (eds.), Advances in Neural Information Processing Systems 30, pp. 6626 6637. Curran Associates, Inc., 2017.
Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Pieter AbbeeL. Flow++: Improving flow-based generative models with variational dequantization and architecture design. In International Conference on Machine Learning, pp. 2722 2730. PMLRR, 2019.
Paul Honeine and Cedric Richard. Preimage problem in kernel-based machine learning. TEEE Signal Processing Magazine, 28(2):77-88, 2011.
Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 8571-8580. Curran Associates, Inc., 2018.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint ar arXiv:1312 meee:3112.1111 6114, 2013.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible Ixl con- volutions. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 10215-10224. Curran Associates, Inc., 2018.
Durk P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Πlya Sutskever, and Max Welling, Improved variational inference with inverse autoregressive flow. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 4743 4751. Curran Associates, Inc., 2016.
Stefan Klus, Ingmar Schuster, and Krikamol Muandet. Figendecompositionn of transfer operators in reproducing kernel hilbert spaces. Journal of Nonlinear Science, 30(1):283 315, 2020.
JT-Y Kwok and IW-H Tsang. The pre-image problem in kernel methods. IEEE transactions on neural networks, 15(6):1517 1525, 2004.
Bas Lemmens and Roger Nussbaum. Nonlinear Perron-Frobente Theory, volume 189. Cambridge University Press, 2012.
Chun-Liang Li, Wei-Cheng Chang, Yu Cheng, Yiming Yang, and Barnabas Poczos. Mmd gan: Towards deeper understanding of moment matching network. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30, pp. 2203 2213. Curran Associates, Tnc., 2017.
11

---

Under review as a conference paper at ICLR 2022
Rui Liu, Yu Liu, Xinyu Gong, Xiaogang Wang, and Hongsheng Li. Conditional adversar- ial generative flow for controllable image synthesis. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Dieter H Mayer. The ruclle-araki transfer operator in classical statistical mechanics. 1980.
Sebastian Mika, Bernhard Schölkopf, Alex Smola, Klaus-Robert Müller, Matthias Scholz, and Gunnar Rätsch. Kernel pea and de-noising in feature spaces. In M. Kearns, S. Solla, and D. Cohn (eds.), Advances in Neural Information Processing Systems, volume 11, pp. 536.542. MIT Press, 1999.
K. Muandet, K. Fukumizu, B. Sriperumbudur, and B. Schölkopf. 2017. doi: 10.1561/ 2200000060.
Ali Rahimi and Benjamin Recht. Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning. In D. Koller, D. Schuurmans, Y. Bengio, and L.. Bottou (eds.). Advances in Neural Information Processing Systems, volume 21. Curran Associates, Inc., 2009. URL https::/procceednnss www.eennurrirroeeeeeeoee ccaerpp/pcc/ppppe2200000/ fflee//cee8830200005000000C000ee
pdf
M. Kafaei Razavi, A. Kerayechian, M. Gachpazan, and S. Shateyi. A new iterative method for finding approximate inverses of complex matrices. Abstract and Applied Analysis, 2014 4: 563787, Sep 2014. ISSN 1085-3375. doi: 00.1155/201//5620 URL https::/doi. org/ 10. 20.1155/204/556778
Mehdi S. M. Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly, Assessing generative models via precision and recall. In S. Bengio, H. Wallach, H. Larochelle, and R. Garnett (eds.). Advonces in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL httt:tttss////preeediii pdf
K. Grauman, N.
Ingmar Schuster, Mattes Mollcnhauer, Stefan Klus, and Krikamol Muandet. Kernel condi- tional density operators. In Silvia Chiappa and Roberto Calaudra (eds.), Proceedings of the Twenly Third International Conference Arlificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pp. 993-1004. PMLR, 26-28 Aug 2020a.
Ingmar Schuster, Mattes Mollenhauer, Stefan Klus, and Krikamol Muandet. Kernel condi- tional density operators. volume 108 of Proceedings of Machine Learning Research, pp. 993-1004, Online, 26-28 Aug 2020b. PMLR.
H. Sebastian Seung and Daniel D. Lee. The manifold ways of perception. Science, 250 2268 2269, 2000. ISSN 0036-8075. doi: 10.1126/sciende 29t.:mc.....05500.2220
Alex Smola, Arthur Gretton, Le Song, and Bernhard Schölkopf. A hilbert space embedding for distributions. In Marcus Hutter, Rocco A. Servedio, and Eiji Takimoto (eds.), Algorithmic Learning Theory, pp. 13 31, Berlin, Heidelberg, 2007. Springer Berlin Heidelberg. ISBN 978-3-5-00-75225-7
Le Song, Kenji Fukumizu, and Arthur Gretton. Kernel embeddings of conditional distribu- tions: A unified kernel framework for nonparametric inference in graphical models. IEFE Signal Processing Magazine, 30(4):98-11)) 11198111,, 2013.
Akash Srivastava, Lazar Valkov, Chris Russell, Michael U Gutmann, and Charles Sutton. Veegan: Reducing mode collapse in gans using implicit variational learning. arXiv preprial or Xiv: 775.100000000 077617 2017.
Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoclkopf. Wassorstein aulo-encoders. In International Conference on Learning Represenlations, 2018. URL
https://openrrviiet wtwpe/rwwwwwwweeeeeeeeeeeeeeeeee
net .ett://oorumm.deek///1111-0
Stanislaw M Ulam. A collection of mathematical problems. New York, 29, 1960.
Arash Vahdat and Jan Kantz. NVAE: A deep hierarchical variational autoencoder. In Newral Information Processing Systems (NeurIPS), 2020.
12

---

Under review as a conference paper at ICLR 2022
Christopher Williams and Matthias Seeger. Using the nyström method to speed up kernel machines. In T. Leen, T. Dietterich, and V. Tresp (eds.), Advances in Newral Information Processing Systems, volume 13. MIT Press, 2001. URL httpps:/pprocceednggg .nwuigis.eippss Papeeerrr..-........
Matthew O Williams, Ioannis G Kevrekidis, and Clarence W Rowley. A data driven approximation of the koopman operator: Extending dynamic mode decomposition. Journal of Nonlinear Science, 25(6):1307-1345) 2015a.
Matthew O. Williams, Ioannis G. Kevrekidis, and Clarence W. Rowley. A data driven approximation of the koopman operator: Extending dynamic mode decomposition. Journal of Nonlineor Science, 25(6), 2015b. doi:
0t07///00333-015-9955555
Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nyströmformer: A nyström-based algorithm for approximating sell-attention. In Thirly-Fifth AAAI Conference on Artificial Inlelligence, AAAI 2021, Thirty- Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021,
The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021,
Virtual Event, February 2-9, 2021, pp. 14138-14144 AAAT Press, 2021. URL. https 7/ojs. //ojs.aai.coo
wwa..aai.og....... orggi.og//aa index. pttppp//AA//arrrille//ete///////
Hao Henry Zhou, Vikas Singh, Sterling C. Johnson, and Grace Wahba. Statistical tests and identifiability conditions for pooling and analyzing multisite datasets. Proceedings of the National Academy of Sciences, 115(7):1481 -144486 2018. ISSN 0027-8424. doi:
10.1073/ 10.1073/pnas. paa...777.7717.111.....
13

---

Under review as a conference paper at ICLR 2022
A CHOICE OF KERNEL IS RELEVANT YET FLEXIBLE
In some cases, one would focus on identifying (finite) eigenfuntions and modes of the underlying operator (Williams et al. 2015b Brunton et al. 2016). But rather than finding certain modes that best characterize the dynamics, we care most about minimizing the crror of the transferred density and therefore whether the span of functions is rich exppeessie enough. In particular, condition (iii) in Proposition 3.2 requires an imput RKHS spanned by sufficiently rich bases (Fukumizu al. 2013). For this reason, the choice of kernel here cannot be completely ignored since it determines the family of functions contained in the induced RKHS.
To explore the appropriate kernel setup for our application, we empirically evaluate the effect of using several different kernels via simple experiment on MNIST. We first train autoencoder to embed MNIST digits on to a hypersphere S², then generate samples from kPF by the procedure described by Alg. [ using the respective kernel function as the input kernel k. Subplot (b) and (c) in Fig. 7 show the generated samples using Radial Basis Function (RBF) kernel and arc-cosine kernel, respectively. Observe that the choice of kernel has an influence the sample population, and kernel function with superior empirical behavior is desirable.
Motivated by this obscrvation, evaluated the Neural Tangent Kernel (NTK) (Jacot et al. 2018), a well-studied neural kornel in recent works. We use it for a few reasons, (a)NTK, in Cheory, corresponds to trained infinitely-wide neural network, which spans a rich set of functions that satisfies the assumption. on hypersphere, the positive-defnniteness of NTK is proved in (Jacot et al. 2018). Therefore, invertibility of the Gram matrix Kzz = 4 is almost guaranteed if the prior distribution Pz is restricted on a hypersphere (c) NTK can be con-asymptotically approximated (Arora) let al. 2019). (d) Unlike other parametric kernels such as RBF kernels, NTK is less sensitive to hyperparametters as long as the number of units used is large enough (Arora et al. 2019). Subplot (d) of Fig, shows that kPF learned with NTK input kernel is able to generate samples that are more consistent with the data distribution. However, we should note that NTK is merely convenient choice of kernel that requires less tuning, and is not otherwise central to our work. In fact, as shown in our experiment in Tab. 2 a well-tuned RBF kernel may also achieve similar performance. Indeed, in practice, any suitable choice of kernel may be conveniently adopted into the proposed framework without major modifications.
(b) For well-conditioned inputs (i.e., no duplicates)
Data letent points
Sampled (RDF)
Sempled points (Are-cosine)
Sampled points (NTI)
Figure 7: 10k samples from MNIST dataset (left to right) (a) projected on S shown in (0, using auto-encoder, and 10K generated samples from kPF with imput kernel of type (b) RBF (c) arccos (d) NTK. Color of sampled points represents the class of their nearest training set neighbor in the output RKHS.
14

---

Under review as a conference paper at ICLR 2022
B DENSITY ESTIMATION WITH KPF OPERATOR
The displayed transferred density with the kPF operator on toy data in Fig. is approzimated using the empirical kernel conditional density operator (CDO) (Schuster et al. 2020a), since there is currently no known methods that can exactly reconstruct density from the transferred mean embeddings. The marginalized transferred density pP: has the following form
PP: = Cp = -x Cx2C2z2z
(11)
where Cy=Eynpl(ig) p in G. The above density function is also element of RKHS 9, and therefore evaluate the density at x= by using the reproducing property Ppe (p=(x)) = {pp,, [(=x,,)))) The results in Schuster et al. (2020a) show that the empirical estimate of PP3 may be constructed from m samples of the relerence density hejml and n training samples (xiheej and {2s he()) as
((y, is the covariance operator of a independent reference density
m
PP. = + - all)
Σ Baly)s))
(12)
where
B = m (Ly +a'l) Lyx(K2 + +al)
(13)
and
Ly [116,,3) Villes E mmxm
Lyx [1(16, (z)lis Rmxx Kz = [k(z,, 2z)lis Rx
(14)
uniform density in the square (+4, +-4) as the reference density p and constructed PP: using m = 10000 samples [yi]imm]] from p. Due to the form of the empirical kernel CDO, where the estimated density function PP, is a linear combination of (1(ys) )em (as Eq. 12 the approximation can be inaccurate if reference samples are relatively sparse and the densities are 'sharp'. In those cases, to obtain better density estimate, either increase the number reference samples used to construct the empirical CDO (which be computationally difficult due to the need to compute (Ly + or, with some prior knowledge to the true distribution, choose a reference density which is localized around the ground truth density.
In Fig.
may
Figure 8: Inaccurate density estimation result from kernel CDO using 10k samples from uniform reference density P
Therefore, show more faithful density estimate of the transferred distribution for visualization purpose, we use composite of the uniform density and the true density with weight 4 as the reference density in Fig. 2 In this case, approximately 20% of the reference samples concentrates around the high-density areas of the true density, which helps to form a better basis for Ppc- Note that the choice of does not affect the transferred density embedding PeAz since is independent px and pz- After this modification, the reconstructed density more accurately reflects the true density compared to GMM and GLOW, indicating the transferred distribution by kPF in RKHS matches better to the ground truth distribution. This also reflected in the generated samples in Fig. 9 where samples generated by the kPF operator are clearly more aligned with the ground truth distribution.
15

---

Under review as a conference paper at ICLR 2022
GT
GMM
Glow
NTK-
kPF
Figure Sample comparisons of the distribution matching methods
16

---

Under review as a conference paper at ICLR 2022
C EFFECT OF 7 ON SAMPLE QUALITY
In the sampling stage, our proposed method finds the approximate preimage of the transferred kernel embeddings by taking the weighted Fréchet mcan of the top y neighbors among the training samples. The choice of y therefore influences the quality of generation.
From Figure 10 can observe that, in general, FID worsens increases. This observation aligns with our inluition of preserving only the local similarities represented by the kernel, and similar ideas have been previously used in the literature (Hastie et al. 2001 Kwok & Tsang 2004). However, significantly decreasing y leads to the undesirable result where the generator merely generates the training samples (in the extreme case where y 1, generated samples will just be roconstructions of training samples). Therefore, in our experiments, we choose y to achieve balance between generation quality and the distance training samples.
100
MNIST
CelrbA
Size of neighborhood 7
Figure 10: FID versus y few computer vision datasets.
17

---

Under review as a conference paper at ICLR 2022
D WEIGHTED FRÉCHET MEAN ON THE HYPERSPHERE
While the weighted Fréchet Mean Euclidean space can be computed in closed-fore as a weighted arithmetic mean (as in Eq. 10 the hypersphere there is known closed-form solution. Thus, we adopt the iterative algorithm in (Chakraborty & Vemuri, 2015) for an approximate solution given data points X (xi.... (XI...X, .,.,,, and weight vector S:
M1
M4+1 cos( IIM + sin(I[sa+i]] II)
M
where, v sin(0) (Xi+1 M, cos( A ( This algorithm iterates through
the data points once, yielding a complexity of only O(rd), where d is the dimension of X. Under the prescribed iteration, M, converges asymptotically the true Fréchct mcan for finite data points. We refer the readers to (Chakraborty & Vemuri 2015) for further details.
18

---

Under review as a conference paper at ICLR 2022
E FAST APPROXIMATION OF MOORE-PENROSE INVERSE
When computing the inverted kernel matrix Kiny in Algo. I conventional approaches typically performs SVD Cholesky decomposition. Both procedures are hard to parallelize, and thercfore, can be slow when is large. Alternatively, we can utilize an itcrative procedure proposed in Razavi et al. (2014) to approximate the Moore-Penrose inverse.
11 = |/([
(15)
7++1 2,(13T KZ, -
(16)
where
| max
Kij | bc max
Kij
(17)
Since this iterative procedure mostly involves matrix multiplications, it can be efficiently parallelized and implemented on GPU, The same procedure has also seen success in approx- imating large self-attention matrices in language modeling (Xiong et al. 2021 For the NVAE experiment, we run this itcration for 10 steps and use Kiuv Z10-
19

---

Under review as a conference paper at ICLR 2022
F NYSTROM APPROXIMATION OF KPF
Due to the need to store and compute a kernel matrix inverse (K + An!) or K", the memory and computational complexity of kPF is at least O(n²) and O(n²), respectively. The sup-quadratic complexity hinders the use kPF on extremcly large datasets. In our experiments, we already adopted simple subsampling strategy which randomly select 10k training samples from each dataset (s. 50k samples) to fit our hardware configuration which works well. But for larger datasets with potentially more modes, a larger set of subsamples must be considered, and in those cases kPF may not be suitable for commodity/ /affordable hardware. In order to overcome this problem, we can combine kPF with conventional kernel approximation methods such as the Nyström method (Williams & Seeger b.
Let (X., Z.) be a size subsct of the training set (which we refer as the landmark points) and (V.,, 2.) be their corresponding kernel feature maps. The weighting coefficients for each prior sample z" ~ Z derived in Alg,
[ can be approximated by
s L(K A k(=",-)
~ Lvv. Wt, Lid, (Kr, W1, KT. + (n=))
Lv. Wh. Lg (An)= Ko (AnW), + Ko. Ko, (' k(2",))
(18)
where Lv. 1 TV. E Raxx, We =V E Ruxe Ko. =T E Raxv Wa, = = o.
Rexe, and the last identity is due to applying the Woodbury formula (Kg, W KT. +XnI)1. Assuming < n, the memory complexity reduced to ((wv) and the computation complexity to O(no²).
We empirically evaluated the Nyström-approximatee kPF on the CelebA experiment and It can be observed that when v is sufficiently large, the performance of Nyström approximated kPF's as good the ones using the full kernel
present the result in Tab. !
matrices.
100
500
1000
w/o Approximation
10,000
45.9
41.6
42.3
41.8
30,000
46.3
40.5
42.4
50,000
45.2
44.1
42.0
Table 4: FIDs of samples generated by Nyström-ane ceström-approxiimate NTK-kPF on CelebA. n denotes the size of the training data subset we consider in computing kPF, while v denotes the size of selected landmark points for Nyström approximation. Without approximation, we cannot fit the kernel matrices onto a GPU with 11GB RAM when n > 10, 000. It is worth noting that the approximated kPFs can perform similarly to the full kPF even with < 0.05n, which indicates that Nyström approximation does not sacrifice much in terms of performance while delivering significant efficiency gain.
20

---

Under review as a conference paper at ICLR 2022
G DOES kPF MEMORIZE TRAINING DATA?
Since in kPF, samples are generated by linearly interpolating between training samples, it is natural to wonder whether it 'fools' the metrics by simply replicating training samples. For comparison, we consider an alternative scheme that generates data through direct manipulation of the training data, namely Kernel Density Estimation (KDE).
We fit KDEs by varying noise levels and compare their FIDs and nearest samples in the latent space to kPF in Fig. 11 We observe that, although KDE can reach very low FIDs when a is small, almost all new samples closely resemble some instance in the training set, which clear indication of memorization. In contrast, kPF can generate diverse samples that do not simply replicate the observed data.
KDE, a= = 0.005, FID: 30.9
(b) KDE, a a= 0.01, FID: 36.3
KDE, = 0.02, FID: 48.0
(d) kPF, FID: 41.0
Figure 11: Comparing KDE to kPF. Generated samples are presented in D. followed by their 5 nearest neighbors in the latent space (ordered from closest to furthest)
21

---

Under review as a conference paper at ICLR 2022
H ASSESSING SAMPLE DIVERSITY
Although FID is one the most frequently used measures for assessing sample quality of generative models, certain diversity considerations, such mode collapse, may not be conveniently deduced from it (Sajjadi ct al. 2018). To enable explicit examination of generative models with respect both accuracy (i.e., generating samples within the support of the data distribution) and diversity (i.e., covering the support of the data distribution
as much as possible), Sajjadi al. 2018 proposed an approach to evaluate generative models with generalized dehniiions of precision and recall between distributions. Quality of generation can then be assessed by evaluating the PRD curve, which depicts the trade-offs between accuracy (precision) and diversity (recall). We present the PRD curves in Fig. 12
The observations align with our results in Tab. 2 and kPF performs competitively in both accuracy and sample diversity.
MNIST
CIFAR-10
CelebA
1.0
1.0
SBAEure-aur
0..0
SHARROENTT
SRLARar
0.0
SHALEN
SILAEake
0.
SRAEnis
SRA.Eass
SRAEGLS
SRAAR
Varilla VAR
Twoosiigg VAI
Vonilla VAF.
Two stage VAB
0.2
0.5
0.2
Vinilo VAB
0.0
0.0
0.0
Figure 12: PRD curves on all datasets. kPF is competitive to the other methods terms of Area Under Curve (AUC)
22

---

Under review as a conference paper at ICLR 2022
I EXPLORING KERNEL CONFIGURATIONS
To investigate the implication kernel choices on generation quality, we tested 25 different kernel configurations for CelebA generation (results presented in Tab. For RBF kernels used in the CelebA experiments of the main lext, we bandwidth of on 2 0.71 when used as input kernel and Jout R 0.34, and we adopt the same notation here.
Output kernel
RBF
RBF (ar = dout (2)
RBF (o Gout)
RBF (ar 20
RBF (ar = 40
Inpul kernel
(a Jout/ /4)
RBF (a 0tn/2)
41.50
41.20
41.21
50.92
66.06
RBF (ar din)
41.90
42.11
41.91
45.83
50.70
RBF
42.20
42.82
42.69
65.76
70.19
(a 20
NTK
41.90
41.56
41.73
37.86
37.89
(L 8, 10,000)
Arecos
41.71
42.03
42.22
52.83
63.13
(L 1, deg 1)
Table 5: FID table for different kernel configurations.
can seen that kernel configurations and parameters indeed has a non-trivia impact on the generation quality, with NTK-kPF being the most robust to the choice of parameters. This aligns with our previous observations and offers some support for using NTK as an input kernel despite the additional compute cost.
23

---

Under review as a conference paper at ICLR 2022
J EXPERIMENTAL DETAILS
In this section, we provide the detailed specifications for all of our experiments. We have also provided our code in the supplemental material.
J.1 DENSITY ESTIMATION ON Toy DENSITIES
We generaled 10000 samples from each of the toy densities to learn the kPF operator. The input kernel & is ReI.U-activated NTK corresponding to fully-connected network with depth and width w = 10000 at each layer, and the output kernel is a Gaussian kernel. Unless specified otherwise, always uses Gaussian kernel as the output kernel for the remainder of this appendix. The bandwidth of the output kernel was adjusted scparately for density estimation and sampling for the purpose of demonstration. For comparison, also fit, /tstetimatee a 10-component GMM and a Glow model with 50 coupling layers, where each of them were trained until convergence.
J.2 IMAGE GENERATION WITH COMPUTER VISION DATASETS
To generate results in Tab. 2 and Tab. we first trained an autoencoder for each datasct following the model sctup GGosh ct al., 2020), which uses modified Wasserstein autoencoder (Tolstikhin et al. 2018) architect with batch normalization. Additionally, we applied spectral normalization on both the encoder and the decoder, following (Ghosh let al.) 2020), to obtain regularized autoencoder. The latent representations were projected onto hypersphere before decoding to image space. We trained the models on two NVIDIA GTX 1080TT GPUs. A detailed model specification is provided below in Table 6
We used an NTK with L and = 10000 as the input kernel (i.e. the embodding kernel of pz) for NTK-kPF, and a Gaussian kernel with bandwidth omn = / for RBF-KPF, The bandwidth for the output Gaussian kernel is selected by grid search over x E [8]}, where daata is the empirical data standard deviation, based on cross- validation of degree 3 polynomial kernel MMD between the sampled and the ground-truth latent points. Further, to mitigate the deterioration of performance of kernel methods in high-dimensional setting due the curse of dimensionality (Evangelista et al. 2006), in practice, we model Z as a space with fewer dimensions than the input space X. As a rule of the thumb, we choose Z such that | X//4.
To generate images from kPF learned on NVAE latent space, we used the pre-trained checkpoints provided in (Vahdat & Kautz 2020) to obtain the latent embeddings for 2000 FFHQ images. We then construct the kPF from the concatenated latent space of the lowest resolution (8 8). During sampling, prior samples at those resolutions are replace by the kPF samples, while for other resolutions samples remain generated from inferred Gaussian distributions. The batchnorm statistics were readjusted for 500 iterations following (Vahdat & Kautz 2020). We use rbf kernels as input and output kernels, with bandwidths σe, 9) chosen by the median heuristic (~ 100 for input and ~ 70 for output in our experiments).
J.3 IMAGE GENERATION FOR BRAIN IMAGES
For the high-resolution brain imaging dataset, we used custom version of ResNet (He et al.) 2016) with 3D convolutions. The detailed architecture is shown in Fig, 7 Due to the large size of the data, we trained the model on 4 NVIDIA Tesla V100 GPUs.
Mandatory ADNI statement regarding data use. Data used in preparation of this article were obtained from the Alzheimer's Disease Neuroimaging Initiative (ADNI) database www............................) As such, the investigators within the ADNI contributed to the design and implementation of ADNI and/or provided data but did not participate in analysis or writing of this report. A complete listing of ADNI investigators can be found in the ADNI Acknowledgement List.
The T1 MR brain dataset we utilize consists of images from 184 subjects diagnosed with Alzheimer''s s disease and 292 healthy controls/ normal subjects. Images were first coregistered
24

---

Under review as a conference paper at ICLR 2022
MNIST
CIFAR-10
CelebA
5x5 conv, stride
ResBlockes x 2
Conv 4x4
128
Conviss 4x4
128
Conv 5xc5
122
Encoder
ResBlockes X 2
Conv 4x4
Conv266 4x4
Conv 5x55
ResBlocki28
Encoder
Convs12 4
Conv 4x4
Conv 512
3x55
ResBlock2se
Conv 4x4
Conv 1024
Conv
1024
ResBlockass x
ConvT:3
512
ConvTT
512
Convisse
512
ResBlocki2s x
Decoder
Conv'l T
4x4
256
Conv
Conv'T 256
Decoder
ResBlockas
ConvT1"
Conv T
Conv'T CBn
122
ResBlockes
Conv'
5x5 conv, stride
Table 6: Model architecture for computer vision experiments. Subscript denotes the number out- put channels and superscript denotes the window size of the convolution kernel. Batch normaliza- tion and activation is applied between cach pair of convolution layers
Table 7: Model architecture for exper- iments for image generation on brain imaging dataset. Subscript denotes the number of output channels. Up- sampling and downsampling per- formed using strided convolutions.
to a MNI template and segmented to preserve only the white matter and grey matter. Then, all images were resliced and resized to 160 x 196 x 160 and rescaled to the range of [-1, 1]. Voxel- based morphometry (VBM) was used to obtain the p-value map data and generated images.
25

---

Under review as a conference paper at ICLR 2022
K MORE SAMPLES
In this section we present additional uncurated set of samples on MNIST, CIFAR-10, CelebA based pre-trained SRAE and FFHQ based NVAE. From the figures, it can be seen that kPF produces consistent and diverse samples, often better in quality than the alternatives.
10
0
041276 7
0
5 3
8 2
k
3
0
7
6
8
111 3339
1
5
531
3
2
2
7
8
5 0
6
3 2
2 0 2
1
314
5
4
8
2
3
/ 2 2 7 x
8
/
3
8 9 982009
4 2
1
8
5
6 2 0 10640477
5
2
5
6
0
1
9
9
0 6 6
2
0
7 8
0 7
7
0
7 7
4
0698796
6
9 8
7
9
6
/
0
s
3
6
100
S
4
3
2
7
9
9
9
2
9
2
4
3
7
4
1
1
110
1
3
5 8
5
8
2
8165400
8
1
6
4
0
0 9
/
6
0
7
6
9
3
666677
07
7
2
/
9
9
1
8
6
3
45645/
3
00034000 2
3
4
Figure 13: MNIST results from SRAEGxM (top left), SRAEG (top right) and our SRAFNTKAPP.
26

---

Under review as a conference paper at ICLR 2022
Figure 14: CIFAR results from SRAEGMM (top left), SRAEGlow (top right) and our SRAENTK-APF.
27

---

Under review as a conference paper at ICLR 2022
Figure 15: CelebA results from SRAEGMM (lop left), SRAEGioW (top right) and our SRAENTK-LPF (bottom).
28

---

Under review as a conference paper at ICLR 2022
Figure 16: Additional samples from +NV+N pre-trained on FFHQ
29


Reviews:
[{"review_id": "uBQREjb-nwS", "reviewer": "Reviewer_mptM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors propose a new method for an important problem in machine learning. The new perspective of incorporating RKHS tools into generative models is interesting and novel. Nonetheless, I am not completely convinced that the method leads to a substantial benefit compared to existing approaches. My main comments are :\nSince the method relies on interpolation using RKHS it probably can’t extrapolate beyond the support of the given distribution. Doesn’t this limit the method in terms of its ability to generate new samples? I mean here that FID should not be the only metric for evaluating a generative model; other metrics are more meaningful in that sense, for example, in terms of the diversity of the generated samples.\n-The authors demonstrate that their method has low runtime compared to several baselines; however, some methods are missing from these evaluations—namely, Vanilla VAE and GMM.\n-On the same point, what is the inference time required by the method to generate a new sample? I think it should be longer than of a vanilla VAE.\n-Can the method be used for extremely large datasets? The kernel inverse calculation should take long and also is limited by the memory restrictions.\nMinor comments:\n-we often approaches-> we often approach\n-by via -> remove by\n-SGD mentioned before it is defined.\n-the eq. above eq. 2 ends with a comma, should be period.\n-period missing after eq. 2.\n-kPF mentioned in caption 1, before it is defined.\n-The caption in figure 1 doesn’t provide a clear explanation for the proposed scheme, this should be extended.\n-”A natural extension of Klaus et al.”-> perhaps you mean “A natural extension of the work of Klaus et al.”\n-In general some of the citations should be in brackets.\n-”There, for” this sentence is not clear.\n-”See Klaus” - why is See capitalized.\n-X’ and x’ are not well defined.\n-step 10 in algorithm 1- this is python notation; perhaps you want to use a mathematical notation?\n-NVAE in the caption of figure 3 is only defined on the next page. Please rearrange.\n-SRAE is also only defined after its first mention.\n", "summary_of_the_paper": "The authors propose a new type of generative model. The new scheme is based on a kernel transfer operator that leads to a cheap method for distribution matching. The authors rely on rigorous theory on RKHS and propose a framework for transferring a prior distribution linearly (in RKHS) to the data distribution. The authors demonstrate that the proposed approach leads to improved approximations of observed distributions. Specifically, the new approach can lead to the generation of new images form a given distribution and requires less training time compared to existing baselines. The paper is mostly well written, and the method relies on solid justifications. The authors demonstrate the usefulness of the method and compare it to several existing methods. ", "summary_of_the_review": "To summarize, the authors address an important problem and propose a new generative model with several benefits over existing methods in the low sample size regime. The results demonstrate that the method leads to improvement in retrieval tasks. However, the technique has some limitations (namely, the inference may take a long time, and there might be memory limitations for large datasets). The experiments do not provide a complete picture of the method's capabilities, namely how diverse the new points are and whether they really differ from the original samples. To demonstrate that a generative model is useful, the authors could try to use the model to enrich imbalanced datasets and see if clustering or classification improves.\nFor these reasons, I recommend a weak rejection of the paper.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "rVxMpDV1L5z", "reviewer": "Reviewer_wZ8o", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n* A conceptually simple kernel method to model densities in latent spaces of AEs.\n* A simple linear operator in RKHS that maps the embedded base to the embedded data distribution, neat idea!\n* Results appear to be comparable to sota methods of density modelling, while an order of magnitude faster.\n\nWeak points:\n\nNovelty / Presentation\nSection 3 of the paper on first instance looks like the authors are presenting a novel concept here. However, the transfer operator is just the conditional embedding operator of Song et al, marginalized over the conditioning variable / applied to the mean embedding of Z, plus a pre-image estimation (see alos below). The equations are exactly the same as in e.g. Song et al 2013 (e.g. see Fig 5 in that paper).\nThere is a lot of equation shuffling where definitions are plugged into each other, (eq 6, eq 7, and the unnumbered eq before 6, Proposition 3.3 and its proof), that re-write definitions of the underlying operators in one form or another. I am quite skeptical about this form of presenting the method, and about the usefulness of these equations.\nIn addition, the connections to dynamical systems (all of which are reviewing external literature) do not appear to be necessary for the rest of the paper, especially considering Song's work, which is essentially the RKHS version of the standard rules of conditional probability.\n\nPreimages.\nAs the authors mention, pre-images are problematic as RKHS embeddings are not necessarily surjective. I am therefore skeptical about the deterministic pre-image estimation methods presented here, especially for sampling. What if the distribution represented by \\mu_X is multi-modal? The same point z could be mapped to two totally different x in that case, but with this deterministic approach, it won't. This should be discussed and investigated, even if performance suggests that this is not necessarily a problem.\n\nComputational costs.\nThe method needs to store, and invert a kernel matrix between the data points, which costs O(n^2) and O(n^3) respectively. This becomes prohibitive from a few thousand points. I am very surprised that this is supposedly computationally cheaper that other methods. I also think the comparison done is very vague: the kernel method is dominated by a Cholesky decomposition / matrix inversion, while other methods rely on SGD/etc. So supposedly, there is a convergence question in Fig 5. Wouldn't it be better to present the other methods on a graph time-vs-accuracy? I find it also find \"additional training time\" confusing, and it is not very clear what is exactly compared here.\nQuestion: The provided code contains code for Nystrom approximations. Are these actually used in the experiments? (Seems hard to invert a 10k kernel matrix otherwise). If so this should be described/mentioned.\n\nPre-trained AEs.\nI understand that the presented method will not work well when applied directly on high-dimensional data, so the using an AE and work in its latent space makes sense. What I do not understand however, is how this can be compared (in terms of compute and performance) to the other methods in the paper. E.g. How is the method compared to a VAE when \"all models share the same encoder/decoder\"? Are the other models applied on the original data, or on the AE latents? Does it make sense to apply e.g. GLOW to AE latents? It would be great to make this more clear. For the NVAE case, I am even more confused: do we first train and NVAE, then use the kernel method in its latent space, and then compare it to the NVAE that we trained in the first place?\n\nBrain data experiment.\nIt is a bit unclear why generating samples from this dataset would be desirable, how these samples can be \"scientifically meaningful\", and what the results show (e.g. colourbar is not annotated, etc). Some more motivation would help.\n\nFinally, comparisons to GANs would be appropriate for the way the method is evaluated (only samples, density only shown for the toy examples)\n\n\nMinor:\n* Double period p2 \"deep generative models and dynamical systems..\"\n* Inconsistent pre-trained vs pretrained.\n* Typo in Intro \"we often approaches this problem\"\n* \"Use mean embedded operator\" -> \"Use mean embedding operator\"\n* p1 \"the lower bound of likelihood\" -> \"a lower bound of the likelihood\"\n* many times citations are not properly punctuated, i.e. citep should be used\n* The tensor-product operator in eq 1 is not defined.\n* \"Let us temporarily assume Z=X\" ... this is never resolved.\n* What does point (b) mean \"The training scheme directly learns the mapping from the prior rather than approximation\" ? ", "summary_of_the_paper": "The paper proposes a density modelling approach that is based on applying conditional embedding operators to RKHS embeddings of a base distribution.\nThe approach, thanks to use results from the kernel literature, is conceptually simpler and appears less computation heavy compared to VAE/flow models, while at the same time performing favorably.", "summary_of_the_review": "Great idea, though there are some issues with presentation and experiments. Not very convincing as is, but a lot of potential to be improved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HVVqd564AN", "reviewer": "Reviewer_UugV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n- The idea of using kernel transfer operators for generative modeling is novel and interesting, and. The empirical results on various image datasets are also quite good.\n\nWeaknesses: \n- Notation is a bit confusing at times, as some items are not defined. For example, the authors should define the tensor product operation in Definition 2.3. Additionally, \\Delta_t in the integral right before Eq. (2) is not defined, and I’m not sure what z(.) is supposed to be when describing the underlying dynamics in Section 3. And why is the forward operator defined as f^* = (I + g^*)?\n\nQuestions:\n- Why can we just embed Z into X as mentioned in the 1st paragraph of Section 3?\n\nMiscellaneous/minor comments and typos:\n- There are numerous typos/awkward sentence phrasings that should be cleaned up.\n- “We often [approach] this” in the intro\n- Citation formatting is confusing (should use parentheses)\n- “We [demonstrate]” at the end of Section 1\n- “MDS-based” in Section 4.1\n\n", "summary_of_the_paper": "The paper proposes to leverage the Perron-Frobenius operator to simplify the mapping from Z -> X in latent variable generative models. Specifically, this “forward operator” can be approximated by a closed-form linear operator in the RKHS. They evaluate their method on both synthetic settings and high-dimensional image datasets such as MNIST, CIFAR-10, CelebA, and FFHQ.", "summary_of_the_review": "This paper proposes to use kernel transfer operators for generative models, by leveraging a kernelized embedding of the “forward operator” (generative model). The authors experiment with various kernels and obtain favorable results on high-dimensional image generation tasks.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "F6d6TCyjDZ2", "reviewer": "Reviewer_hWmF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I'm deeply impressed by the novelty. It is a smart idea to start with the formulation of data generation as transforming a tractable prior to the data distribution, and then implement the formulation in RKHSs using a corresponding transfer operator. Up to my knowledge I did not notice similar ideas before. The technical details seem good to me. The paper also presents a satisfying introduction to involved techniques (nevertheless the paper may need to highlight the novel technical contribution).\n\nHowever, the generative modeling approach is actually quite different from common ones, which leads to my two major concerns.\n* The presentation seems misleading.\n  First comes my understanding of the method. It is basically not of the same type of common deep generative models: it is non-parametric, it does not have a training stage, and its generation process is a manipulation on all training examples. In this sense, it is of the same type of kernel density estimation (KDE). Also, the method constructs the transfer operator by assuming the independence between $z$ and $x$. This makes $z$ not the usual latent variable: it does not hold any information of data, but would rather act as the random seed. (In analogy to KDE, this $z$ may correspond to the random noise around each data point).\n\n  What I felt most misleading is the title. There is no distribution matching: the method constructs a transfer operator based on the distributions at the two ends, instead of matching a learnable distribution to a fixed target distribution. Also, the method itself is not a deep generative model: it is non-parametric (does not involve a parametric model), so \"deep\" does not make sense (the deep AE is independently pretrained).\n\n  I may prefer presenting the method as it is and highlighting its unique utility/benefit as it is (instead of telling its benefits as a deep generative model, on which see the issues below; it could have other utility like learning-free/model-free data augmentation).\n\n* Practical usefulness as a generative model.\n  - Calculating the inverse of Gram matrix (Step 6 in Alg. 1) seems to cost a complexity cubic to the dataset size. Although Fig. 5 shows the training time comparison, I think asymptotically, increasing the dataset size may not lower the convergence time of parametric methods (as long as the data samples are iid), but the cost of the proposed method increases cubicly anyway.\n  - Following the above point, in the development of generative models people are trying to explain that the model is not just remembering the training data. This type of method instead just generates by manipulating training data, at the other extreme. Is this acceptable? Particularly, the time complexity for deployment / data generation, would then depend on the training dataset size.\n  - As mentioned, since $x$ and $z$ are taken as independent random variables, this $z$ cannot serve as a representation of data. (Posterior $p(z|x)$ is the prior $p(z)$. Also for this reason I did not understand the first limitation listed in Sec. 6. This is also what makes the approach in the same type of KDE.) This disallows the usage for embedding learning, manipulated generation, dimensionality reduction, clustering, etc.\n  - As also mentioned in the paper, the method requires both marginal distributions to be supported on the whole space but not on a lower-dimensional manifold. When the latter is the case, an auxiliary model is required for dimensionality reduction.\n\nI have some other minor questions.\n* As I feel the method is of the same type as KDE, an experimental comparison with KDE is expected.\n* I wonder how the choice of kernel and kernel bandwidths are set and how they influence the performance.\n* What's the difference between the Perron-Frobenius operator in Definition 3.1 and the push-forward operator of a distribution by the same map $f$ (unnecessarily invertible)? Also, the domain of $f$ (and $\\mathcal{P}$) seems better be $\\mathcal{Z}$ (and $L^1(\\mathcal{Z})$).\n* I did not quite understand the method in Sec. 4.1. Particularly, what is $X$ and what is $X'$? Which one is the output of the inverse map?\n* The generated data is a _linear_ transformation on training data samples. Is it flexible enough? Can it generate all possible samples following the data distribution?\n", "summary_of_the_paper": "The paper presents a novel (or an unusual type of) generative modeling approach that is kernel-based and non-parametric. The basic idea is using an operator that maps from the RKHS of Z to the RKHS of X, so that data generation can be done by mapping the prior distribution $p_{prior}(z)$ to the RKHS of Z, applying the operator, and project down the result in the RKHS of X to the data space. The operator is constructed as the RKHS analogy of the desired conditional distribution p(x|z) (which makes $\\int p(x|z) p_{prior}(z) dz = p_{data}(x)$), so that it transforms the RHKS embedding of $p_{prior}(z)$ to that of $p_{data}(x)$. However, to guarantee a desired $p(x|z)$, the operator is chosen by assuming $x$ and $z$ are independent. The operator can be estimated by samples of the joint distribution, which amounts to samples of $p_{data}(x)$ (i.e., training dataset) and samples from $p_{prior}(z)$ under the independent assumption, and the maps between the sample space and the RKHS can also be estimated. Experiments show the utility of the method for data generation for densely supported distributions.", "summary_of_the_review": "I appreciate the novelty and I think the idea is worth being noticed. Nevertheless the presentation (story) seems a little misleading, which may not be a proper way to present the merit of the method.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Perron-Frobenius operator (P) is a well-known tool which maps the density of a dynamical system at time t (p_t) to that at t+1 (p_{t+1}): p_{t+1} = P p_t. The idea has recently been extended (kernel Perron-Frobenius operator (kPF); Klus et al. 2020) to map a probability measure p_Z to p_X via covariance operators (4) associated to a reproducing kernel; this corresponds to the transformation of the kernel mean embedding of Z to that of X as it is recalled in (5). The authors use the kPF technique in generative modelling to map the known prior (p_Z) to the data-generating distribution (p_X), and illustrate the idea numerically.

While the focus of the paper (generative modelling) is relevant, the reviewers had several severe issues with the submission:
1) the manuscript lacks clarity of presentation at multiple points,
2) the reviewers had concerns with the scalability of the approach (which unfortunately has not been analyzed),
3) the submission is a straightforward application of a well-established tool (kPF) in the literature; the paper lacks novelty.

Significantly more effort is required before publication.
Decision: Reject