Can Kernel Transfer Operators Help Flow based Generative Models?
Authors: Zhichun Huang
Rudrasis Chakraborty
Xingjian Zhen
Vikas Singh
Venue:  (2021)

Abstract: Flow-based generative models refer to deep generative models with 
tractable likelihoods, and offer several attractive properties including 
efficient density estimation and sampling.  Despite many advantages, 
current formulations (e.g., normalizing flow) often have an expensive memory/runtime footprint, which hinders their use in a number of applications. 
In this paper, we consider the setting where we have access to an autoencoder, which is
suitably effective for the dataset of interest. Under some mild conditions,
we show that we can calculate a mapping to a RKHS which subsequently enables deploying 
mature ideas from the kernel methods literature for flow-based generative models. Specifically, we can explicitly map the RKHS distribution (i.e., 
approximate the flow) to match or align with  
a template/well-characterized distribution, via kernel transfer operators. This leads to a direct and resource efficient approximation avoiding iterative optimization. We empirically show that this simple idea yields competitive results on popular datasets such as CelebA,
as well as promising results on a public 3D brain imaging dataset where the sample sizes are much smaller. 

Under review as a conference paper at ICLR 2021
CAN KERNEL TRANSFER OPERATORS HELP FLOW BASED GENERATIVE MODELS?
Anonymous authors Paper under double-blind review
ABSTRACT
Flow-based generative models refer to deep generative models with tractable like- lihoods, and offer several attractive properties including efficient density estima- tion and sampling. Despite many advantages, current formulations (e.g.. normaliz- ing flow) often have an expensive memory/runtime footprint, which hinders their use in number of applications. In this paper, we consider the setting where we have access to an autoencoder, which is suitably effective for the dataset of in- terest. Under some mild conditions, show that can calculate mapping to a RKHS which subsequently enables deploying mature ideas from the kernel methods literature for flow-based generative models. Specifically, itly map the RKHS distribution (i.e., approximate the flow) to match or align with a eeeplatteweel-cccaaatrriieed distribution, via kernel transfer operators. This leads to direct and resource efficient approximation avoiding iterative optimization. We empirically show that this simple idea yields competitive results on popular datasets such as CelebA, as well as promising results on public 3D brain imag- ing dataset where the sample sizes are much smaller.
explic-
INTRODUCTION
flow-based generative model refers deep generative model composed using a of invertible transformations. While GANs and VAEs remain the two dominant generative models in the com- munity, flow based formulations have continually evolved and now offer competitive performance in applications including audio/speech synthesis Kim et al. (2019) (2200), text to speech Miao et al. (2020), photo-realistic image generation Kingma Dhariwal (2018), and learning cross-domain mappings Mahajan et al.] (2020). An important property of such models is the explicit use of tractable likelihood function, which enables leveraging maximum likelihood principles during train- ing well as efficient/exact density estimation and sampling. The formulation is invertible by design but this involves higher memory requirements. For example, permitting the bijective mapping to be expressive enough involves increases in the memory footprint Lee al. (2020); Kim et al. (0119), Chen
resubssDacooben an issue that is focus of several recent results Jacobsen et al. (2018); Chen et al. (2016). Moreover,
in these models need calculate the inverse and backpropagate through all invertible transfor- mations during training. Calculating the inverse incurs a multiplicative increase cost, usually as function of the feature dimension, relative the calculation of the likelihood, issue addressed to some extent in Dinh et al. (2017); Kingma & Dhariwal (2018).
At high level, flow-based generative model bijectively pushes the data density from a source to target, i.e., from a known simple distribution to an unknown (may be intractable) data distribution. During training, we seek to learn this bijective mapping by maximizing the likelihood of the mapped
Figure 1: Reeressentttive generated images (of resolution
128 x 128) using proposed algorithm.

---

Under review as a conference paper at ICLR 2021
training samples. In the generation step, need the inverse of this mapping (given such an inverse exists) to mäp from a sample drawn from the known distribution back to the input (data) space. When the Jacobian of the transformation mapping can be efficiently computed or estimated (e.g., having a lower triangular form), directly optimizing the likelihood of the training samples is possible. However, in training flow-based generative either must restrict the expressiveness at each layer or fall back on more numerically heavy solutions, see (Chen et al. 2018). Next, we discuss how several existing results may provide a simplification strategy.
1.1 RELATED WORKS AND RATIONALE
operators Song et al. 22200), that offers arguably easier, optionally linear, ieear dynamics of measurements that evolve temporally. For instance, described in Arbabi 2018 Lusch et al. 2018), if we view the data/measuurmmnts as evaluations of functions of the state (of a dynamical system) where the functions are also called observables then the entire set of such functions forms linear vector space. Transfer operators on this space describe a linear evolution of the dynamics, i.e., finite-dimensional nonlinear dynam- ics are replaced by anfnite-dimensionall linear dynamics [Brnton et al. (211)) perfectly evolving one set of measurements to another over time if the space can be well characterized. Of course, this is not practically beneficial because constructing such infinite-dimensional spaces could be in- tractable. Nonetheless, results in optimal control demonstrate that the idea can still be effective in specific cases, using approximations with either spectral analysis of large but finite number of func- tions Williams et al. (2015) via a search for potential eigenfunctions of the operators using neural networks Li et al. (207)); Lusch et al. 2018)). Within the last year, several results describe its poten- tial benefits of such operators in machine learning problems well Li et al. (2020): Azencot et al. 2020).
Our starting point is the existing literature on Koopman and
22009; Fukumizu et al. 2 Klus et al.
procedure that can be used to analyz yze non-
If we consider the transformations that flow-based generative models learn as a mon-linear dynamics, also used in (Chen et al. 2018). data-driven approximation strategy one can consider is to map the given data (or distribution) into infinite dimension space of functions through the kernel trick, which may allow the use of well known results based on kernel methods, including old and new results on powerful neural kernels Neal (1994); Jacot al. (2018); Arora et al. (019). Utilizing these results, a mean embedding on the corresponding Reproducing Kernel Hilbert Space (RKHS) would correspond to the distribution in the imput space (the distribution from which input samples are drawn). Therefore, the problem of identifying a nonlinear mapping (or dynamics) in the input space (going from an intractable distribution to a known distribution or vice-versa) reduces to estimating a linear mapping operator between two empirical mean kernel embeddings where recent results on kernel transfer operators Klus et al. (2020) could be relevant or applicable. However, due to the high variability of the data, estimation of the distribution directly in the input space, as we will see shortly, can be difficult. But, if the input space is dinenssiiionll or otherwise structured, this problem could be mitigated. Fortunately, for many image datasets, can identify such that, in theory, the above pipeline could be instantiated, enabling us to learn a transfer operator.
-iw-dimesiional latent space
Conceptually, it is not difficult to how the foregoing idea could potentially help (or simplify) flow-based generative models. In principle, using a transfer operator, one could push-forward the input data distribution a target distribution of our choice, if both have already been mapped to a sufficiently high dimensional space. If - additionally - the operator could also be inverted, this strategy may, least, be viable. Of course, several key components are missing. We need to (a) as- sess if setting up suitable infinite dimensional space is possible, (b) identify if we can estimate the transfer operator and then finally, (c) check if the procedure works at all. In the following sections of the paper, we will verify these key components and show that using only linear operator yield surprisingly competitive results on several image generation tasks.
PRELIMINARIES
Auto-encoders. Images often lie close to an (unknown) lower dimensional manifold M C R such that dim(M) < m, and operating with densities in a lower dimensional setting is often much easier.
2

---

Under review as a conference paper at ICLR 2021
VAEs leverage this explicitly via auto-encoders, If the parameterized empirical density is pr (z). in VAEs, we write it as (=(2))((()d)) where is the low dimensional representation with a suitable prior. We then use decoder distribution 9- (2/2z and an encoder distribution (=(x12) and train the parameters 7 and T'. In practice, (-) and (-((-) are assumed to be Gaussian, but it is known that jointly fitting the manifold as well regressing to a particular prior distribution can be challenging in VAEs Kingma et al.) (2016); Dai & Wip (2019). Now, consider the following approach, where we do not impose a distributional assumption on z. If a well-regularized auto-encoder is able to capture information about the data generating density Alain & Bengio (2014), we can think of as the input measurements (likely, meaningful representations from the input data) which is subsequently mapped to an infinite dimensional space (of observables of these measurements). Now, if we could push-forward the embedded RKHS distribution to the RKHS mapping of a simpler distribution, similar to flow-based generation, one could easily sample from the simple (e.g., standard normal) distribution and transform it via the learned mapping to samples on the latent space. In summary, instead of explicitly searching for the cigenfunctions, an auto-encoder into a RKHS, Step 2: learn a kernel transfer operator in RKHS in one step. In the remainder of this paper, we will provide the details operationalize this idea and show that this simple approach, in fact, performs surprisingly well with highly favorable computational properties.
propose to Step 1: embed the density from
We now introduce the definition of reproducing kernel Hilbert space (RKHS) and the kernel embed- ding of probability distributions which are the building blocks of this paper.
Definition 1 (RKHS Aronszajn 1950)). Given set X and H be set of functions 6 6:X- R. H. is called a reproducing kernel Hilbert space (RKHS) with corresponding product (...) if there exists function k X X - R. (called reproducing kernel) such that (i) VX E X, HX,(X) (0) (XX,))) (ii) H (pan(()(XX,-);, X E X}), where is the completion.
The kernel mean embedding can be used to embed a probability measure in a RKHS.
Definition 2 (Kernel Mean Embedding Smola et al. (2007)). Given a probability measure p on X with an associated RKHS H equipped with a reproducing kernel i such that supxex k(X,X) < 00, the kernel mean embedding of in RKHS H. denoted by jau. E H. is defined as Pn ()((X, and the mean embedding operator E: L (X) - H defined as pn Ep.
For a characteristic kemel, the mapping from an input space distribution to its corresponding kernel mean embedding is one-to-one. Thus, two distributions in the input space are identical if and only if their kernel mean embedding matches exactly. This property enables Maximum Mean Discrep- ancy (MMD) for distribution matching. For finite number of samples (X.) drawn from the probability measure 14, an empirical estimation of jyu is piz = 1 E. k(X,.).
N
Definition 3 (Flow-based generative model Dinh et al. (014)). A flow-based generative model ex- plicitly learns the data distribution by trying to bijectively map it to a tractable density via invertible transformations. Formally, given a random variable z following tractable density, i.e., z ~ pe (z), a flow-based based model learns an invertible mapping fo such that the data sample x = fz(z) and the corresponding data distribution, Pr(x) = fzPo(z), where, fa = =Idel is the push-forward operator.
3 A SIMPLIFICATION STRATEGY FOR FLOW-BASED GENERATIVE MODELS
Flow-based generative model directly learn the mapping from a known distribution the target data distribution in the input space through likelihood maximization. During image generation, the fact that data may often lie on or near now-dimensional manifold usually not explicitly leveraged. If one could estimate this manifold perfectly, we could uniquely identify a distribution in the input space. But often, the manifold can only be empirically estimated. Our simplifying assumption here that with some structural or smoothness constraints on the distribution on the latent space, we should still be able to approximately identify the true distribution. VAES use this intuition to make the latent distribution consistent with standard normal distribution, and several contemporary works also propose estimating flow on latent spaces Kingma et al. (2016): Mahajan et al. (2020). We will follow this rationale and use an autoencoder here, benefiting from the reduced variance in the latent space, our approach will effectively learn the mapping from a known simple distribution to the empirical latent distribution.
3

---

Under review as a conference paper at ICLR 2021
Structure of the latent space. We will assume that suitable auto-encoder is either provided or be successfully trained from scratch for the data at hand. Given measurements from an input data space, M, we seek to learn mapping E M + L R with d m, where M C R t and the latent space. Additionally, we also learn the inverse of the mapping which is denoted by D L M, i.e., we require D o E id (identity). Further, of R For simplicity, we model it as unit hypersphere gd-1, This essentially ensures that L is (a) bounded (b) geodesically complete (c) geodesically convex space.
require L to be bounded subset
Nx..
Operator learning:
sample
enczuder
Kgk
decuder
N
latent
Generation:
latent keeeel
N
ssuplie
N x
price
sample
(N)
(M)
(e)
(4)
Figure 2: A description of generative procedure in our method. (a) schematic of auto-encoder (b) kernel representations of latent space and noise from prior (c) operator learning and inferencing solve pre-image and decode the generated latent representation to obtain final generation result data space
Remark 1. A bounded latent space with specified structure provides several benefits any such constraints. First, it guarantees that the Gram matrix constructed with a (positive-defil definitt)) kernel function bounded, which is required for valid empirical estimation of the mean embedding in Def. 2 Song et al. (2009). Moreover, the projection onto the hypersphere also helps ensure the cesttive-deiiniteess of kernels constructed using the latent representations, which discussed in 3.2 At this point, the requirement of geodesic completeness and convexity may not be apparent but these properties will be useful for sample generation.
without
3.1 USING TRANSFER OPERATORS TO ESTIMATE FLOW
If training the auto-encoder provides suitably well-structured latent space, desired, can now study how the density transfer operators can help transfer a tractable density to the density on the latent space, i.e., mimicking the mechanics within flow-based generative model.
Transfer operators. Let (Xth>o be a stationary and ergodic Markov process defined on X. Then, we can define the transition density function, (with time lag r), by
P[X++ A|X, 21
p- (ulxddy where a measurable set on X'
Let (2X) the space of probability densities. We wish to emulate the dynamical system of interest by instead utilizing well-known transfer operator, namely the (Ruelle) Farron-Frobenuus operator Mayer (1980), where the "transfer" terminology originates from statistical mechanics. Notice that the notion of in work is not literal it merely serves convenient way describe the transfer operators in the context of evolution of states, and is also analogous to how flow is considered explicitly continuous dynamical systems in Neural ODE (Chen et (018))
Definition 4 (Perron-Frobeniu operator Mayer (980)). The Ferron-Frobente (PF) operator P (2)) + (x' push-forwards or transfers a probability density pt E (2) given the lag T as (Pp.) (v) (x)))
Remark 2. If such operator can be efficiently estimated, probability density from a known distribution P; to the target distribution p++7 whose density is generally unknown. Since do not make any distributional assumptions the target distribution, a linear solution to such dynamics in the input space, in general, may not exist. Nevertheless, in spaces spanned by sufficiently large set of non-linear functions, specifically, RKHS, one
use it to transfer the tractable

---

Under review as a conference paper at ICLR 2021
potentially identify a linear operator that is equivalent to highly non-linear operator in the input space. In Klus et al. (2020), the authors showed two important results that define such operators in RKHS in terms of covariance and cross-covariance operators, namely kernel Perron-Frobening operator (kPF) Pk and embedded Perron-Frebenius operator (ePF) Pe-
Let X ~ Pt and Y ~ Pt+r be observations at time and f +++ Let H. and & be the RKHSs associ- ated with a certain kernel k. 6H and da be their respective feature mapping. The cross-covariance operator Cyx : H + 9 is defined as Cy = Ey (Ya(Y)) @ (x)))), and the covariance operator + H is defined analogously. If both densities of interest live in the same RKHS, that is, Pe E H. and Po+r E H. Pk can be used to push forward the density such that Pt+r = PkPt- However, in our setting, Pa is not directly applicable since the densities we are interested are ele- ments of T (X)). Therefore, the densities need to be first embedded into the RKHS and can then be transferred using the Pg in the embedded form, namely Epitr = (P& a E)P,.
Cxx
operator Klus et al. (020 Given Ps E (x) and Petr E L (2).. Lel μt = Ept and pt+t = Ept+7 be their respective kernel mean embedding. The kernel Ferron-Frobeniux (kPF) operator, denoted by Pg H + G, is defined by pt++ Pepu
Definition 5 (Embedded
Cyx(C'x + +Z)= - Mt (2), under the condition that (i) Cxx is injective (fi) fe E Range(Cxxx) (iit) E| = = H.
Notice that the above defined Pg essentially has the same form as the kernel conditional embedding in [Song et al. (2013). While the first two conditions in the above definition can be satisfied (see Theorem of Fukumizu et al. (2013)) the last condition requires 3f E H s.t. Vx, f(x) = F = rr. which highlights the importance of the kernel choice. With this operator, we can transfer pt to
Pt-rr in their embedded forms. The following proposition demonstrates the implication of using the
embedded Perron-Frobenius operator.
Proposition 1 Klus et al. (2020)) With the above notations, & o P = Pg E.
The commutativity in the proposition shows that the transferred kernel mean embedding of pt by the linear operator we constructed in RKHS is equivalent to the kernel mean embedding of transferred p. by a highly nonlinear operator in the input space.
Remark 3.
Since Pi+r is generally intractable, the operator can only be empirically estimated. Given samples IN ~ p and Y = [ ~ pit,- let TH and &e denote the feature maps of X X = {
and Y, respectively. The sample estimate of the embedded Perron-Frobenius operator is given by Pe = Cyx (Cxx + cf)- & (Gxx + _Ne)) Φ7 (3) where Gxx = D is the Gram matrix of 4H. To generate a sample of the transferred density which is approximately Petr using The
sample x" of Pt, we construct do(y") = PεOH(at) 2a(Gxx + =
distribution of the resulting samples has the following property,
Theorem 1 (Proof in A.Ip. Let Ht+7 be the kernel mean embedding of the true distribution Pttr.
The resulting empirical mean embedding pi+r = 2/ Σ (a())) satisfies Flüi++] = 1447
Theorem I simply implies that PEñi+t is an unbiased estimator of the kernel mean embedding of the true distribution the latent space. If the kernel is characteristic and the exact preimage exists, then de (PedH((") ~ Petr asymptotically, which concludes our main result.
3.2 KERNEL TRANSFER OPERATOR FOR SAMPLE GENERATION
We now present an algorithm for sample generation assuming that a pre-trained auto-encoder is available. The detailed algorithm is described in Figure 3 The idea is simple yet powerful: first generate n samples from simple distribution restricted to a hypersphere (which we model as a uniform distribution on Sm-1). Then, will construct the operator on RKHS described in the previous section using the sampled points and the latent representations of training samples. At inference time, we will use the operator to transfer new points sampled from the simple distribu- tion to the target feature map. Since it is not practically possible to compute the preimage directly from the dnmeite-dimensionall feature map, we left multiply with &e and use the geodesic interpola- tion (gl) module to construct an approximate preimage. Finally, we decodes the interpolated latent representation to the image space. A visual description is shown in Figure 2
5

---

Under review as a conference paper at ICLR 2021
Properties. If Kz is indeed invertible, the empirical mean
Interpolation (gf)
embedding of samples generated using the proposed al-
(a) Let the inputs be (1r C 8 and (wihitt
gorithm (assuming the preimage is exact) is equal the
Initialize
empirical mean embedding of the latent representations, indicating a match in distribution. Further, we use locally-
(c) For 2,- .7
(a) Set "//Eba
arccse(mm 11y).
(b) Update
weighted Fréchet mean on sphere to construct approxi- mate preimage of sample RKHS. Since closed-form solution for the weighted Fréchet mean does not exist on the sphere, we propose use a simple and efficient al- gorithm, namely the geodesic interpolation (gl) Salehian cet al. (2015), that uses the geodesic on hypersphere to
(msin 1)0) + 1 sin (M))
sin(0)
oulput.
Figure 4: The geodesic interpolation algorithmn
iteratively computes the weighted Fréchet mean Fréchet (9488) of top y latent representations (see Fig. 4. The algorithm has the following properties: (i) the geodesically completeness of the latent space guarantees that the geodesic interpolation well-defined (ii) the geodesic convexity of the la- tent space guarantees that the output of "gl" algorithm lies on the latent space (iii) the "gl" algorithm converges asymptotically to the Fréchet mean Salchian et (2015).
Choice of Kernels. In order to be able to linearize the dynamics between the prior and the target distribution, one must first identify a set of nonlinear basis functions such that the corresponding dynamics lies in its span. Known results in dynamical systems guarantee the existence of such linear operator, such as the Koopman operator Koopman tions. But rather than identifying certain modes that best characterize the dynamics Williams et al.) 2015); Brunton et al. (2016) we care most about minimizing the error of the transferred density, and whether the span of functions is rich/expressive enough and can be efficiently computed. There- fore, the choice of kernel is important since it directly determines the family of functions spanned by the operator. We empirically evaluate the effect of using several different kernels by a simple experiment on MNIST. The MNIST digits are first trained for 100 epochs using an autoencoder with latent space restricted to 52, then samples generated using procedure described in Figure 3 using the respective kernel function. Subplot (b) and (c) show the generated samples when using Radial Basis Function (RBF) kernel and arc-cosine kernel, respectively. Observe that the choice of kernel has clear influence on the posterior, but and a kernel with superior empirical behavior would be desirable.
KoopatorKooommn (133)), given an infinite set of basis func-
NTK. We use Neural Tangent Kernel (NTK) Jacot et al. (2018) as our embedding kernel due to the following properties: (a) NTK, in theory, corresponds to trained infinitely-wide neural net- work, and can be non-asymptotically approximated Arora et al. 2019). (b) For well-conditioned inputs (i.e., no duplicates) on the sphere, the positive-definitenss of NTK is proved in Jaacot et al.) 2018). Therefore, invertibility of Gxx is guaranteed if the sampling distribution is restricted
Generator
(a) Let x² be sample drawn from Uia-11. We first post-multiply with k(z",.) get P.k(z'..) &G (Gxx x + Nel) (b) क = =[](1,))]
(4), where, (a) Gxx = [k (z;, (2),
(b) Take inner product, (6--)o of the with P.k(z",,) .) to get
(0G, P.k(zz, ,)) $ t6 (Gxx + Nel) [k x²)] [k((nn))))] Kz(Kz NeT) Lv where Kz
(14, z;) +NeT) [")z
[k(1, 1.). NNN and N2 = (2,,2y)) E R RNNNN v= [k(zz, 2") E R.N.
(c) Notice that both KG and (Kz Nel) can pre-computed.
(d) Let (9o, P.k(2"..)). Let (4,) be latent representations 7 data samples with largest inner products in (in descending order).
(c) Return the generated sample X D (gl [-))_- (w, [..)) Here, D denotes the decoder function, = s//E, a., gl(...) is the geodesic interpolation algorithm (as presented in Fig.
Figure 3: Step by step details of sample generation algorithm.

---

Under review as a conference paper at ICLR 2021
Data points
Sampled points
Sampled peints (Arc-cosine)
Sampled points (NTK)
Figure 5: 10k samples from MNIST dataset (left to right) (a) projected on S shown in (0, Φ) using auto- encoder, and 10K generated samples using (b) RBF (c) arccos (d) NTK. Minimum regularization has been applied to all kernels to ensure invertibility. Color sampled points represents the class their 'nearest" point in the feature space.
hypersphere, which is true by our modeling choice. (c) Unlike other parametric kernels such as RBF kernels, NTK less sensitive to hyperparameters, as long as the number of units used is large enough Arora et al. (2019). The embedding of generated samples for MNIST using NTK is also shown in Observe that the sample distribution more closely resembles the original distribution.
Fig. 5
Nyström's approximation. A known bottleneck with kernel based methods is instantiating a kernel matrix whose size depends on the dataset size. Here, we use use the Nyström's method Nyström 1930 to obtain approximate kernel matrices. More specifically. a kernel matrix K E R can
be approximated by K = KS (ST KS)* 5TK, where S E Nxs is the sampling matrix. To the RecursiveRLS-Nystrin
minimize information loss, Musco (2017) to sample S based on the approximate ridge leverage scores. We refer the readers to the work by Musco & Musco (017)) for detailed algorithm. Notice that after the approximation, we only need to store KS E R x and S KS E Ro* matrices to obtain the Nyström approximation of K, bringing down the memory complexity from O(N2) O(Ns s²). When << N. the savings are significant.
algorithm proposed in Musco &
4 EXPERIMENTAL RESULTS
MNIST
CIFAR
CelcbA
We qualitatively evaluate the proposed gen- erative model on two types of applications, (a) datasets with a sufficient number of sam- ples to model the data distribution (b) datasets with fewer number of samples so that learn- ing the data distribution harder the first case,
WGAN
6.7
55.2
41.3
WGAN-GP
20.3
30.3
Vanilla VAE
13.7
111.0
52.1
Two-stage VAE
18.3
110.3
44.7
SRAE isopp
28.5
104.7
64.3
SRAEGi
26.1
110.9
50.9
SRAFRE rou(ours)
24.7
124.5
57.7
standard vision datasets, including MNIST, CIFARIO, and CelebA, where the num- ber of data samples is larger than the dimension. In order to handle the more challenging second case, we use a dataset of TI Magnetic Reso- nance (MR) images from a public brain imag- ing project called Alzheimer's 's Disease Neuro- Imaging Initiative (ADNI).
SRAE EyTrAPr sas (ourr)
15.0
123.3
39.9
SRAF AAyrK--r APY Nymetm (ours)
15.9
119.4
39.2
Comparative FID values. SRAE in- dicates an autoencoder with hyperspherical la- tent space and spectral regularization following Ghosh et al. (2020). Subscripts indicates the cor- responding sampling techniques on latent space. rand jiterpp: geodesic interpolation with uniform weights among 10 uniformly sampled latent rep- resentations. Glow: samples from Glow model trained on latent representations
Table 1:
The purpose of the first set of experiments to show that the proposed method yields competi- tive measures compared to other non-adversarial generative methods while enjoying the benefit of one step density estimation. In the second setting, our goal is to demonstrate that unlike traditional non-adversarial models, our proposed method pro- duces reasonable sample generation both quantitative and qualitatively.
Computer vision datasets: We evaluate the quality by calculating the Fréchet Inception Distance (FID) Heusel et al. (2017). All models share the same encoder-decoder architecture and trained for 100 epochs. Subscript 10k indicates the kPF is estimated using 10000 latent points, whereas Nyström indicates approximation using 4000 landmark points from all latent points. Comparative results
7

---

Under review as a conference paper at ICLR 2021
shown in Table 1 We compare results to vanilla VAE Kingma & Welling (2013) 2- Stage VAE Dai & Wipi (201)) and Glow + AE (denoted by SRAEctow in the table). FIDs of two best GAN methods are also reported from Dai & Wipt 2019) for comparison (note that the reported scores have been optimized by large-scale neural architecture search). We observe that for images with structured feature spaces, e.g., MNIST and CelebA, the proposed method matches other non-adversarial generative models. On the contrary, on images with less structured features such as CIFARIO, our model performs worse. We suspect that this is due to a non-smooth latent space, and we evaluate this by showing that visually similar images picked by our operators decodes to cittof-diitribution (blurred-out images (see Fig. (6). Here, we use a regularized AE Ghosh et al. (020) with a latent space restricted to the hypersphere (denoted by SRAE).
outperforms
8
8
8
8
8
4
4
4
4 9
4 6 0
4
6
9
6
6 O
0
0
0
0
7
7
7
Figure 6: Generations (in red box) and training samples corresponding to the top-5 latent represen- tations used in geodesic interpolation. can be observed that the samples with top kernel values indeed share high visual similarity.
Further, we present qualitative results on CelebA (in Fig. 7. where we compare our kPF based model with other density estimation techniques on the latent space. Observe that our model gener- ates comparative visual results with SRAEGtow. Hence, when sufficient number of samples are available, our method performs as good as the alternatives, which is beneficial given the efficient training stage. We present training time comparison and show that our proposed method is about x0x faster as in Fig. 9
Figure 7: Comparison of different sampling techniques using AE trained on CelebA 64x64. Left to
right: (1) Interpolation among 10 random latent points (2) samples of SRAE+Glow (3) samples of two-stage VAE (4) samples of KFF-flo using 10k latent points
Brain Imaging dataset: In this section, present results generating high-resolution (160 x 196 x 160) 3D brain images from ADNI consists of 183 samples from group AD (diagnosed as Alzheimer's Disease) and 291 samples from group CN (control normals). In this setting (where = 474 < d = 5017600), it is often hard to model the data distribution using either variational methods or flow-based methods due the high variance of the data or the memory-inefficienccy of the operations. However, benefiting from the regularized and linear nature of our kernel operator Willoughby (1979); AAroa et al.) (2020). we can still generate high-quality samples in such resolu- tion that are in-distribution. As before, we present the comparative results with respect to the VAE model. The generated samples presented in Fig. X clearly demonstrate that proposed method gen- erates sharper images. To evaluate whether these results are also scientifically meaningful (and not merely visually pleasing), we tested consistency between statistical group difference testing on the real images (groups were AD and CN) and the same testing performed on the generated samples. We performed a FWER corrected two-sample t t-test| Ashburner & Friston (2000) in a manner consistent with standard practice Ashburner & Friston (2000); Winkler et al. (2014). The results (see Fig () show that while there is a deterioration in regions identified to be affected by disease (and different
8

---

Under review as a conference paper at ICLR 2021
across groups), most of the statistically-signn significant regions from the test on the original images preserved in tests performed on the generated images.
Figure 8: Left. Top: data, generated samples of Middle: VAE samples, Bottom: kPF samples. Standard Glow failed to fit into system memory for image of this resolution. Right. Statistically significant regions Top: data, Bottom: samples shown in negative log p-value thresholded p < 0.01.
Take-home message: We conclude that due to the kernel approach and one-step formulation of the proposed model, it offers better memory and sample efficiency than other non-adversarial generative methods. This is useful in many tasks, e.g., medical image analysis, where the number of samples is usually much less than the dimension of data. Moreover, case datasets with larger number of samples, the proposed method still performs well (and sometimes better) compared to alternatives, with a sizably smaller resource footprint.
5 DISCUSSION
We have shown that when a dataset allows a low- dimensional structured latent space representation, kernel Perron-Frobenius operator can provide an efficient strat- egy simplify flow-based generative models transfer the density from a known simple distribution to the empir- ical data distribution. While the algorithmic simplifications proposed here can be variously useful, deriving the density of the posterior given mean embedding providing an exact preimage for the generated sample in RKHS remain unresolved this time. While the problem of deriving the density has been addressed in Schhuster et al. (2020), iden- tifying a pre-image can be hard and often ill-poded. We also note that our geodesic interpolation only converges Figure 9: Comparision of additional
Comparison of training time
3300
16k
rpstrem
741-8
672.4
mane dma
asymptotically to the weighted Fréchet mean which we use training time to other density estimation as the approximate preimage, and additional improvements model on the latent space.
on this front are possible.
6 CONCLUSION
In this paper, we show that with the assistance of recent developments in regularized autoencoder and neural kernels, linear kernel transfer operator can potentially be an efficient substitute flow-based generative models. Our proposed method shows comparable empirical results to other state-of-the art generative models on several computer vision datasets, while enjoying higher computational efficiency. The results brain imaging data also shed light on the potential application high- volume data generation, which are typically hard to model using existing methods.
REFERENCES
Guillaume Alain and Yoshua Bengio. What regularized auto-encoders learn from the data- gentratnng distribution. The Journal of Machine Learning Research, 15(1):3563-3593, 2014.
Hassan Arbabi. Introduction koopman operator theory of dynamical systems, 2018.

---

Under review as a conference paper at ICLR 2021
Nachman Aronszajn. Theory of reproducing kernels. Transactions of the American mathematical society, 68(3):337-44) 404, 1950.
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. In H. Wallach, H. Larochelle, A. Beygelz- imer, F. d'Alché-Buu Buc. E. Fox, and R. Garnett (eds.), Advances in Neural Information Process- ing Systems 32, pp. 8141-8150. Curran Associates, Inc., 2019. URL http://appers.nissccc/pppr with-an-iiiiniillyywdddeee-e
9025-on-exxat--compaaaie
Sanjeev Arora, Simon S. Du, Zhiyuan Li, Ruslan Salakhutdinov, Ruosong Wang, and Dingli Yu. Harnessing the power of infinitely wide deep nets on small-data tasks. In International Conference on Learning Reprexentations, 2020. URL. http::///wwweeeeeeee//////o/nnda
John Ashbumer and Karl J Friston. Voxel-based morphometry the methods. Neuroinage, 11(6):
805-821, 2000.
Omri Azencot, N. Benjamin Erichson, Vanessa Lin, and Michael W. Mahoney. Forecasting sequen- tial data using consistent koopman autoencoders. 2020. URL https:/ppooceedigg..mm..co/////p
2020hhash/Shaa3eee3333 1
Steven L. Brunton, Joshua L. Proctor, and J. Nathan Kutz. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proceedings of the Na- tional Academy of Sciences, 1 1133(15):9332-3937, 2016. ISSN 0027-8424. doi: 10. 107
1517384113, /
133/5//93//
Steven L Brunton, Bingni W Brunton, Joshua L Proctor, Eurika Kaiser, and J Nathan Kutz. Chaos as an intermittently forced linear system. Nature communications, 8(1):1-9, 2017.
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Sys- tems 31, pp. 6581- Curran Associates, Inc., 2018. URL http://papers.missccc/aapp 7892- haggl/mmemm.ooooaaaaaaaeeeeeenno inteerentiallequutioonsssss
ordinary differential equations.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint 1rXiv:1604.06/74 2016.
Bin Dai and David Wipf. Diagnosing and enhancing VAE models. In International Conference Learning Representations, 2019. URL https://ppaarwaaw...////oo//tia" c0xx3C3
Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components esti-
mation. arXiv preprint arXiv:1410.8516, 2014.
Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio. Density estimation using real nvp. In ternational Conference on Learning Representations, 2017. URL https://peenrvvie..ee//orumm idd=HkpbnH9lx
Maurice Fréchet. Les éléments aléatoires de nature quelconque dans un espace distancié. In Annales de I' l'institut Henri Poincaré, volume 10, pp. 215-310. 1948.
Kenji Fukumizu, Le Song, and Arthur Gretton. Kernel bayes' rule: Bayesian inference with positive definite kernels. The Journal of Machine Learning Research, 14(1):3753-3783 2013.
Partha Ghosh, Mehdi S. M. Sajjadi, Antonio Vergari, Michael Black, and Bernhard Scholkopf. From variational to deterministic autoencoders. In International Conference on Learning Representa- tions, 2020. URL http::/wppeneeeeeew...////mmmnns
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochre- iter. Gans trained by a two time-scale update rule converge to local nash equi- librium. In I. Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vish- wanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 2627.663. Curran Associates, Inc., 2017. URL http://paperr.mips.cc/appp 7240- -eall-agnn--ttt------rr wawained-b----tw-ttnneeee hww.:eeeeeeepeeeeeeeeoeeeeeeeeee local-to----lllalalllllal
10

---

Under review as a conference paper at ICLR 2021
Jörn-eerrik Jacobsen, Arood W.M. Smeulders, and Edouard Oyallon. i-revnet: Deep invert-
ible networks. In International Conference on Learning Representations, 2018. URL https:
Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Bisschii and R. Garnett (eds.), Advances in Neural Information Processing Sys- Tems 31, pp. 8571-8580. Curran Associates, Inc., 2018. URL http://paperrsnnps.cc/mme 8076 www.mngeenneeeeeeeoooegrnneeeeee wwwprrrreennenddggeeeeeeeee -aneeeeaaaaeeeeeaiizaiinionn in- in- netsrrssstttwwos.pp.......
Hycongju Kim, Hyconseung Lee, Woo Hyun Kang, Sung Jun Cheon, Byoung Jin Choi, and Nam Soo Kim. Wavenode: A continuous normalizing flow for speech synthesis. ICML Workshop on Invertible Neural Networks, Normalizing Flows, and Explicit Likelihood Models, 2020. URL
https://nveribleworrssop http://ttwwewwooooooo......eeeee wtt.thb...//cccppteeeee.. http://cpppee.ppppppppps//ii/./p
Sungwon Kim, Sang-Gil Lee, Jongyoon Song, Jachyeon Kim, and Sungroh Yoon. FloWaveNet :
A generative flow for raw audio. volume 97 Proceedings of Machine Learning Research, pp.
3370-3377 Long Beach, California, USA, 15-15 Jun 2019. PMLR. http://///c/r mir.
press/v97/kim19%
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint wrXiv:1412.6988 2014.
Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint 1r1iv:1312.6114, 2013.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. In Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 10215- Curran Associates, Inc., 2018. URL http://pppers.nisscccopper Ixl-
8224 glo4ggloww--oo wwwgggloo-gggeneee gewerative-reeeeeeeee
Durk Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling- Improved variational inference with inverse autoregressive flow. In D. D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 4743-4751. Curran Associates, Inc., 2016. URL lhttp://papers http://papers.iips.ccaaapa 65811 improved- -niproood--arriaiinnnall -itiuational-iffeeneeee.. with inverse autoregressive- flow.pdf
Stefan Klus, Ingmar Schuster, and Krikamol Muandet. Eigendecompositioons of transfer operators in reproducing kernel hilbert spaces. Journal of Nonlinear Science, 30(1):283-315, 2020.
Bernard O Koopman. Hamiltonian systems and transformation in hilbert space. Proceedings of the national academy of sciences of the united states of america, 17(5):315, 1931.
JT-Y Kwok and IW-H Tsang. The pre- me-image problem in kernel methods. IEEE transactions on neural networks, 15(6)1517-1525, 2004.
Sang-gil Lee, Sungwon Kim, and Sungroh Yoon. Nanoflow: Scalable normalizing flows with sub-
linear parameter complexity. arXiv preprini arXiv:2006.062800 2020.
Qianxiao Li, Felix Dietrich, Erik M. Bollt, and loannis G. Kevrekidis. Extended dynamic mode decomposition with dictionary learning: A data-driven adaptive spectral decomposition of the koopman operator. Chaos: An Interdisciplinary Journal of Nonlinear Science, 17110):003-11, 2017. doi: 10.1063/ 0000066114993666 14993844. URL httpp:///oo.orr// wttpo////..../////11.110.
Yunzhu Li, Hao He, Jiajun Wu, Dina Katabi, and Antonio Torralba. Learning compositional koop- operators for model-based control. In International Conference on Learning Representa- fions, 2020. URL httpp::/oppnneevee.eeeeffommmmmm
Bethany Lusch, J. Nathan Kutz, and Steven L. Brunton. Deep learning for universal linear embeddings of nonlinear dynamics. Nature Communicationss 9(1), 2018. doi: 10.1038/ s41467-018-07210-0 httpg////wooooog./.../..8/4///..

---

Under review as a conference paper at ICLR 2021
Shweta Mahajan, Iryna Gurevych, and Stefan Roth. Latent normalizing flows for many-to-man -o-m cross-domain mappings. In International Conference on Learning Representations, 2020. URL http:///wwccccottooo//oooddd/a/d
Dieter Mayer. The ruelle-araki transfer operator in classical statistical mechanics. 1980.
C. Miao, S. Liang, M. Chen, Ma, S. Wang, and Xiao. Flow-tts: non-autoregressive network for text to speech based on flow. In ICASSP 2020 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7209-7213, 2020.
Cameron Musco and Christopher Musco. Recursive sampling for the nystrom method. In Advances in Neural Information Processing Systems, pp.
2017.
Radford M. Neal. Priors for infinite networks. In Bayesian Learning for Neural Networks. Springer, URL https:://doi.or// http:://00..00070111110000000000
1994. doi: 10.1007/978-14461-007755
Roman Novak, Lechao Xiao, Jiri Hron, Jaehoon Lee, Alexander A. Alemi, Jascha Sohl-Dickstee and Samuel S. Schoenholz. Neural tangents: Fast and easy infinite neural networks python.
In International Conference on Learning Represenlutions, 2020. URL http:///ggttuubbooo///////////// neural-tangents
Evert J Nyström. Über die praktische auflösung von integralgleichungen mit anwendungen auf randwertaufgaben. Acta Mathematica, (1)::185-204, 1930.
Hesamoddin Salehian, Rudrasis Chakraborty, Edward Ofori, David Vaillancourt, and Baba C Ve- muri. An efficient recursive estimator of the fréchet mean on hypersphere with applications to medical image analysis. Mathematical Foundations of Computational Anatomy. 2015.
Ingmar Schuster, Mattes Mollenhauer, Stefan Klus, and Krikamol Muandet. Kernel conditional density operators. volume 108 of Proceedings of Machine Learning Research, pp. 993-1004, Online, 26-28 Aug 2020. PMLR. URL http://proceeedigg...ii press/v 108/schuster20a.htmll
Alex Smola, Arthur Gretton, Le Song, and Bernhard Schölkopf. A hilbert space embedding for distributions. In Marcus Hutter, Rocco A. Servedio, and Eiji Takimoto (eds.), Algorithmic Learn- ing Theory, pp. 13-31, Berlin, Heidelberg, 2007. Springer Berlin Heidelberg. ISBN 978-3-540- 75225-7.
Le Song, Jonathan Huang, Alex Smola, and Kenji Fukumizu. Hilbert space embeddings of condi- tional distributions with applications to dynamical systems. In Proceedings of the 26th Anmual International Conference on Machine Learning, ICML '09, pp. 961-968, New York, NY, USA, 2009. Association for Computing Machinery. ISBN 9781605585161. doi: 10. 10.1145/1553374. 1553497. URL
Le Song, Kenji Fukumizu, and Arthur Gretton. Kernel embeddings of conditional distributions: A unified kernel framework for nonparametric inference in graphical models. IEEE Signal Process- ing Magazine, 30(4):98-111, 2013.
Matthew O. Williams, Ioannis G. Kevrekidis, and Clarence W. Rowley. A data-driven approx- imation of the koopman operator: Extending dynamic mode decomposition. Journal of Non- linear Science, 25(6), 2015. doi: 10.10077:00333--01--92558. URL httis://doi.org//00 1007/
s000332-015-925-5
Ralph A. Willoughby. Solutions of ill-posed problems (a. n. tikhonov and v. y. arsenin). SIAM Review, 21(2):266-26) 267,, 1979. doi: 10.1137/1021044, http://t////ooo.rr////// 11.pp//./.....1111111110440
Anderson M Winkler, Gerard R Ridgway, Matthew A Webster, Stephen M Smith, and Thomas Nichols, Permutation inference for the general linear model. Neuroimage, 92:381-397, 2014.
12

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 PROOF FOR THEOREM 1
The proof follows easily from the fact that Pe is linear and independent to [0:/i=m
Elp E T
Peo(x.))
E|Pe] E 7
Σ
Pepe = p14T o(x,)]
A.2 ABLATION STUDIES
We present ablation studies on MNIST (Fig. Top. showing importance and different components of our proposed method. Our proposed framework in gen-
4 / 4
4
9
7
7
6
eral consists of (a) learning latent space (b) estimating transfer operator using kPF. We replace the first part in two ways (i) remove AE and directly transfer the distribution from image space to the known distribu- tion (ii) replace AE with known invertible basis rep- resentation, e.g., mapping onto Fourier basis. In both these cases, we observe that the generation quality de- teriorates. Notice that such degradation can be justi- fied due to the lack of smoothness on the input space.
C 0
+
9
2
3
3
7
0 5
4
3
9
7
3
1
4
0
6
0
2
9
2
Figure 10: Top to bottom: no latent space
with kPF, (2) interpolation in Fourier space,
Next, we fixed the AE and replace the kPF operator with a normalizing flow model Kingma & Dhariwal (2018) as transfer operator. Observe that the genera- tion quality are comparable between the two methods.
(3) Glow as transfer operator, (4) kPF as
transfer operator
We further show the advantage of using a regularized AE regular one. The regularization scheme applied is derived Crosh et al. (200)), where we
MNIST
CIFAR
CelebA
SAENTR zol SRAENTRAPF io
22.3
133.6
41.2
15.0
123.3
39.9
apply spectral normalization on all encoder-decoder transformations. Regularization has been shown to lead to smoother latent spaces/Alaie & Bengio (2014), and therefore is critical in our interpolation- based sampling approach. The effect of such regulariza- tion can be seen in Table 2 where it leads higher FIDs in all three datasets using smoother latent space.
Table 2: vanilla vs. regularized AE.
Take-home message: summarize, the presented ablation study demonstrates the importance the latent space as well as the richness of our one-step estimated operator. Moreover, the boundedness and smoothness of the latent space crucial as can be justified by replacing AE with a Fourier basis or by using regularized AE.
A.3 ADNI DATASET AND BRAIN IMAGE PROCESSING
Data used in preparation of this article were obtained from the Alzheimer's Disease Neuroimaging Initiative (ADNI) database (odni.loni. isc.e. As such, the investigators within the ADNI con- tributed to the design and implementation of ADNI and/or provided data but did not participate in analysis or writing of this report. complete listing ADNI investigators can be found at:
The TI MR brain data we use consists of images from 184 subjects diagnosed with Alzheimers's disease and 292 control normal subjects. Images were first coregisted to a MNI template and seg- mented to keep only the white matter and grey matter. Then, all images were resliced and resized to 160 x 196 x 160 and rescaled to the range of [-1, 1]. Voxel-basee morphometry (VBM) was used to obtain the valuu map of data and generated images.
13

---

Under review as a conference paper at ICLR 2021
A.4
FID SCORES ON CELEBA DATASET USING RAE DECODER
WGAN
2-Stage VAE
SAE&PF 46.1
SRAE&PF
FID 56.5
45.25
44.6
Table 3: FID scores using the architecture and training protocol from Ghosh et al. 2020
A.5
IMAGE QUALITY EVALUATION
8
2
0
5
9
3
7
6
5
4
3
4
5 2
2
5
9
7
4 7 5 7 4 3 5 9 8 0 2
C3
3
r
3
VAE
5
3
R 0
2
1
0
2
0
3
7
5
>
1
9
R
6
G
3
4 2 6
8
1
O
2 4
1
5
3
9
0
5
8
7
9
3 6 1 0 0 1 O < 0 6 2
5
6
6
0
2S- AS-VAB
5
1
3
8
2
6
6
5
9
8
7
1
6
9 5
8
6
8
9
8 0 6
1
4
3
C 0
O
0
6
1
9
2
3
9
3
7 9 2 6
6
SAE,PF
1
1
0
7
6
9
7
4 4
0
3
6
f
A 1 2 8 O 4
2
4
9
3 S 6 7 4
2 7
9
0
4
5 0 5 4
4
4
+
7
4
1 1
5
3
2 5 4 6
SRAE,PF
44543447
55
1 70957 45
608
6
5
8 2
Figure 11: Randomly picked samples from (1) VAE (2) 2-Stage VAE (3) SAE (4)
SRAENTRAPF rok- All models trained with the architecture in Ghosh al. (2020) without tuning for hyperparamters
A.6 CHOICE O
In the inference stage, proposed method finds the approximate preimage of the transferred kernel embeddings by interpolating among the top y latent representations of the training samples weighted by their kernel values. The choice of therefore has implications Figure [2 we can observe that, in general, FID worsens as y increases. This observation aligns with our intuition of preserving only the local similarities in kernel embeddings, and similar idea has
the generation quality. From
14

---

Under review as a conference paper at ICLR 2021
However, significantly decreasing leads to the undesirable result where the generator merely generates the training samples (in the extreme case where 7 1, generated samples will just be reconstructions of training samples). Therefore, in our experiments, we choose y = 10 to achieve a balance between generation quality and the distance to training samples.
been used in the literature Kwok & Tsang 2004).
250
mnist
cfar
celeba64
200
150
RD
5000
Figure 12: FID versus choice of
A.7 EXPERIMENTAL SPECIFICATIONS
For all experiments in Table I except the GAN experiments, we adapted custom ResNet imple- mentation where each block has residual connection
BatchNorm - Swish >> 3x3 Conv
BatchNorm Swish 3x3 Conv ->
The detailed architecture is given by the following table
MNIST
CIFAR-10
CelebA
5x5 conv, stride
ResBlock32
ResBlocka2 x
ResBlock32 x
ResBlockes 2
ResBlocke4 2
ResBlocke4
Encoder
ResBlock12s x 2
ResBlock12s x 2
ResBlock12s x 2
ResBlock256 x 2
ResBlock256 X 2
ResBlock256 2
ResBlocks x 2
ResBlock256 2
ResBlock256 2
ResBlocks12 2
ResBlock 128 x 2
ResBlock x 2
ResBlock25a x 2
ResBlockga
ResBlockga
ResBlock
Decoder
ResBlockga
2
ResBlockgg 2
ResBlockgs x 2
ResBlock32 x
5x5 conv, Stride
Table 4: Detailed network architecture for experiments in Table [ Subscript denotes the number of input channels. Upsampling and downsampling are performed using strided convolutions.
We train each model for 100 epochs on every dataset using Adam optimizer Kingma & Ba (014)) with 3 = (0.9, 0.999) and batch size of 512. The learning rate starts from le-3 and is halved every 30 epochs. We used implemetation of Novak et al. (2020) to estimate the NTK with four fully- connected layers, each with 10000 units, and Eir activation. VAE and 2-Stage VAE are trained using learnable-y in Dai & Wipf (2019). For experiments in A.4 and A.5 procedure and architecture in Ghosh et al. (2020)
adopted the training
15


Reviews:
[{"review_id": "Zlsnh_iZ7z", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**** Summary ****\n\nThe authors build a generator that builds on top of the latent space of a “well-trained auto-encoder”. The generator consists of several steps: 1) sampling an latent element from the spherical latent space, 2) using a kernel Perron-Frobenius operator to embed the sampled latent element 3) selecting latent representations of real samples based on the indices of the highest values of the newly calculated embedding 4) calculating the geodesic interpolation of these latent representations 5) using the decoder of the autoencoder to generate new samples from the latent element resulting from geodesic interpolation. The authors describe the related work, the methods they propose and present experimental results on 4 datasets. \n\nTo summarise, I believe that the contributions of the authors are the following:\n- Proposing a generator that relies on a kernel Perron-Frobenius operator (relevant: medium)\n- Comparing variants of their generator to state of the art generative models on four datasets (relevance: medium-low)\n\n**** Positive points of the paper: ****\n\nRelation to prior work and clarity:\nThe authors attempt to describe extensively the theory behind their work. However, as we will see below, the prior work section could be clearer. Additionally, the link between concepts introduced in prior work and presented in the newly developed algorithm should be more straightforward. See below for detailed comments on improvable points.\n\nNovelty:\nThe method consists of a novel combination between an existing generative model and an existing kernel transfer operator. I believe that the combination is new however it is not thoroughly justified. \n\nTheoretical setting:\nThe authors introduce in prior work several of the necessary definitions for the following part. However, some required hypotheses do not always seem to be carefully verified. See below for detailed comments on improvable points.\n\nExperimental setting:\nThe authors compare their model to several state-of-the-art generative models and show some results on one dataset using different types of kernels. However, details from the experiments are missing to be able to evaluate it thoroughly and the experiments are not reproducible. See below for detailed comments on improvable points. \n\n**** Improvable points of the paper ****\n\nRelation to prior work and clarity:\nIn general, some sentences of the paper would benefit from being rewritten as they are too convoluted in my opinion. For example, p3, Section 3, first paragraph: “We will follow this rational… variance is decreased”; p6, second paragraph, “to push a uniform … wrapped on S2”. Other sentences are not theoretically justified: p3, last paragraph: “Moreover, the projection … that we will use later”; p.4 Remark 2, “But notice that performing these …. is more sensible”.\n\nAdditionally, it seems that the prior work section starts with the introduction and ends at Section 3.2 (excluded). The structure of the paper does not allow a simple identification of the prior work from the method as Section 3 contains prior work and new developments. \n\nAlso, it seems that most of the elements of the prior work sections are coming from Fukumizu et al. (2013) and Klus et al. (2020). However, it is not always possible to understand the content of the paper without having to read the definitions/properties in the respective papers. For example, a) Fukumizu et al does not seem to mention the name of the Perron-Frobenius operator, b) in Definition 5 it is not clear why there is a P_eps and a P_k, especially when the variable eps is not defined before, c) Definition 5, it is not clear whether P_kg should belong or not to an RKHS for the Definition to be valid (as mentioned in Klus et al.) d) p4, last paragraph, the equation giving P_kg and P_eps are not introduced properly (especially the need for epsilon) and we need to look at the referred paper to understand where it is coming from e) p5, second paragraph, G_xx is not defined (I understand it is the gram matrix but it should be written), the same holds for N f) p5, second paragraph, equation 3 should be better explained and a reference to Klus et al. and Fukumizu should be made. g) p5, second paragraph, we don’t understand what mu_t is. h) Proposition 1 is not proven (and is written as a definition for reversibility in Klus et al) i) p5, in Notations, \\mathcal{L} is not introduced. j) p5, Section 3.2, D (decoder, but should be introduced) and top \\lambda are not clear. k) p5, Section 3.2, it is not clear what s is and it has a different typesetting in the bullet points 1 and 2. l) The transpose symbol is not always the same along the paper and can be confused with the iteration number in Figure 4. m) it is not clear where Proposition 1 is needed in practice. \n\nTheoretical setting:\nSome concepts are not well defined. For example:\n- The authors always talk about a well-behaved VAE. However, the authors do not define what they mean by well-behaved. In addition, looking at Table 1, it seems that the proposed model is not systematically the best performing when the VAE is well behaved (the FID for vanilla VAEs are often better than the proposed model). \n- I am not sure what “true solution” means p6, end of Properties. It would be useful if the authors could give the intuition to make the reader understand what the true solution means and the need of the geodesic interpolation. \n- Some hypotheses are not verified, for example, a) is K_z invertible in practice (see Properties p6), b) is the experimental covariance is a good approximation for the true covariance.\n\nIn general, the algorithm seems to be explained twice with different degrees of precision at the bottom of p5 and in Figure 3 (which should be referred to in the text). The paper would benefit from merging both explanations. \n\nIt is not clear to me why the new sampling strategy would follow better the distribution of real samples. I think this should be better explained. \n\nExperimental setting:\nThe experiments are described in a way that is not reproducible. We do not know which hyperparameters are chosen (for the AE and for the new development) and how they were chosen. We also do not understand how the 4000 landmark points are selected. It is not clear what SRAE_rand_interp is. I do not understand how Figure 5 is done, is it 2D projection of the generated samples? If yes, with which dimension reduction method? (“wrapped on S^2” a bit above the figure is not clear). \n\nA better ablation study could be done:\n- In the Supplementary, the authors say that they use a regularised AE (with spectral normalisation) for the SRAE. However, the results of a regularised AE without the development of the authors are not shown.\n- The authors do not justify theoretically the need for selecting the top \\gamma samples. Therefore, it would be interesting to vary \\gamma and observe differences in the results. \n\nIn general, the results seem rather poor. The new method outperforms vanilla VAE or basic SRAE only in one of the three datasets for which we have quantitative results. The generative adversarial models outperform systematically the proposed model.\n\n\n**** Typos: ****\n“Which is benefit, given the efficient” -> beneficial, efficiency\nGrammian -> Gramian or Gram\n\n*****************\nRECOMMANDATIONS:\nBecause of the points enumerated above, I recommend to reject the paper in its current form. I would increase the grade if a) the paper is rewritten in order to be more structured and clear b) the experiment section is better described and more thorough (ablation study) c) some answers are brought to the theoretical concerns. \n\n\n*****************\nAFTER REBUTTAL\nI read the rebuttal of the authors and the other reviews. I will increase the grade to a 5. \n\nThe reasons are that I think that the authors improved significantly the paper with this revision. However, I believe that the paper would benefit from experiments on a larger number of datasets, in order to better understand on which type of datasets their method shows better performance. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "V5DAy1HrfQ", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method for a deep generative model utilizing a trained autoencoder in conjunction with kernel conditional embeddings. The authors validate their model experimentally on CelebA and a brain imaging dataset.\n\nOverview : I found this paper very confusing with crucial mathematical definitions and exposition often inconsistent or imprecise. Outside of this the general exposition in many ways very strange and confusing. Finally the lack of experimental code makes this paper a very clear reject in my opinion. The rest of the review will list some concrete examples of the issues I mentioned.\n\nMathematical imprecision:\n- The Phi operator in this paper is problematic. I'm guessing this is meant to be the kernel feature map, but in Def 1 it is introduced as an arbitrary element of the RKHS. In Def 5 Phi_H and Phi_G are initially introduced via the phrase \"where an element is denoted by Phi_G\" seeming to imply that is again simply an example element of their respective RKHSs. Later in this definition the well-known (kernel) cross covariance operator \"C_YX = E_YX [ Phi_G (y) \\otimes Phi_H (x)]\" which I am assuming implies that it is the kernel feature map, since otherwise this quantity is simply a scalar. Finally in the description of their main algorithm step 1 the refer to Phi_G^T which is highly problematic since Phi_G is presumably well defined here, meaning it is the kernel feature map which is not a linear operator (the whole point of kernel methods being that this operator is nonlinear), thus its \"transpose\" isn't something that makes sense. The authors give a vector which is equal to the expression in which Phi_G appears, this however depends on a \\bold{1} vector defined as \" are the top lambda (a hyper-parameter) latent representations\" which is not explained further. Presumably this is referring to approximate inverse kernel feature map (which has seen some research), but this needs to be further explained, or at the very least a citation is needed. \n\nGeneral Exposition:\n-I am not sure why the approach to the paper is being framed in a Markov setting, when it seems like the authors really only need kernel conditional embeddings which have the exact same form as the Perron Frobenius operator the authors use. The authors even cite the kernel conditional embedding paper Song et al. 2009. Things like (1) which only serve to complicate the exposition unnecessarily considering there is no Markov process type of structure in the main algorithm.\n\n-p. 2 Rationale: \"For instance, it is known that given infinite-dimensional observables of the input measurements, there exists a linear transfer operator that perfectly pushes one probability density to another. Of course, this is not practically beneficial because explicitly constructing such infinite-dimensional observables for the data could be intractable. Nonetheless, results in control theory demonstrate that the idea can still be effective in specific cases, using approximations with either spectral analysis of large but finite number of observable functions \" This sort of thing needs to be decompressed, I don't know what is meant by an \"observable\" I assume it has something to do with control theory or Markov process theory. This seems to niche for me for the deep learning crowd.\n\n-The Riemannian geometry in the paper needs to be explained more for the average deep learning researcher, e.g. its not clear why the \"geodesic interpolation\" is doing at or why its algorithm makes sense.\n\n-The main algorithm needs to be explained with more details as to _why_ we are doing the steps.", "extra_scores": {}, "extra_text": {}}, {"review_id": "IykCvK389xu", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##### Post-rebuttal update\n\nDear authors, Thanks for the response. I strongly encourage you to revise the paper using languages other than flow-based generative models.\nFrom the rebuttal I could not see whether you understand my point or not: The conditional mean embedding operator defines an **integration** instead of an invertible **transformation**, which differentiate itself from any flow-based model (at least those you referenced).\nAs for now, I cannot recommend for acceptance.\n\n--------------\n\nThe paper proposes a different kind of generative model that is composed of autoencoder in the bottom, and a standard distribution p(z), and a conditional kernel mean embedding defined by a collection of sample pairs (z_i, l_i). The distribution of the latent codes p(l) is modeled by the kernel sum rule that corresponds to the marginalization \\int p(l|z)p(z) dz (ultimately defined by the collection of sample pairs). \n\n##### Originality & Significance\nThe proposal of modeling the latent distribution of an autoencoder in a nonparametric way (using kernel mean embeddings) is original to my best knowledge. To do this, you need to solve the pre-image problem (we can get the mean embedding of p(l) through kernel operations but it is generally difficult to map it back to samples). This paper adopts a previous approach (geodesic interpolation, Salehian et al. 2015), which I'm not aware of but appears to be theoretically sound: \"the gI algorithm converges asymptotically to the true solution.\"\n\n##### Clarity\nThe clarity is low. The title (and introduction) is very misleading as it says a lot about flow-based generative model. However, this paper is really about modeling the latent distribution of an autoencoder using conditional mean embeddings (what the paper calls embedded Perron-Frobenius operator).. In every aspect I find it more close to a latent-variable model (there is a latent prior over the z space). And the distribution in the L space is the result of the kernel sum rule that corresponds  to the integral \\int p(l|z)p(z) dz. Why do the authors prefer to call it a flow?\n\n##### Others\n\n* One strength of this paper is that the work is very careful in choosing appropriate kernels, the importance of which is often ignored in works that deploy kernel methods. Figure 5 clearly shows the benefits of a good kernel (NTK) for this task. The result also shows that Nystrom methods seem to work well with NTK \n\n* Definition 1: the RKHS should be the completion of the span.\n\n* I don't understand the method used to generate figure 6 (1), what is \"interpolation among 10 random latent points\"? Why is this a sensible baseline?\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "6u1O1HKDf5A", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper starts with an autoencoder trained on vision data. Autoencoders aren't generative models, strictly, so in order to make it so, they leverage a simple linear transformation on over RKHSs. The kernel they use is the NTK. In order to generate, the construct a reduced sample version of the Kernel using Nystrom's approximation, then use a geodesic interpolation algorithm to inverse map the transferred sample from the prior in the RKHS space back to the latent space for use in the decoder.\n\nThe method appears to work, which in itself is very interesting as this means one should be able to construct generative models (a la VAEs) starting only from a simple autoencoder. The results aren't that impressive compared to the baselines compared, but surprisingly they don't compare to any flow-based algorithms, such as Normalizing Flows or related methods like Hamiltonian Variational Inference. VAEs using NICE or Real-NVP to transfer the latent space from a autoencoder would be good baselines.\n\nIt would be good to have clearer references for the \"two-stage VAE\" in the experimental results section: I was unable to easily find what this model is from this draft.\n\nI found the geodesic interpolation part a bit difficult to follow and not clearly motivated. Could you go into more detail on its component parts and why they were included (e.g., what appears to be the importance sampling part)? Linking this more clearly to the \"properties\" section would help I think.\n\nWhat about the effect of the pretrained AE? Is reconstruction a good measure for the fitness of an AE for this or is the regularization important? How does one know one has a good autoencoder for this procedure? Could you provide results comparing metrics on the autoencoder (e.g., reconstruction loss, L2 or L1 on the parameters or latent space) vs NLL using the kernel method?\n\nSome other comments:\nSection 3.2 seems to have a few errors:\non 1. k -> k_z\non 2. is lambda supposed to be gamma?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All four reviewers were against accepting the paper. A major point shared by everyone was lack of clarity: this included its overall writing, its discussion toward prior work, and imprecise math to explain the ideas. The paper did improve quite a bit over its revisions. Whether this clarified all of the reviewers' understanding of the paper remains unclear. The work may ultimately need another cycle of reviews to assess its quality.

Another shared point are a number of recommended ablations in the experiments, as well as going through more comprehensively in the set of studied datasets (R3), effect of AE choices (R2), and alternatives to the geodesic (R1, R2).
Decision: Reject