Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: **** Summary ****

The authors build a generator that builds on top of the latent space of a “well-trained auto-encoder”. The generator consists of several steps: 1) sampling an latent element from the spherical latent space, 2) using a kernel Perron-Frobenius operator to embed the sampled latent element 3) selecting latent representations of real samples based on the indices of the highest values of the newly calculated embedding 4) calculating the geodesic interpolation of these latent representations 5) using the decoder of the autoencoder to generate new samples from the latent element resulting from geodesic interpolation. The authors describe the related work, the methods they propose and present experimental results on 4 datasets. 

To summarise, I believe that the contributions of the authors are the following:
- Proposing a generator that relies on a kernel Perron-Frobenius operator (relevant: medium)
- Comparing variants of their generator to state of the art generative models on four datasets (relevance: medium-low)

**** Positive points of the paper: ****

Relation to prior work and clarity:
The authors attempt to describe extensively the theory behind their work. However, as we will see below, the prior work section could be clearer. Additionally, the link between concepts introduced in prior work and presented in the newly developed algorithm should be more straightforward. See below for detailed comments on improvable points.

Novelty:
The method consists of a novel combination between an existing generative model and an existing kernel transfer operator. I believe that the combination is new however it is not thoroughly justified. 

Theoretical setting:
The authors introduce in prior work several of the necessary definitions for the following part. However, some required hypotheses do not always seem to be carefully verified. See below for detailed comments on improvable points.

Experimental setting:
The authors compare their model to several state-of-the-art generative models and show some results on one dataset using different types of kernels. However, details from the experiments are missing to be able to evaluate it thoroughly and the experiments are not reproducible. See below for detailed comments on improvable points. 

**** Improvable points of the paper ****

Relation to prior work and clarity:
In general, some sentences of the paper would benefit from being rewritten as they are too convoluted in my opinion. For example, p3, Section 3, first paragraph: “We will follow this rational… variance is decreased”; p6, second paragraph, “to push a uniform … wrapped on S2”. Other sentences are not theoretically justified: p3, last paragraph: “Moreover, the projection … that we will use later”; p.4 Remark 2, “But notice that performing these …. is more sensible”.

Additionally, it seems that the prior work section starts with the introduction and ends at Section 3.2 (excluded). The structure of the paper does not allow a simple identification of the prior work from the method as Section 3 contains prior work and new developments. 

Also, it seems that most of the elements of the prior work sections are coming from Fukumizu et al. (2013) and Klus et al. (2020). However, it is not always possible to understand the content of the paper without having to read the definitions/properties in the respective papers. For example, a) Fukumizu et al does not seem to mention the name of the Perron-Frobenius operator, b) in Definition 5 it is not clear why there is a P_eps and a P_k, especially when the variable eps is not defined before, c) Definition 5, it is not clear whether P_kg should belong or not to an RKHS for the Definition to be valid (as mentioned in Klus et al.) d) p4, last paragraph, the equation giving P_kg and P_eps are not introduced properly (especially the need for epsilon) and we need to look at the referred paper to understand where it is coming from e) p5, second paragraph, G_xx is not defined (I understand it is the gram matrix but it should be written), the same holds for N f) p5, second paragraph, equation 3 should be better explained and a reference to Klus et al. and Fukumizu should be made. g) p5, second paragraph, we don’t understand what mu_t is. h) Proposition 1 is not proven (and is written as a definition for reversibility in Klus et al) i) p5, in Notations, \mathcal{L} is not introduced. j) p5, Section 3.2, D (decoder, but should be introduced) and top \lambda are not clear. k) p5, Section 3.2, it is not clear what s is and it has a different typesetting in the bullet points 1 and 2. l) The transpose symbol is not always the same along the paper and can be confused with the iteration number in Figure 4. m) it is not clear where Proposition 1 is needed in practice. 

Theoretical setting:
Some concepts are not well defined. For example:
- The authors always talk about a well-behaved VAE. However, the authors do not define what they mean by well-behaved. In addition, looking at Table 1, it seems that the proposed model is not systematically the best performing when the VAE is well behaved (the FID for vanilla VAEs are often better than the proposed model). 
- I am not sure what “true solution” means p6, end of Properties. It would be useful if the authors could give the intuition to make the reader understand what the true solution means and the need of the geodesic interpolation. 
- Some hypotheses are not verified, for example, a) is K_z invertible in practice (see Properties p6), b) is the experimental covariance is a good approximation for the true covariance.

In general, the algorithm seems to be explained twice with different degrees of precision at the bottom of p5 and in Figure 3 (which should be referred to in the text). The paper would benefit from merging both explanations. 

It is not clear to me why the new sampling strategy would follow better the distribution of real samples. I think this should be better explained. 

Experimental setting:
The experiments are described in a way that is not reproducible. We do not know which hyperparameters are chosen (for the AE and for the new development) and how they were chosen. We also do not understand how the 4000 landmark points are selected. It is not clear what SRAE_rand_interp is. I do not understand how Figure 5 is done, is it 2D projection of the generated samples? If yes, with which dimension reduction method? (“wrapped on S^2” a bit above the figure is not clear). 

A better ablation study could be done:
- In the Supplementary, the authors say that they use a regularised AE (with spectral normalisation) for the SRAE. However, the results of a regularised AE without the development of the authors are not shown.
- The authors do not justify theoretically the need for selecting the top \gamma samples. Therefore, it would be interesting to vary \gamma and observe differences in the results. 

In general, the results seem rather poor. The new method outperforms vanilla VAE or basic SRAE only in one of the three datasets for which we have quantitative results. The generative adversarial models outperform systematically the proposed model.


**** Typos: ****
“Which is benefit, given the efficient” -> beneficial, efficiency
Grammian -> Gramian or Gram

*****************
RECOMMANDATIONS:
Because of the points enumerated above, I recommend to reject the paper in its current form. I would increase the grade if a) the paper is rewritten in order to be more structured and clear b) the experiment section is better described and more thorough (ablation study) c) some answers are brought to the theoretical concerns. 


*****************
AFTER REBUTTAL
I read the rebuttal of the authors and the other reviews. I will increase the grade to a 5. 

The reasons are that I think that the authors improved significantly the paper with this revision. However, I believe that the paper would benefit from experiments on a larger number of datasets, in order to better understand on which type of datasets their method shows better performance. 

review_title: Interesting concept but lack of clarity in prior work and explanation of the model + non-reproducible experiments

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper starts with an autoencoder trained on vision data. Autoencoders aren't generative models, strictly, so in order to make it so, they leverage a simple linear transformation on over RKHSs. The kernel they use is the NTK. In order to generate, the construct a reduced sample version of the Kernel using Nystrom's approximation, then use a geodesic interpolation algorithm to inverse map the transferred sample from the prior in the RKHS space back to the latent space for use in the decoder.

The method appears to work, which in itself is very interesting as this means one should be able to construct generative models (a la VAEs) starting only from a simple autoencoder. The results aren't that impressive compared to the baselines compared, but surprisingly they don't compare to any flow-based algorithms, such as Normalizing Flows or related methods like Hamiltonian Variational Inference. VAEs using NICE or Real-NVP to transfer the latent space from a autoencoder would be good baselines.

It would be good to have clearer references for the "two-stage VAE" in the experimental results section: I was unable to easily find what this model is from this draft.

I found the geodesic interpolation part a bit difficult to follow and not clearly motivated. Could you go into more detail on its component parts and why they were included (e.g., what appears to be the importance sampling part)? Linking this more clearly to the "properties" section would help I think.

What about the effect of the pretrained AE? Is reconstruction a good measure for the fitness of an AE for this or is the regularization important? How does one know one has a good autoencoder for this procedure? Could you provide results comparing metrics on the autoencoder (e.g., reconstruction loss, L2 or L1 on the parameters or latent space) vs NLL using the kernel method?

Some other comments:
Section 3.2 seems to have a few errors:
on 1. k -> k_z
on 2. is lambda supposed to be gamma?
review_title: Review for Can Kernel Transfer Operators Help Flow based Generative Models?

final_score: rating: 5: Marginally below acceptance threshold
review_content: ##### Post-rebuttal update

Dear authors, Thanks for the response. I strongly encourage you to revise the paper using languages other than flow-based generative models.
From the rebuttal I could not see whether you understand my point or not: The conditional mean embedding operator defines an **integration** instead of an invertible **transformation**, which differentiate itself from any flow-based model (at least those you referenced).
As for now, I cannot recommend for acceptance.

--------------

The paper proposes a different kind of generative model that is composed of autoencoder in the bottom, and a standard distribution p(z), and a conditional kernel mean embedding defined by a collection of sample pairs (z_i, l_i). The distribution of the latent codes p(l) is modeled by the kernel sum rule that corresponds to the marginalization \int p(l|z)p(z) dz (ultimately defined by the collection of sample pairs). 

##### Originality & Significance
The proposal of modeling the latent distribution of an autoencoder in a nonparametric way (using kernel mean embeddings) is original to my best knowledge. To do this, you need to solve the pre-image problem (we can get the mean embedding of p(l) through kernel operations but it is generally difficult to map it back to samples). This paper adopts a previous approach (geodesic interpolation, Salehian et al. 2015), which I'm not aware of but appears to be theoretically sound: "the gI algorithm converges asymptotically to the true solution."

##### Clarity
The clarity is low. The title (and introduction) is very misleading as it says a lot about flow-based generative model. However, this paper is really about modeling the latent distribution of an autoencoder using conditional mean embeddings (what the paper calls embedded Perron-Frobenius operator).. In every aspect I find it more close to a latent-variable model (there is a latent prior over the z space). And the distribution in the L space is the result of the kernel sum rule that corresponds  to the integral \int p(l|z)p(z) dz. Why do the authors prefer to call it a flow?

##### Others

* One strength of this paper is that the work is very careful in choosing appropriate kernels, the importance of which is often ignored in works that deploy kernel methods. Figure 5 clearly shows the benefits of a good kernel (NTK) for this task. The result also shows that Nystrom methods seem to work well with NTK 

* Definition 1: the RKHS should be the completion of the span.

* I don't understand the method used to generate figure 6 (1), what is "interpolation among 10 random latent points"? Why is this a sensible baseline?



review_title: Review

final_score: rating: 2: Strong rejection
review_content: This paper proposes a method for a deep generative model utilizing a trained autoencoder in conjunction with kernel conditional embeddings. The authors validate their model experimentally on CelebA and a brain imaging dataset.

Overview : I found this paper very confusing with crucial mathematical definitions and exposition often inconsistent or imprecise. Outside of this the general exposition in many ways very strange and confusing. Finally the lack of experimental code makes this paper a very clear reject in my opinion. The rest of the review will list some concrete examples of the issues I mentioned.

Mathematical imprecision:
- The Phi operator in this paper is problematic. I'm guessing this is meant to be the kernel feature map, but in Def 1 it is introduced as an arbitrary element of the RKHS. In Def 5 Phi_H and Phi_G are initially introduced via the phrase "where an element is denoted by Phi_G" seeming to imply that is again simply an example element of their respective RKHSs. Later in this definition the well-known (kernel) cross covariance operator "C_YX = E_YX [ Phi_G (y) \otimes Phi_H (x)]" which I am assuming implies that it is the kernel feature map, since otherwise this quantity is simply a scalar. Finally in the description of their main algorithm step 1 the refer to Phi_G^T which is highly problematic since Phi_G is presumably well defined here, meaning it is the kernel feature map which is not a linear operator (the whole point of kernel methods being that this operator is nonlinear), thus its "transpose" isn't something that makes sense. The authors give a vector which is equal to the expression in which Phi_G appears, this however depends on a \bold{1} vector defined as " are the top lambda (a hyper-parameter) latent representations" which is not explained further. Presumably this is referring to approximate inverse kernel feature map (which has seen some research), but this needs to be further explained, or at the very least a citation is needed. 

General Exposition:
-I am not sure why the approach to the paper is being framed in a Markov setting, when it seems like the authors really only need kernel conditional embeddings which have the exact same form as the Perron Frobenius operator the authors use. The authors even cite the kernel conditional embedding paper Song et al. 2009. Things like (1) which only serve to complicate the exposition unnecessarily considering there is no Markov process type of structure in the main algorithm.

-p. 2 Rationale: "For instance, it is known that given infinite-dimensional observables of the input measurements, there exists a linear transfer operator that perfectly pushes one probability density to another. Of course, this is not practically beneficial because explicitly constructing such infinite-dimensional observables for the data could be intractable. Nonetheless, results in control theory demonstrate that the idea can still be effective in specific cases, using approximations with either spectral analysis of large but finite number of observable functions " This sort of thing needs to be decompressed, I don't know what is meant by an "observable" I assume it has something to do with control theory or Markov process theory. This seems to niche for me for the deep learning crowd.

-The Riemannian geometry in the paper needs to be explained more for the average deep learning researcher, e.g. its not clear why the "geodesic interpolation" is doing at or why its algorithm makes sense.

-The main algorithm needs to be explained with more details as to _why_ we are doing the steps.
review_title: Very confusing paper.

Meta-review: All four reviewers were against accepting the paper. A major point shared by everyone was lack of clarity: this included its overall writing, its discussion toward prior work, and imprecise math to explain the ideas. The paper did improve quite a bit over its revisions. Whether this clarified all of the reviewers' understanding of the paper remains unclear. The work may ultimately need another cycle of reviews to assess its quality.

Another shared point are a number of recommended ablations in the experiments, as well as going through more comprehensively in the set of studied datasets (R3), effect of AE choices (R2), and alternatives to the geodesic (R1, R2).
Decision: Reject