Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper proposes a density score, which is defined based on a density or some other score such as Fréchet Inception Distance (FID).

While the given problem is an important one, there are some substantial problems with the proposed approach:
1) The paper does not mention the limitations of the FID which, as described in (Shmelkov et al, ECCV 2018. How good is my GAN?), are: i) it is based on the pre-trained Inception network and therefore does not exactly match the distributions over the data in other datasets; and ii) crude approximation of the scores by Gaussian distributions. Overall, this score is empirical and aims at circumventing the subjective analysis and it should be better reflected in the paper. 
2) The justification of the normal density based index (Section 2) seems weak. While it is obvious that this score could be used, is it possible to make the empirical assessment? E.g. compare between two scores on an extensive amount of data.
3) In addition to the previous comment, there is a substantial problem in the experimental results that all the observations are qualitative and based only on a few images. Further analysis of the realism index on the real images which are not included into the training dataset could improve the analysis. 
4) Following up on the previous comment, the experiments on the real images may need the values from the latent space corresponding to the real images. Currently, the model has been mostly assessed using DCGAN (with the assessment of VAE in Figure 4 but only for FID score), while it is stated after equation (1) that 'This case covers both GAN-like and AE-like models.'  This might be used for the assessment of the realism index on real images as stated before. If the proposed model were assessed with variational auto-encoders (VAE) and flow based models, it would make it possible to transform between the latent representation and the data themselves.  From another perspective, experimental results on different types of models (VAE-type, flow based type such as GLOW) for different types of interpolation are needed for the sake of experimental completeness. It would help emphasise the limitations of the method and difference in interpolation results in different models.  
4) In the optimisation section, the following statement is made: 'However, to accelerate the process of minimisation, we alternate the gradient step with the following one: first choose two random numbers i < j from the set {0, . . . , k} and then consider the linear interpolation between xi and xj given with...'. Could the authors elaborate on why does this acceleration happen? It might be necessary to give some references on the experimental results or at least provide some line of support for this phrase.
***
The following comments are not as critical but fall into the category of ‘nice to have':
5) Although the reviewer is aware that there were some experiments in the appendix on the value \epsilon, it might be a good idea to have more studies on the influence of this regularisation parameter for other datasets rather than just MNIST
6)  While figure 1 appears in the beginning of the paper, on page two, it is discussed on page seven, in the experimental section. Placing the figures closer to the narrative would improve the reading experience. 
review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: This paper introduced a linear interpolation method that could be applied to the latent space of a generative model.  With their method, interpolating instances generated by those generative models all maintain high quality in terms of the realism index they proposed.

This paper first introduced the quantity realism index, which is a measure of how well a generated instance in the latent space is fitted to the ground truth manifold of the data space. The definition of realism index is the probability measure of the sublevel set of some f (f:latent -> R+) function threshold by the f(z) for some z. f(z) could be the density function of feature in the latent space. 

There are two types of realism index which has analytic form introduced in this paper. The one based on normal density. Another is based on Frechet inception distance. If the density of latent feature is normal, normal density based index is used and it could be approximated with a analytic form; while if the density is not accessible, then f is the gaussian density of certain transformation of features in the latent space. 

If an arbitrary f is used, a kernel density estimator is used to estimate the density of log(f). After the realism index is introduced, the optimal interpolation  is the one has the highest cumulative realism index along the interpolation curve. 

To optimize, a linear interpolation is used as initialization. All the intermediate results are updated iteratively.

Results show better interpolation than linear ones. But not many baselines are available.

Overall, I think the problem is very interesting and important. The results seem reasonable although only beating an obviously flawed baseline.

Typo in Equation (3): w->s?

-----------------

After reading rebuttal and other reviews, I agree that quantitative results is crucially missing especially when the proposed method involves proposing an optimization method to find the best geodesic. I think the authors should find a reasonable model that can at least show the proposed geodesic is better than a linear interpolation in the accumulated realism score. Also some quantitative evaluation of the proposed optimization scheme will be very helpful.
review_title: Official Blind Review #2

Meta-review: This paper introduces a realism metric for generated covariates and then leverage this metric to produce a novel method of interpolating between two real covariates. The reviewers found the method novel and were satisfied with the response form the authors to their concerns. However, Reviewer 4 did have reservations about the response to his/her points 3 and 4. Moreover, in the discussion period it was decided that while the method was well justified by intuition and theory, the empirical evaluation—which is the what matters at the end of the day—was unconvincing.  
Decision: Reject