Realism Index: Interpolation in Generative Models With Arbitrary Prior
Authors: Łukasz Struski
Jacek Tabor
Igor Podolak
Aleksandra Nowak
Krzysztof Maziarz
Venue:  (2020)

Abstract: In order to perform plausible interpolations in the latent space of a generative model, we need a measure that credibly reflects if a point in an interpolation is close to the data manifold being modelled, i.e. if it is convincing. In this paper, we introduce a realism index of a point, which can be constructed from an arbitrary prior density, or based on FID score approach in case a prior is not available. We propose a numerically efficient algorithm that directly maximises the realism index of an interpolation which, as we theoretically prove, leads to a search of a geodesic with respect to the corresponding Riemann structure. We show that we obtain better interpolations then the classical linear ones, in particular when either the prior density is not convex shaped, or when the soap bubble effect appears.

Under review as a conference paper at ICLR 2020
REALISM INDEX: INTERPOLATION IN GENERATIVE MODELS WITH ARBITRARY PRIOR
Anonymous authors Paper under double-blind review
ABSTRACT
In order to perform plausible interpolations in the latent space of generative model, we need a measure that credibly reflects if a point in an interpolation is close the data manifold being modelled, i.c. it convincing. In this paper, we introduce a realism index of a point, which can be constructed from an arbitrary prior density, or based on FID score approach in case prior is not available. We propose a numerically efficient algorithm that directly maximises the realism index of an interpolation which, as we theoretically prove, leads to a search of a geodesic with respect to the corresponding Riemann structure. We show that we obtain better interpolations than the classical linear ones, in particular when either the prior density is not convex shaped, when the soap bubble effect appears.
INTRODUCTION
Since the advent of the Variational Auto-Encoder (VAE) (Kingma & Welling, 2013) and the Gen- erative Adversarial Network (GAN) (Goodfellow et al., 2014), generative models became an area of intensive research, with new models being developed (e.g. Kingma & Dhariwal (2018); Larsen et al. (2015); Tolstikhin et al. (2017)). In these models, the data distribution is mapped into the latent space. An important profit from introducing the latent space is the ability to construct interpolations, i.e. traversals between latent representations of two different objects. Meaningfulness of decoded in- terpolations often used as supporting argument for networks generalisation capability (Bowman et al., 2015; Dumoulin et al., 2016).
Interpolation is used commonly to show that models do not overfit, but generalise well (Dumoulin et al., 2016; Goodfellow et al., 2014; Higgins et al., 2016; Kingma & Welling, 2013). Intuitively, good interpolation should decode to meaningful objects, give a gradual transformation, and reflect the internal structure of the dataset. More precisely, we require that the interpolation curve (after transporting to the input space) is smooth and relatively short, while at the same time it goes through regions of high probability in the latent space (see leftmost projection in Fig. 1). However, in some cases, even for a Gaussian prior, a linear interpolation could be of poor quality, e.g. due to the so called "soap bubble effect" (Husar, 2017: Lesniak et al., 2019). This may result low quality samples in the middle of the path (White, 2016). The above argument puts the usability of simple linear interpolation in question and motivates further research in this area (Agustsson et al., 2017; Brock et al., 2016; Kilcher et al., 2017; Larsen et al., 2015; Lesniak et al., 2019; White, 2016).
In this paper, we construct general interpolation scheme, which works well for arbitrary priors. We introduce notion of a realism index of an element of the latent space, which naturally gener- alises to arbitrary curves. We show that in general the proposed method can be regarded as a search for geodesics in a respectively modified local Riemann structure. The realism index can be either defined internally, with the use of the prior latent density, or by some external feature space, e.g. similarly to Fréchet Inception Distance (FID) score (Heusel et al., 2017). In addition, a simple to implement iterative algorithm, that optimises an interpolation with respect to the intro- duced index. As consequence of approach, we obtain interpolation which simultaneously tries to optimise the two following features:
propose
the interpolating curve goes through regions where the generated points are realistic,
the length of the curve transported the input data space is small.

---

Under review as a conference paper at ICLR 2020
1.0
1.0
5.
0.0
0.2
3.6
8.
52
-24
30 index of point from path
10 index of point from path
-0.
3.4 0.6 0.0
Figure 1: (left right) Interpolation (projected onto two selected dimensions) for semicircle distribution: green - linear, grey consecutive interpolation steps, red final proposed interpolation; values of the proposed reality index ri for consecutive points in the linear interpolation; values of ri for points in the optimised interpolation.
The above approach is especially valuable if either the prior density is not convex-shaped when a kind of soap-bubble effect appears (i.c. when in the generative model, the points generated near the origin of much worse quality than the ones chosen randomly).
2 REALISM INDEX OF A POINT
Let X C R. be a dataset. Consider the case when we are given manifold model for X, which consists of decoder & from the latent Z RD to the input data space:
R.
This case covers both GAN-like and AE-like models. We define the realism index Z function
ri Z [0,
such that high values of ri(z) indicate that ((z) is indistinguishable from the elements of X. In general, the optimal choice of can be nontrivial, and may depend on the generative model in question. In this paper we study two possible natural constructions. If the density f in Z is given, and its high values at point = imply that G(=) is more realistic, we can base the construction of the realism index on the density. This assumption was empirically observed to be true for some generative models, such as the GLOW model (Kingma & Dhariwal, 2018). A different external approach can be constructed by using separate network, which checks if given point has similar features to that of the samples in the dataset (idea based on the Fréchet Inception Distance FID score).
Normalisation procedure. In all of our constructions we assume that we are given a function
f + R4
with higher values indicating more realistic points. To obtain the realism index based on f we first need to apply the following normalisation procedure
Let us first show the basic idea. Suppose that we have ordered the elements of the set X (in the representation given from the latent Z) according to the order introduced by f. Then as the realism index of given point a we understand its normalised order in the sequence f()- This motivates us to state the following definition.
Definition 1. Let X be random vector in Z,, such that our data comes from the distribution GXX). We define the realism index based on the function f by the formula
ri(z; f) == p(f(()))))))))))))))))))))())p S f(z)).
where p denotes the probability.
In general, even if is bounded, the obvious normalisation given by ri(s)) f(x)/ max would not work, Appendix A, Remark 1.
2

---

Under review as a conference paper at ICLR 2020
Observe that ri(z,f) f) ((w:f(m))(E)) f(s)ds. Clearly it is the PDF of random variable f(x) point f(2).
The proposed index lies in the [0, 1] interval, and attains the value of 1 only for points where f attains maximal value. If f is clear from the context, for notational convenience we shall write ri(2) instead of ri(z: f).
To practically apply the above concept, we need to be able to tune it to different generative models. To achieve this, we choose value and, similarly to the approach used in GLOW (Kingma & Dhariwal, 2018), we rescale
(z)(2) = (1 .r(z),
(4)
so that (2) E [E, 1 - E]. See Sec. 4 for further discussion on this rescaling.
Realism index based on the normal density. We shall now discuss the realism index based on the standard normal density f = .N 1) in the D-dimensional latent Z. Let us choose point from the latent and let X denote the random vector with density. Note that almost all points generated from the Gaussian density in RD lie inside the sphere S(0, D1/2). Points outside of this sphere can be considered unrealistic, while those inside might be obtained with random sampling.
We want to compute the probability
ri(z) - p([(p((()))))))))))))))))))))- S f(=)).
(5)
By some easy calculations we get (see Appendix A for more details)
ri(z) 2 2 (VD- ||=]|]
erf
(6)
We can see that the density based realism index correctly identifies the latent points that lay inside the sphere S(0, D1/2) having an index approximately 1, and points outside sphere as having index approximately O. Observe, that this behaviour is not recognised by the density itself, which has no clear change at the border of the sphere S(0,21/3)
Realism index based on the Fréchet Inception Distance (FID). If the density either not avail- able or not completely reliable, we can base the realism index on an external measure of sample credibility. To that end, we use an approach inspired by the Fréchet Inception Distance (Heusel et al., 2017) (FID). To compute FID score we start with pretrained Inception network I. The entire training set is passed through / to obtain set of feature vectors, and we estimate density N N(rr, E by computing its mean Jrr and covariance matrix Σr. We apply the similar procedure to the data W generated by our model, and obtain the density (ww,,aw). Then we compute the Fréchet distance between two normal densities by the formula
FID(N(Pr, N Ew)) ||r mwll + Tr(Er + Σw Ew)
However, in contrast to the original definition, we aim to compute the credibility of a single gen- erated point, not the distance between two distributions. To achieve that goal, we simply the likelihood of the point x transported through with respect to density N ((ur, Er):
f(z) = N(ur (I))),
and apply ri, as the constructed realism index. Observe that this realism index is based also on the density, but not in the latent space itself, but in some feature space constructed with the use of external network I.
Numerical estimation of the realism index. Clearly, for an arbitrary function the realism in- dex does not have a closed form. order to obtain differentiable estimation ri, we draw sample W = (w:); from the random variable X (or simply choose it from the dataset), and compute values of (f(w:)); Since the considered values are non-negative, to estimate the density we first proceed by logarithm to whole of R by taking 4 log fw,,,, and compute either kernel or GMM density esti- mation g of the random variable log(f (x)). Finally, we obtain that the estimator of the realism index ri(2; f) is given with the use of the cumulative density function of g: ri(2; f' edf, adlg((og (log (())))
3

---

Under review as a conference paper at ICLR 2020
3 REALISM INDEX OF AN INTERPOLATION
Our concept for the definition the realism index for path is inspired by transition between movie frames. The interpolation may be viewed of frames, where the first frame denotes the beginning of the path and last its end. Interpreting realism index as a probability that a given frame is realistic, we can define the respective index of the curve as the product of all its points.
where
Realism index for naturally parameterised curves. Let 7 [0,7] Z be an interpolating curve, such that
r(0) and ((T) y, for some given n Z. Additionally, we assume that the Gy is naturally parameterised
(1()))(011 = for / E [0, T].
(8)
We discretize the curve by fixing a time-step T/k (where denotes the number of frames) and consider the sequence of intervals [y(0), (TTk))(T [n(T - rT)). To obtain the reality
measure of this sequence we compute the product of all realisticity values of its points to the power equal to their "duration" T/k
ri(?(1))) .... ri()()))))
where t are the arbitrarily chosen intermediate points in the intervals [iT/k, (i 11TT/kk 1)T/k]. the logarithm of the above expression and proceeding with & - oo we get
By taking
log (t)) 00. Σ log r(t)) - S
(10)
Consequently, we introduce the realism index of a naturally parameterised curve [0, 2] + IR by the formula
ri(>) exp 1/ log (ogr(((()))))) [0, 1].
(11)
Since every curve be uniquely naturally parameterised, we interpret its index as the index of its natural reparameterisation. Therefore we arrive at the following general definition.
Definition 2. Let ri be given realism index Z. For an arbitrary curve y [0,T] - Z we define the realism index ri of y with
ri()) -cxp cxxp (1g1
[0.
(12)
We further prove that the realism index of curve is equal to its length with respect certain Riemann structure on the latent space. We will utilise this result the next section order to connect the search of optimal interpolation the search of geodesies. Directly from the definition we get the formula for the realism index in terms of the latent
log - log ri(>) I ((())) - log ((() (())))(())(()))))))))))
lo log beg
= _o
where (z()) denotes the derivative of G at point æ. Consequently, we obtain the following theorem:
Theorem 1. Let the Riemann structure in the latent space Z be defined with the local scalar prod- uct (, () at point using the following formula
(v, w)x x TT Agw where Ag log (1i()))d(())
(13)
Then
ri(>) exp(hlength((s (.)))) (+)z)),
(14)
where length is the number of points in path.
4

---

Under review as a conference paper at ICLR 2020
4 OPTIMAL INTERPOLATION
Considering the results from the previous section, we are able to formulate the definition of an ri-optimal curve.
Definition 3. Let ri be realism index in Z for a generative model G : Z +
2Z-R* IRN, Let 2, E Z be fixed. We call a curve Y [0, - Z such that 7(0) T(T) ri-optimal (or, shortly, optimal) interpolation if has the maximal realism index from all curves joining x with y-
Study first the issue of searching for, at least locally, optimal interpolations. Theorem allows us to reformulate the problem as a task of finding geodesics. Consequently, the standard results from Riemann geometry apply (see Spivak (1999, Chapter 9)). Without loss of generality we reduce the problem to the case I' = 1 and optimise the length functional. However, due to the uniqueness of the local minima and local convexity of the functional, we can minimise the energy functional instead
E = S
(15)
The additional advantage is that curves which minimise the energy functional are parameterised proportionally to the natural parameterisation. Concluding, by applying Theorem 1, the optimal curve : [0, Z, 700 n, 7(1 = y. with respect to the realism index minimises:
1// S. log" (((()
E) -
(16)
In general, the search for geodesics can lead to nontrivial computations involving second derivatives. However, for some special cases, we can significantly simplify the minimisation process. To justify this claim, we first introduce the formula for the discretization of the integral in the energy functional. Let y [0, 1] > RN be curve such that
(0) x and (1) y:
(17)
and divide the interval [0, 1] into & equal sub-intervals, and denote the values
(i/k)) n; for 0, O.....k.
(18)
For vectors 21 21...., Tk-1 in R (where zo x, 2x v). approximate the value of (16) with
(T) log²
11/A) (2/8)/111111
1/k
k-2
log" ri(a)+r((a))
||11
x012 + k:
log"
ri(z,) +((z,))((((.11)))
| z;|12
2
2
==1
log
ri(es
()rr(())
24-1 11.
Considering all the above computations, compute optimal interpolation we need to minimise | xx12 -
2 E (z 1) = log ri(x)-ri(z1)
k-2 log ri(s:)++i(z...)) 2 2/12
k
+ log ri(es ()+ by - - x2-1
(19)
2
over 21: ......,,2 Tk-1 E R
Optimisation procedure. In all the experiments in the paper we achieve this goal using the stan- dard gradient descent method initialising x; with a linear interpolation x; (1 { x + f.y for i 1,... k However, to accelerate the process of minimisation", we alternate the gradient step
This step is essential when the initial interpolation passes through regions with density close to zero, which could result in vanishing gradient problem, see Appendix C.
5

---

Under review as a conference paper at ICLR 2020
with the following one: first choose two random numbers j from the set {0, ...,,, k} and then consider the linear interpolation between 2, and zy given with
Tt J + J for I E {i 1.
1}.
Finally, if the linear interpolation has smaller energy then the original part, i.c. when
Egg . -.-1) < E (1i+1: :. ((j])))), we replace 2y by 3y for = {i + J 1 1}.
Effects of s-regularisation on optimisation. Given reality index ri(.), see eq. (4), have introduced its regularisation (z) = (1-2)1
E [0, 1/2). In practice, in all experiments we typically choose E 0.1. The reason behind such regularisation twofold. First, if E 0 and ri is zero (or numerically close to zero) on some subset of the domain (for example if the density is zero), the optimisation for points with initial interpolating interval there is unmanageable (observe that in the realism of the curve we take the logarithm of the index). The case when = 0 and ri = at some subset the domain can also cause problems, as then logarithm the reality index this set is zero, and consequently the interpolating curve has cost of staying or going through this region. Consequently, in our experience it best to regularise ri by restricting its image to a subset of interval [e, - =].
Let us now discuss the special limiting case when E = 1/2. Then clearly
rig 1/2.
Let us first recall that we are given decoder (generator) G 2 - R-Y, such that the data set lies in the manifold given by M = G(Z). Let us consider the interpolating curve y [o, 1] - Z. Observe that in that case the energy function does not depend ri and equals
(y) = 1/2 L/ (rgg( (((((((((()))))))()))))))))))))
= log² L. 110()(0))
log)(2) length(G o
Thus the minimal (T) value would be obtained for a curve joining with y whose length mea- sured in the input space is minimal. Thus if π ((x) and (gy) are points in the input space, then the interpolating curve with minimal energy is equal to one that connects points and ÿ with curve in the manifold M which has minimal length (measured in input space). Consequently, al- though the limiting case does not take into account the realism index of the interpolation, it still will usually produce interpolations comparable to the common linear. See Appendix D for additional analysis.
EXPERIMENTS
In all the considered experiments we apply the regularised version or realism index rig with e = 0.1. First we going to consider the case when the index is based on the prior density in the latent, and next we briefly discuss the realism index based on the FID score.
Density based index. In this part we demonstrate our method's ability produce more mean- ingful interpolations. In order to achieve this goal we use a DCGAN model (Radford et al., 2015) trained on MNIST and Celeb-A datasets (LeCun et al., 1998; Liu et al., 2015). We consider non-trivial latent created from conjunction of three multidimensional Gaussian distributions
p(z) = Di Ex)(s) for E Z,
(20)
where p4 = (i,, 0. ... ..".....0'' 0)T R20 for #1 (2, 6). وز (0. A3 (2 and Σ,
Mr Mia for M12 = Mn1 0, M22 is 2-D array with 0.5 on the diagonal and zeros elsewhere,
Mn Mga
and M() = [ ]. (2 (2) = [] M (3) = - From here on we shall call it semicircle, see
Fig. (leftmost subfigure).

---

Under review as a conference paper at ICLR 2020
Figure 2: Examples results for linear interpolation (top row in each example) and our results (bottom row in each example) for Celeb-A dataset, with semicircle prior trained DCGAN. Observe that since the linear interpolation the middle point far from the density of the data, we can often observe in it some artefacts.
Results of sampling from a GAN trained with the semicirele prior shown in Fig. 2 with respective projection shown in Fig. 1. We randomly choose two points from different ends of the distribution and start the algorithm's is minimisation procedure given with eq. (19) with linear interpolation (green line in Fig. 1). It is obvious that the midpoints of initial path are sampled from a very low- density areas, completely outside the prior.
In Fig. you can also the ri index values for the linear (center) and proposed (rightmost image) interpolations. Our method allows for more dynamic behaviour in this matter, especially at the end points. Images in the path obtained with proposed algorithm differ more, and at the same time seem more real.
Figure 3: Examples of results for linear interpolation (top rows of interpolation pairs) and our FID- score based results (bottom rows of interpolation pairs) for Celeb-A dataset. The interpolation was optimised had 50 midpoints, and we show here 10 of them.
FID-score based realism index. In order to test our FID-score based realism index, we perform experiments on GAN, trained on the Celeb-A dataset. Since samples from a high dimensional Gaussian approximately cluster on a sphere, points in the latent space that are very close to the origin are virtually never during training. Therefore, samples generated in that may may not be realistic. We selected a GAN model, for which we noticed that decoding latent codes near the origin gives low quality samples (the earlier mentioned soap-bubble effect), which makes

---

Under review as a conference paper at ICLR 2020
for perfect test environment for the FID-score based realism index. We aim choose interpolation endpoints x' and y in such a way, that the linear interpolation between them would pass close to the origin in the latent space. To that end, first sample uniformly, and then produce from by multiplying by- -1 a random subset of dimensions.
The results of these experiments yield very low quality samples (especially in the middle the interpolation). After optimising the linear paths using the FID-score realism index, the paths avoid unrealistic regions in the latent space, producing samples of much higher quality. Note that this experiment does not require access to the density. Instead, it is based only on the training samples.
be seen in Fig. 3. We
that the linear interpolations
Even though FID score has some limitations (Shmelkov et al., 2018), observe that its application our model results more realistic interpolations, as compared linear ones. Use of a measure more adapted to evaluating quality of data could result in further improvement in the realism of interpolations.
Further discussion. In this part discuss when the linear approximations will be close those given by linear. Let us first consider the case when the generator is linear with prior uniform distri- bution in the latent.
Observation 1. Let U be a convex bounded set in Z. and let f denote the uniform distribution on U. We consider the case of linear generative model, i.e. where G linear map x -> Ar (with A injective). Then the ri-optimal interpolations are given by the linear interpolations.
Proof. Clearly, we can equivalently compute the realism index of a curve y connecting two points y by computing the standard euclidean distance of Ay in the convex AU. More precisely: ri(y; unit)) = exp(-length(A-)) Since linear maps move intervals onto intervals, we obtain the assertion of the observation.
In practice, similar behaviour happens when the derivative of the generator has small variation and the realism index close to being constant one. Observe, that by equation (6), for a genera- tive model with high dimension of the latent space and Gaussian prior, the realism index is almost constant on a linear interpolation of arbitrary randomly chosen points (except for a possibly small neighbourhood of the endpoints). Consequently, when the derivative of generator G does not vary too much in the vicinity of the linear interpolation, the linear interpolation will be close to opti- mal find by our approach. In practice, we observe this behaviour in auto-encoder based generative
Linear (top) and our interpolation (bottom) in the WAE (Tolstikhin et al., 2017) model. Observe that the consecutive elements of the interpolations are practically identical.
Figure 4:
models, such as Wasserstein Auto-Encoders, see Fig. 4. This follows from the fact that too high variation (of derivative) of the decoder is penalised (regularised) by its approximate inverse given by the encoder.
CONCLUSIONS
In this paper studied the problem of generating meaningful interpolation from previously trained generative model, either GAN or generative auto-encoder. We claim that a good inter- polation should both reveal the hidden structure of the dataset, well be smooth and follow the true data distribution, i.e. produce realistic elements.
In order to produce curves satisfying these conditions define realism index of path, which takes into account both density values and and differences between consecutive decoded images to ensure smoothness. We show how define realism index using either known density or, in
8

---

Under review as a conference paper at ICLR 2020
it is either not available not reliable, how base it some external measure, e.g. the FID score. We have proved that this interpolation procedure is equal to finding a geodesics with reality index equal to its length in respect to some latent space Riemann structure.
For the practical use, we have defined the notion of an optimal interpolation, and proposed a simple and efficient numerical procedure for its search. The experiments show that the constructed inter- polations are in superior to the linear ones, making it possible to escape regions of low data density or low data quality, both for the density- and FID-based approaches. This especially visible if the prior density is not Gaussian, when the linear interpolations often proceed through regions in space of extremely low density, Another example when the linear interpolations are suboptimal to our method is given by the standard GAN model when the soap bubble effect appears.
REFERENCES
Eirikur Agustsson, Alexander Sage, Radu Timofte, and Luc Van Gool. Optimal transport maps for distribution preserving operations on latent spaces of generative models. arXiv:1711.01970, 2017.
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal Józefowicz, and Samy Bengio. Generating sentences from a continuous space. 1xXiv:5511.0644 2015.
Andrew Brock, Theodore Lim, James M. Ritchie, and Nick Weston. Neural photo editing with introspective adversarial networks. 17Xiv:100.00000 2016.
Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Olivier Mastropietro, Alex Lamb, Martin Ar- jovsky, and Aaron Courville. Adversarially Learned Inference. arXiv: 1666: 00704, 2016.
I. J. Goodfellow, J. Abbdet-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative Adversarial Networks. arXiv:1406.26611 2014.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium. In Advances in Neural Information Processing Systems, pp. 6626-6637, 2017.
Irina Higgins, Loic Matthey, Arka Pal, Christofer Burgess, Glorot Xavier, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. B-VAE: Learning basic visual concepts with con- strained variational framework. In Proceedings of International Conference on Learning Reepre- sentations ICLR, 2016.
Ferenc Husar.
Gaussian distributions soap-bubbles. httt:://ww.innfereme ve/ 2017. Ac-
cessed: 2019-03-18.
Norman Lloyd Johnson, Samuel Kotz, and Narayanaswamy Balakrishnan. Continuous univariate distributions, vol. 1. John Wiley & Sons, 1994.
Yannic Kilcher, Aurélien Lucchi, and Thomas Hofmann. Semantic interpolation in implicit models. arXiv:1710.11 11381, 2017.
Diederick P. Kingma and Prafulla Dhariwal. Glow: generative flow with invertible 1x1 convolu- tions. arXiv:7807.03039 2018.
Diederick P. Kingma and Martin Welling, Auto-Encoding Variational Bayes. arXiv:1312.611, 2013.
Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, and Ole Winther. Autoencoding beyond pixels using a learned similarity metric. arXiv:1512.09300 2015.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 1):2278222233) 2324, 1998.
Damian Lesniak, Igor Sieradzki, and Igor Podolak. Distribution interpolation trade off in generative models. In Proceedings of International Conference on Learning Representations ICLR, May 2019.

---

Under review as a conference paper at ICLR 2020
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision ICCV, December 2015.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks, arXiv:1511.06444 2015.
How good is my gan? Konstantin Shmelkov, Cordelia Schmid, and Karteek Alahari.
arXiv:/807.09499, 2018.
Michael D Spivak. A comprehensive introduction to differential geometry, Volume One, Third Edi- tion. Publish or perish, 1999.
Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein Auto- Encoders. 1rX15517 2017.
Tom White. Sampling generative networks: Notes on a few effective techniques. arXiv:1609.04400 2016.
10

---

Under review as a conference paper at ICLR 2020
A CASE OF THE NORMAL DENSITY.
We shall now compute the probability index of the standard normal density f = No0, 7) in the D-dimensional latent Z. Let choose a point z from the latent and let X denote the random vector with density f. We want to compute the probability
ri(z) - p([(x) S f(=)).
(21)
Observe that from the definition of normal density we have
p([()= 2 f()) p[||X 2 | 1 p X|X|² < ||
(22)
Since |||] has the chi-square distribution with degrees of freedom, we obtain that
ri(z) 1))
(23)
where F(r; D) denotes the cumulative distribution function of the chi-square density x² (with D degrees of freedom). Let us now proceed with an asymptotic analysis. Observe that
x||I1| < ||=| = -(v < =II+
(24)
If Y x²) then for large D > 30, v2Y /2D - is approximately normally distributed (see Johnson et al. (1994, formula (18.23) on p. 426)). Consequently
ri(z) - p[ S =1) ~ 인 ([22|2=|| -).
(25)
where & denotes edf of standard Gaussian. Since (r) = 1 [1 eer((r//2) we get
ri(z)
1 2
erf (V D ||=1|
(26)
Remark 1. The above formula implies that for the normal density f = ((, RD the realism index is approximately 1 in the ball ( rp 3) and approximately outside of the ball ( 3), where TD VVDD D This behaviour is natural and expected, most of the points generated from normal density concentrate around the sphere (0..rp)
Let us now discuss why the normalisation given by eq. (4), and not the seemingly natural given for bounded ff by the the formula
f(22)
ri(z)
max S
Consider the case when NO, in RD. Then ri(2) exp(-11=1//2), which means that ri(0) = 1, but ri(2) = exp(-D/2 + 1/4) for 2 E S(O,TD). Since the randomly generated point from N(0, 1) has norm close to rD. this implies that almost every point which comes from the normal density would have realism index close to O, which would be of undesired and pathological behaviour.
In practice, observe this behaviour in auto-encoder based generative models, such as Wasserstein Auto-Encoders, see Fig. 4. This follows from the fact that too high variation (of derivative) of the decoder penalised by its approximate inverse given by the encoder.
B DENSITY BASED REALISM INDEX IN GAN MODELS
In this section present additional results for the optimisation of the density based realism index of an interpolation curve in a GAN model. The experiments are conducted on the Celeb-A and MNIST datasets, using the DCGAN architecture. The optimisation procedure initialised with linear interpolation consisting of 50 points and implemented using Adam optimiser. To show the relation of the starting linear interpolation to the one obtained from the proposed procedure, see e.g. Fig. 1, we perform a projection of h interpolation 2; points onto point (ar, y) E IR such that
x 20 + y 2z = ā; for = O,...,k,
(27)
where = XXX z; are the latent space points z; Z and X [20,
11

---

Under review as a conference paper at ICLR 2020
Figure 5: Interpolation points in DCGAN model trained with normal prior. each pair the top represents the linear path and the bottom shows the results of our optimisation procedure. Each line consists of 10 equally spaced images selected from the 50 points that form the path.
04
Index of vol from pach
Index of point from path
Index of interval from path
index of point 23 path
Figure 6: Top: Projections (see (27)) of the latent sample's density (blue dots), linear (green line) and proposed (red line) interpolations shown in Fig. 5 from top to bottom, respectively. Bottom: The squared La distances between consecutive points and the realism index ri of each point in the path from the last example in Fig. 5. Left: The initial linear path. Right: The path at the end of optimisation procedure.
The resulting images for Celeb-A for the Gaussian prior are shown in Fig. 5, and Fig. shows their path projections. can be easily noticed that the proposed interpolation gives more dynamical objects. Also the interpolation is pulled by the density. Similar observations can be observed for
12

---

Under review as a conference paper at ICLR 2020
semicirele prior. The resulting images for Celeb-A together with their path projections for semicircle prior are shown in Fig. 7. We made also the same experiments for MNIST dataset. The resulting images for this data for the Gaussian and semicircle prior are shown in Fig. 8.
In the case of auto-encoder based architectures we advice the use the WAE model, since obtains slightly better interpolations and reconstructions than VAE see Fig. 9.
-37
-34
-00
Figure 7: Top: Interpolation points in DCGAN model trained with semicircle prior: In each pair the top represents the linear path and the bottom shows the results of our optimisation procedure. Each line consists of 10 equally spaced images selected from the 50 points that form the path. Bottom: Selected projections of the latent sample's 's density (blue dots), linear (green line) and proposed (red line) interpolations shown in 'top" part together with the interpolation progress (in gray).
C OPTIMISATION PROCEDURE
the performed experiments we use the optimisation described in Section reduce the number of necessary iterations in our method. In the initial phase of the optimisation of the linear interpolation, the value of the gradient for the middle points (located in the middle of the interpolation curve) can be extremely small due the diminishing density. This is particularly evident for DCGAN model with a semicircle prior, see Fig. 10. In consequence, this part the curve converges slower than the points lying closer the ends of the curve. Our experiments for semicircle distribution show that
13

---

Under review as a conference paper at ICLR 2020
4
4
4 9
7
7
7
7
7
4
9
9
9
9
4
4
4
4
4
9
9
9
9
9
7
9
9
4
4
4
4
4
4
4
4
9
6
?
I
1 7
4 1
4 4
4 4
4 4
7
7
7
7
7
7
7
7
7 7
F 7
J
/
/
7
7
7
7
7
7
4
4
4 4
4
4
4
5
5 5
5
4
4
4
4
5
5
5
5
5
5
5
Figure 8: Linear and proposed interpolation paths for the MNIST dataset (a semicircle latent prior model in left column, a multidimensional normal in right column) in a DCGAN model.
8 8
8
8
8
8
6
6
8
8
8
8
8
6
6
6
6
6
6
6
6
6
8
8
8
8
8
8
8
8
8
8
8
6
6
6
6
Figure 9: Comparison of interpolations in VAE (left) and WAE (right) models. The top line the linear interpolation, the bottom one is the obtained with use of the density realism index approach.
we obtain the same final result without using the described acceleration, but with up to times more iterations. This optimisation procedure is crucial only for non typical densities; for the classical normal prior no significant impact on the number of iterations is observed.
D THE & VALUE IMPACT ON THE INTERPOLATION CURVE
In Fig. 11 we present squared L2 distances between consecutive points in the optimised path from the last example from Fig. 2. In this experiment use different values of epsilon 10-3, 0.01, 0.1, 0.25 for rig (see eq. (4)). As one can see, distances between consec- utive points do vary much for values much larger than zero.
Higher e values result in more equally spaced interpolations, and the interpolation optimisation process is faster. However, higher & result in puts different emphasis on terms of the formula optimised (see eq. (19) and discussion on e in Sec. 4), i.c. lengths start to be more important (hence the inter-point distances) than individual point realities.
On the other hand, lower E values result in non-equally spaced points on the interpolation, which is easily visible in Fig. 11, while the optimisation process is slower.
Empirically, we found = = 0.1 to be optimal, and therefore it is used as default value. Experiments conducted on a Celeb-A dataset gave similar results.
1.0
1.0-
0.6
0.6
0.2
0.2
-0.2
-0.2
-0.4
-0.2
0.0
0.2
0.4
0.6
0.8
1.0
-0.4
-0.2
0.0
0.2
0.4
0.6
0.8
1.0
Figure 10: Interpolations (projected defines by (27)) for semicircle distribution: green linear, grey - consecutive interpolation steps, red - final proposed interpolation. The picture on the left shows result of our method without "Optimisation procedure', and right side with optimisation.
14

---

Under review as a conference paper at ICLR 2020
124
1.9
164
19
30 path
23
26
path
= 10-5
(b) 10
(c)
10
327
12-
2
36
190
30
Index interval
20 30
43
33 34
Interval from path
54
Interval from 4
20
50
(d) E 0.01
(e) E 0.1
(f)
0.25
Figure 11: The squared L2 distances between consecutive points in the last example path from Fig. for our interpolation and different values of for rig (see eq. (4)).
15


Reviews:
[{"review_id": "SJltRSD29B", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a density score, which is defined based on a density or some other score such as Fréchet Inception Distance (FID).\n\nWhile the given problem is an important one, there are some substantial problems with the proposed approach:\n1) The paper does not mention the limitations of the FID which, as described in (Shmelkov et al, ECCV 2018. How good is my GAN?), are: i) it is based on the pre-trained Inception network and therefore does not exactly match the distributions over the data in other datasets; and ii) crude approximation of the scores by Gaussian distributions. Overall, this score is empirical and aims at circumventing the subjective analysis and it should be better reflected in the paper. \n2) The justification of the normal density based index (Section 2) seems weak. While it is obvious that this score could be used, is it possible to make the empirical assessment? E.g. compare between two scores on an extensive amount of data.\n3) In addition to the previous comment, there is a substantial problem in the experimental results that all the observations are qualitative and based only on a few images. Further analysis of the realism index on the real images which are not included into the training dataset could improve the analysis. \n4) Following up on the previous comment, the experiments on the real images may need the values from the latent space corresponding to the real images. Currently, the model has been mostly assessed using DCGAN (with the assessment of VAE in Figure 4 but only for FID score), while it is stated after equation (1) that 'This case covers both GAN-like and AE-like models.'  This might be used for the assessment of the realism index on real images as stated before. If the proposed model were assessed with variational auto-encoders (VAE) and flow based models, it would make it possible to transform between the latent representation and the data themselves.  From another perspective, experimental results on different types of models (VAE-type, flow based type such as GLOW) for different types of interpolation are needed for the sake of experimental completeness. It would help emphasise the limitations of the method and difference in interpolation results in different models.  \n4) In the optimisation section, the following statement is made: 'However, to accelerate the process of minimisation, we alternate the gradient step with the following one: first choose two random numbers i < j from the set {0, . . . , k} and then consider the linear interpolation between xi and xj given with...'. Could the authors elaborate on why does this acceleration happen? It might be necessary to give some references on the experimental results or at least provide some line of support for this phrase.\n***\nThe following comments are not as critical but fall into the category of ‘nice to have':\n5) Although the reviewer is aware that there were some experiments in the appendix on the value \\epsilon, it might be a good idea to have more studies on the influence of this regularisation parameter for other datasets rather than just MNIST\n6)  While figure 1 appears in the beginning of the paper, on page two, it is discussed on page seven, in the experimental section. Placing the figures closer to the narrative would improve the reading experience. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "Hye8ZAkRFH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduced a linear interpolation method that could be applied to the latent space of a generative model.  With their method, interpolating instances generated by those generative models all maintain high quality in terms of the realism index they proposed.\n\nThis paper first introduced the quantity realism index, which is a measure of how well a generated instance in the latent space is fitted to the ground truth manifold of the data space. The definition of realism index is the probability measure of the sublevel set of some f (f:latent -> R+) function threshold by the f(z) for some z. f(z) could be the density function of feature in the latent space. \n\nThere are two types of realism index which has analytic form introduced in this paper. The one based on normal density. Another is based on Frechet inception distance. If the density of latent feature is normal, normal density based index is used and it could be approximated with a analytic form; while if the density is not accessible, then f is the gaussian density of certain transformation of features in the latent space. \n\nIf an arbitrary f is used, a kernel density estimator is used to estimate the density of log(f). After the realism index is introduced, the optimal interpolation  is the one has the highest cumulative realism index along the interpolation curve. \n\nTo optimize, a linear interpolation is used as initialization. All the intermediate results are updated iteratively.\n\nResults show better interpolation than linear ones. But not many baselines are available.\n\nOverall, I think the problem is very interesting and important. The results seem reasonable although only beating an obviously flawed baseline.\n\nTypo in Equation (3): w->s?\n\n-----------------\n\nAfter reading rebuttal and other reviews, I agree that quantitative results is crucially missing especially when the proposed method involves proposing an optimization method to find the best geodesic. I think the authors should find a reasonable model that can at least show the proposed geodesic is better than a linear interpolation in the accumulated realism score. Also some quantitative evaluation of the proposed optimization scheme will be very helpful.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a realism metric for generated covariates and then leverage this metric to produce a novel method of interpolating between two real covariates. The reviewers found the method novel and were satisfied with the response form the authors to their concerns. However, Reviewer 4 did have reservations about the response to his/her points 3 and 4. Moreover, in the discussion period it was decided that while the method was well justified by intuition and theory, the empirical evaluation—which is the what matters at the end of the day—was unconvincing.  
Decision: Reject