Title: REALISM INDEX: INTERPOLATION IN GENERATIVE MODELS WITH ARBITRARY PRIOR

Abstract: In order to perform plausible interpolations in the latent space of a generative model, we need a measure that credibly reflects if a point in an interpolation is close to the data manifold being modelled, i.e. if it is convincing. In this paper, we introduce a realism index of a point, which can be constructed from an arbitrary prior density, or based on FID score approach in case a prior is not available. We propose a numerically efficient algorithm that directly maximises the realism index of an interpolation which, as we theoretically prove, leads to a search of a geodesic with respect to the corresponding Riemann structure. We show that we obtain better interpolations than the classical linear ones, in particular when either the prior density is not convex shaped, or when the soap bubble effect appears.

1. INTRODUCTION:

Since the advent of the Variational Auto-Encoder (VAE) (Kingma & Welling, 2013) and the Generative Adversarial Network (GAN) (Goodfellow et al., 2014) , generative models became an area of intensive research, with new models being developed (e.g. Kingma & Dhariwal (2018) ; Larsen et al. (2015) ; Tolstikhin et al. (2017) ). In these models, the data distribution is mapped into the latent space. An important profit from introducing the latent space is the ability to construct interpolations, i.e. traversals between latent representations of two different objects. Meaningfulness of decoded interpolations is often used as a supporting argument for networks generalisation capability (Bowman et al., 2015; Dumoulin et al., 2016) .

Interpolation is used commonly to show that models do not overfit, but generalise well (Dumoulin et al., 2016; Goodfellow et al., 2014; Higgins et al., 2016; Kingma & Welling, 2013) . Intuitively, a good interpolation should decode to meaningful objects, give a gradual transformation, and reflect the internal structure of the dataset. More precisely, we require that the interpolation curve (after transporting to the input space) is smooth and relatively short, while at the same time it goes through regions of high probability in the latent space (see leftmost projection in Fig. 1 ). However, in some cases, even for a Gaussian prior, a linear interpolation could be of poor quality, e.g. due to the so called "soap bubble effect" (Husar, 2017; Lesniak et al., 2019) . This may result in low quality samples in the middle of the path (White, 2016) . The above argument puts the usability of simple linear interpolation in question and motivates further research in this area (Agustsson et al., 2017; Brock et al., 2016; Kilcher et al., 2017; Larsen et al., 2015; Lesniak et al., 2019; White, 2016) .

In this paper, we construct a general interpolation scheme, which works well for arbitrary priors. We introduce a notion of a realism index of an element of the latent space, which naturally generalises to arbitrary curves. We show that in general the proposed method can be regarded as a search for geodesics in a respectively modified local Riemann structure. The realism index can be either defined internally, with the use of the prior latent density, or by some external feature space, e.g. similarly to Fréchet Inception Distance (FID) score (Heusel et al., 2017) . In addition, we propose a simple to implement iterative algorithm, that optimises an interpolation with respect to the introduced index. As a consequence of our approach, we obtain an interpolation which simultaneously tries to optimise the two following features:

• the interpolating curve goes through regions where the generated points are realistic,

• the length of the curve transported to the input data space is small. The above approach is especially valuable if either the prior density is not convex-shaped, or when a kind of a soap-bubble effect appears (i.e. when in the generative model, the points generated near the origin are of much worse quality than the ones chosen randomly).

2. REALISM INDEX OF A POINT:

Let X ⊂ R N be a dataset. Consider the case when we are given a manifold model for X, which consists of a decoder G from the latent Z = R D to the input data space:

EQUATION

This case covers both GAN-like and AE-like models.

We define the realism index on Z as a function

EQUATION

such that high values of ri(z) indicate that G(z) is indistinguishable from the elements of X. In general, the optimal choice of ri can be nontrivial, and may depend on the generative model in question. In this paper we study two possible natural constructions. If the density f in Z is given, and its high values at point z imply that G(z) is more realistic, we can base the construction of the realism index on the density. This assumption was empirically observed to be true for some generative models, such as the GLOW model (Kingma & Dhariwal, 2018) . A different external approach can be constructed by using a separate network, which checks if a given point has similar features to that of the samples in the dataset (idea based on the Fréchet Inception Distance FID score).

Normalisation procedure. In all of our constructions we assume that we are given a function

f : Z → R +

with higher values indicating more realistic points. To obtain the realism index based on f we first need to apply the following normalisation procedure1 .

Let us first show the basic idea. Suppose that we have ordered the elements of the set X (in the representation given from the latent Z) according to the order introduced by f . Then as the realism index of a given point x we understand its normalised order in the sequence f (i) . This motivates us to state the following definition. Definition 1. Let X be a random vector in Z, such that our data comes from the distribution G(X).

We define the realism index ri based on the function f by the formula

EQUATION

where p denotes the probability.

Observe that ri(z, f ) = {w:f (w)≤f (z)} f (s)ds. Clearly it is the PDF of random variable f (X) at point f (z).

The proposed index lies in the [0, 1] interval, and attains the value of 1 only for points where f attains maximal value. If f is clear from the context, for notational convenience we shall write ri(z) instead of ri(z; f ).

To practically apply the above concept, we need to be able to tune it to different generative models.

To achieve this, we choose a value ε < 1 and, similarly to the approach used in GLOW (Kingma & Dhariwal, 2018), we rescale ri

EQUATION

so that ri ε (z) ∈ [ε, 1 -ε].

See Sec. 4 for further discussion on this rescaling.

Realism index based on the normal density. We shall now discuss the realism index based on the standard normal density f = N (0, I) in the D-dimensional latent Z. Let us choose a point z from the latent and let X denote the random vector with f density. Note that almost all points generated from the Gaussian density in R D lie inside the sphere S(0, D 1/2 ). Points outside of this sphere can be considered unrealistic, while those inside might be obtained with random sampling.

We want to compute the probability

EQUATION

By some easy calculations we get (see Appendix A for more details)

EQUATION

We can see that the density based realism index correctly identifies the latent points that lay inside the sphere S(0, D 1/2 ) as having an index approximately 1, and points outside sphere as having index approximately 0. Observe, that this behaviour is not recognised by the density itself, which has no clear change at the border of the sphere S(0, D 1/2 ).

Realism index based on the Fréchet Inception Distance (FID). If the density is either not available or not completely reliable, we can base the realism index on an external measure of sample credibility. To that end, we use an approach inspired by the Fréchet Inception Distance (Heusel et al., 2017) (FID). To compute FID score we start with a pretrained Inception network I. The entire training set T is passed through I to obtain a set of feature vectors, and we estimate its density N (µ T , Σ T ) by computing its mean µ T and covariance matrix Σ T . We apply the similar procedure to the data W generated by our model, and obtain the density N (µ W , σ W ). Then we compute the Fréchet distance between two normal densities by the formula

F ID(N (µ T , Σ T ), N (µ W , Σ W )) = µ T -µ W 2 2 + T r(Σ T + Σ W -2(Σ T Σ W ) 1/2

). However, in contrast to the original definition, we aim to compute the credibility of a single generated point, not the distance between two distributions. To achieve that goal, we use simply the likelihood of the point x transported through I with respect to density N (µ T , Σ T ):

f (x) = N (µ T , Σ T )(I(x)),

and apply ri f as the constructed realism index. Observe that this realism index is based also on the density, but not in the latent space itself, but in some feature space constructed with the use of external network I.

Numerical estimation of the realism index. Clearly, for an arbitrary function f the realism index does not have a closed form. In order to obtain differentiable estimation of ri, we draw sample W = (w i ) i from the random variable X (or simply choose it from the dataset), and compute values of (f (w i )) i . Since the considered values are non-negative, to estimate the density we first proceed by logarithm to whole of R by taking l i = log f (w i ), and compute either kernel or GMM density estimation g of the random variable log(f (X)). Finally, we obtain that the estimator of the realism index ri(z; f ) is given with the use of the cumulative density function of g: ri(z; f ) ≈ cdf g (log f (z))).

3. REALISM INDEX OF AN INTERPOLATION:

Our concept for the definition of the realism index for a path is inspired by transition between movie frames. The interpolation may be viewed as set of frames,where the first frame denotes the beginning of the path and last its end. Interpreting realism index as a probability that a given frame is realistic, we can define the respective index of the curve as the product of all its points.

Realism index for naturally parameterised curves. Let γ : [0, T ] → Z be an interpolating curve, such that γ(0) = x and γ(T ) = y,

for some given x, y ∈ Z. Additionally, we assume that the Gγ is naturally parameterised

EQUATION

We discretize the curve by fixing a time-step T /k (where k denotes the number of frames) and consider the sequence of intervals

[γ(0), γ(T /k)], . . . , [γ(T -T /k), γ(T )].

To obtain the reality measure of this sequence we compute the product of all realisticity values of its points to the power equal to their "duration"

EQUATION

where t i are the arbitrarily chosen intermediate points in the intervals [iT /k, (i + 1)T /k]. By taking the logarithm of the above expression and proceeding with k → ∞ we get

EQUATION

Consequently, we introduce the realism index of a naturally parameterised curve γ :

EQUATION

Since every curve can be uniquely naturally parameterised, we interpret its index as the index of its natural reparameterisation. Therefore we arrive at the following general definition. Definition 2. Let ri be given realism index in Z. For an arbitrary curve γ : [0, T ] → Z we define the realism index ri of γ with

EQUATION

We further prove that the realism index of a curve is equal to its length with respect to a certain Riemann structure on the latent space. We will utilise this result in the next section in order to connect the search of optimal interpolation to the search of geodesics. Directly from the definition we get the formula for the realism index in terms of the latent

-log ri(γ) = - T 0 log ri(Gγ(t)) • (Gα) (t) dt = - T 0 log ri(γ(t)) • (Gγ) (t), (Gγ) (t) dt = - T 0 log ri(γ(t)) γ (t) T [dG(γ(t))] T dG(γ(t))γ (t)dt = T 0 log 2 ri(γ(t))γ (t) T [dG(γ(t))] T dG(γ(t))γ (t)dt,

where dG(x) denotes the derivative of G at point x. Consequently, we obtain the following theorem: Theorem 1. Let the Riemann structure in the latent space Z be defined with the local scalar product , z at a point z using the following formula v, w z = v T A z w where A z = log 2 (ri(z))dG(z) T dG(z).

(13)

EQUATION

)

where length is the number of points in a path.

4. OPTIMAL INTERPOLATION:

Considering the results from the previous section, we are able to formulate the definition of an ri-optimal curve. Definition 3. Let ri be a realism index in Z for a generative model G : Z → R N . Let x, y ∈ Z be fixed. We call a curve γ : [0, T ] → Z such that γ(0) = x, γ(T ) = y ri-optimal (or, shortly, optimal) interpolation if it has the maximal realism index from all curves joining x with y.

Study first the issue of searching for, at least locally, optimal interpolations. Theorem 1 allows us to reformulate the problem as a task of finding geodesics. Consequently, the standard results from Riemann geometry apply (see Spivak (1999, Chapter 9)). Without loss of generality we can reduce the problem to the case T = 1 and optimise the length functional. However, due to the uniqueness of the local minima and local convexity of the functional, we can minimise the energy functional instead

EQUATION

)

The additional advantage is that curves which minimise the energy functional are parameterised proportionally to the natural parameterisation. Concluding, by applying Theorem 1, the optimal curve γ : [0, 1] → Z, γ(0) = x, γ(1) = y, with respect to the realism index minimises:

EQUATION

)

In general, the search for geodesics can lead to nontrivial computations involving second derivatives. However, for some special cases, we can significantly simplify the minimisation process. To justify this claim, we first introduce the formula for the discretization of the integral in the energy functional. Let γ : [0, 1] → R N be a curve such that

EQUATION

and divide the interval [0, 1] into k equal sub-intervals, and denote the values

EQUATION

For k -1 vectors x 1 , . . . , x k-1 in R N (where x 0 = x, x k = y), approximate the value of ( 16) with

2E y x (γ) ≈ k-1 i=0 log 2 ri(γ((i+1)/k))+ri(γ(i/k)) 2 • ( γ((i+1)/k)-γ(i/k) 1/k ) 2 • 1 k = k log 2 ri(x)+ri(x1) 2 x 1 -x 2 + k k-2 i=1 log 2 ri(xi)+ri(xi+1) 2 x i+1 -x i 2 + k log 2 ri(x k-1 )+ri(y) 2 y -x k-1 2 .

Considering all the above computations, to compute an optimal interpolation we need to minimise

2 k • E y x (x 1 , . . . , x k-1 ) = log 2 ri(x)+ri(x1) 2 x 1 -x 2 + k-2 i=1 log 2 ri(xi)+ri(xi+1) 2 x i+1 -x i 2 + log 2 ri(x k-1 )+ri(y) 2 y -x k-1 2 (19) over x 1 , . . . , x k-1 ∈ R N .

Optimisation procedure. In all the experiments in the paper we achieve this goal using the standard gradient descent method initialising x i with a linear interpolation x i = 1 -i k x+ i k y for i = 1, . . . , k -1. However, to accelerate the process of minimisation 2 , we alternate the gradient step with the following one: first choose two random numbers i < j from the set {0, . . . , k} and then consider the linear interpolation between x i and x j given with xl = j-l j-i x i + l-i j-i x j for l ∈ {i + 1. . . . , j -1}. Finally, if the linear interpolation has smaller energy then the original part, i.e. when E xj xi (x i+1 , . . . , xj-1 ) < E xj xi (x i+1 , . . . , x j-1 ), we replace x l by xl for l = {i + 1, . . . , j -1}.

Effects of ε-regularisation on optimisation. Given a reality index ri(•), see eq. ( 4), we have introduced its regularisation ri ε (z

) = ε+(1-2ε)•ri(z), ε ∈ [0, 1/2].

In practice, in all experiments we typically choose ε = 0.1. The reason behind such regularisation is twofold. First, if ε = 0 and ri is zero (or numerically close to zero) on some subset of the domain (for example if the density is zero), the optimisation for points with initial interpolating interval there is unmanageable (observe that in the realism of the curve we take the logarithm of the index). The case when ε = 0 and ri = 1 at some subset of the domain can also cause problems, as then logarithm of the reality index in this set is zero, and consequently the interpolating curve has no cost of staying or going through this region. Consequently, in our experience it is best to regularise ri by restricting its image to a subset of interval [ε, 1 -ε].

Let us now discuss the special limiting case when ε = 1/2. Then clearly

ri ε ≡ 1/2.

Let us first recall that we are given a decoder (generator) G : Z → R N , such that the data set lies in the manifold given by M = G(Z). Let us consider the interpolating curve γ : [0, 1] → Z. Observe that in that case the energy function does not depend on ri and equals

E y x (γ) = 1 2 1 0 log 2 (ri ε (γ(t))) dG(γ(t))γ (t) dt = 1 2 log 2 (2) 1 0 dG(γ(t))γ (t) dt = 1 2 log 2 (2) • length(G • γ).

Thus the minimal E y x (γ) value would be obtained for a curve joining x with y whose length measured in the input space is minimal. Thus if x = G(x) and ȳ = G(y) are points in the input space, then the interpolating curve with minimal energy is equal to one that connects points x and ȳ with a curve in the manifold M which has minimal length (measured in input space). Consequently, although the limiting case does not take into account the realism index of the interpolation, it still will usually produce interpolations comparable to the common linear. See Appendix D for additional analysis.

5. EXPERIMENTS:

In all the considered experiments we apply the regularised version or realism index ri ε with ε = 0.1. First we are going to consider the case when the index is based on the prior density in the latent, and next we briefly discuss the realism index based on the FID score.

Density based index.:

In this part we demonstrate our method's ability to produce more meaningful interpolations. In order to achieve this goal we use a DCGAN model (Radford et al., 2015) trained on MNIST and Celeb-A datasets (LeCun et al., 1998; Liu et al., 2015) . We consider a non-trivial latent created from a conjunction of three multidimensional Gaussian distributions

EQUATION

where µ i := (μ i , 0, . . . , 0) T ∈ R 20 for μ1 = (2, 6), μ2 = (0, 0), μ3 = (2, -6) and Σ i :=

M (i) 11 M12

M21 M22 for M 12 = M 21 = 0, M 22 is 2-D array with 0.5 on the diagonal and zeros elsewhere, and M

(1)

EQUATION

11 = 1 0 0 3 , M

(3) 11 = 5 -2 -2 2 . From here on we shall call it a semicircle, see Fig. 1 (leftmost subfigure). Results of sampling from a GAN trained with the semicircle prior are shown in Fig. 2 with respective projection shown in Fig. 1 . We randomly choose two points from different ends of the distribution and start the algorithm's minimisation procedure given with eq. ( 19) with a linear interpolation (green line in Fig. 1 ). It is obvious that the midpoints of initial path are sampled from a very lowdensity areas, completely outside the prior.

In Fig. 1 you can also the ri index values for the linear (center) and proposed (rightmost image) interpolations. Our method allows for more dynamic behaviour in this matter, especially at the end points. Images in the path obtained with proposed algorithm differ more, and at the same time seem more real. FID-score based realism index. In order to test our FID-score based realism index, we perform experiments on a GAN, trained on the Celeb-A dataset. Since samples from a high dimensional Gaussian approximately cluster on a sphere, points in the latent space that are very close to the origin are virtually never seen during training. Therefore, samples generated in that area may or may not be realistic. We selected a GAN model, for which we noticed that decoding latent codes near the origin gives low quality samples (the earlier mentioned soap-bubble effect), which makes for a perfect test environment for the FID-score based realism index. We aim to choose interpolation endpoints x and y in such a way, that the linear interpolation between them would pass close to the origin in the latent space. To that end, we first sample x uniformly, and then produce y from x by multiplying by -1 a random subset of dimensions.

The results of these experiments can be seen in Fig. 3 . We can see that the linear interpolations yield very low quality samples (especially in the middle of the interpolation). After optimising the linear paths using the FID-score realism index, the paths avoid unrealistic regions in the latent space, producing samples of much higher quality. Note that this experiment does not require access to the density. Instead, it is based only on the training samples.

Even though FID score has some limitations (Shmelkov et al., 2018) , observe that its application in our model results in more realistic interpolations, as compared to linear ones. Use of a measure more adapted to evaluating quality of data could result in further improvement in the realism of interpolations.

Further discussion. In this part we discuss when the linear approximations will be close to those given by linear. Let us first consider the case when the generator is linear with prior uniform distribution in the latent. Observation 1. Let U be a convex bounded set in Z and let f denote the uniform distribution on U . We consider the case of linear generative model, i.e. where G is a linear map x → Ax (with A injective). Then the ri-optimal interpolations are given by the linear interpolations.

Proof. Clearly, we can equivalently compute the realism index of a curve γ connecting two points x, y by computing the standard euclidean distance of Aγ in the convex set AU . More precisely: ri(γ; uni U ) = exp(-length(Aγ)). Since linear maps move intervals onto intervals, we obtain the assertion of the observation.

In practice, similar behaviour happens when the derivative of the generator has small variation and the realism index is close to being a constant one. Observe, that by equation ( 6), for a generative model with high dimension of the latent space and Gaussian prior, the realism index is almost constant on a linear interpolation of arbitrary randomly chosen points (except for a possibly small neighbourhood of the endpoints). Consequently, when the derivative of generator G does not vary too much in the vicinity of the linear interpolation, the linear interpolation will be close to optimal find by our approach. In practice, we observe this behaviour in auto-encoder based generative Figure 4 : Linear (top) and our interpolation (bottom) in the WAE (Tolstikhin et al., 2017) model. Observe that the consecutive elements of the interpolations are practically identical. models, such as Wasserstein Auto-Encoders, see Fig. 4 . This follows from the fact that too high variation (of derivative) of the decoder is penalised (regularised) by its approximate inverse given by the encoder.

6. CONCLUSIONS:

In this paper we studied the problem of generating a meaningful interpolation from a previously trained generative model, either a GAN or a generative auto-encoder. We claim that a good interpolation should both reveal the hidden structure of the dataset, as well as be smooth and follow the true data distribution, i.e. produce realistic elements.

In order to produce curves satisfying these conditions we define a realism index of a path, which takes into account both density values and and differences between consecutive decoded images to ensure smoothness. We show how to define realism index using either a known density or, in case it is either not available or not reliable, how to base it on some external measure, e.g. the FID score. We have proved that this interpolation procedure is equal to finding a geodesics with reality index equal to its length in respect to some latent space Riemann structure.

For the practical use, we have defined the notion of an optimal interpolation, and proposed a simple and efficient numerical procedure for its search. The experiments show that the constructed interpolations are in superior to the linear ones, making it possible to escape regions of low data density or low data quality, both for the density-and FID-based approaches. This is especially visible if the prior density is not Gaussian, when the linear interpolations often proceed through regions in space of extremely low density. Another example when the linear interpolations are suboptimal to our method is given by the standard GAN model when the soap bubble effect appears.

A CASE OF THE NORMAL DENSITY.:

We shall now compute the probability index of the standard normal density f = N (0, I) in the D-dimensional latent Z. Let us choose a point z from the latent and let X denote the random vector with density f . We want to compute the probability

EQUATION

Observe that from the definition of normal density we have

EQUATION

Since X 2 has the chi-square distribution with D degrees of freedom, we obtain that

EQUATION

where F (r; D) denotes the cumulative distribution function of the chi-square density χ 2 (D) (with D degrees of freedom). Let us now proceed with an asymptotic analysis. Observe that

EQUATION

If Y ∼ χ 2 (D) then for large D > 30, Johnson et al. (1994, formula (18.23 ) on p. 426)). Consequently

√ 2Y - √ 2D -1 is approximately normally distributed (see

EQUATION

)

where Φ denotes cdf of standard Gaussian. Since Φ

EQUATION

Remark 1. The above formula implies that for the normal density f = N (0, I) in R D the realism index is approximately 1 in the ball B(0, r D -3) and approximately 0 outside of the ball B(0, r D +

3), where r D = D -1 2 . This behaviour is natural and expected, as most of the points generated from normal density concentrate around the sphere S(0, r D ).

Let us now discuss why we use the normalisation given by eq. (4), and not the seemingly natural given for bounded f by the formula ri(z) = f (z) max f .

Consider the case when f = N (0, I) in R D . Then ri(z) = exp(-x 2 /2), which means that ri(0) = 1, but ri(z) = exp(-D/2 + 1/4) for z ∈ S(0, r D ). Since the randomly generated point from N (0, I) has norm close to r D , this implies that almost every point which comes from the normal density would have realism index close to 0, which would be of undesired and pathological behaviour.

In practice, we observe this behaviour in auto-encoder based generative models, such as Wasserstein Auto-Encoders, see Fig. 4 . This follows from the fact that too high variation (of derivative) of the decoder is penalised by its approximate inverse given by the encoder.

B DENSITY BASED REALISM INDEX IN GAN MODELS:

In this section we present additional results for the optimisation of the density based realism index of an interpolation curve in a GAN model. The experiments are conducted on the Celeb-A and MNIST datasets, using the DCGAN architecture. The optimisation procedure is initialised with a linear interpolation consisting of 50 points and implemented using Adam optimiser. To show the relation of the starting linear interpolation to the one obtained from the proposed procedure, see e.g. Fig. 1 , we perform a projection of k interpolation z i points onto point (x, y) ∈ R 2 such that

EQUATION

where zi := X(X T X) -1 X T z i are the latent space points z i ∈ Z and X = [z 0 , z k ]. 

C OPTIMISATION PROCEDURE:

In the performed experiments we use the optimisation described in Section 4 to reduce the number of necessary iterations in our method. In the initial phase of the optimisation of the linear interpolation, the value of the gradient for the middle points (located in the middle of the interpolation curve) can be extremely small due to the diminishing density. This is particularly evident for a DCGAN model with a semicircle prior, see Fig. 10 . In consequence, this part of the curve converges slower than the points lying closer to the ends of the curve. Our experiments for a semicircle distribution show that we obtain the same final result without using the described acceleration, but with up to 3 times more iterations. This optimisation procedure is crucial only for non typical densities; for the classical normal prior no significant impact on the number of iterations is observed.

D THE ε VALUE IMPACT ON THE INTERPOLATION CURVE:

In Fig. 11 we present squared L 2 distances between consecutive points in the optimised path from the last example from Fig. 2 . In this experiment we use different values of epsilon ε = 10 -5 , 10 -4 , 10 -3 , 0.01, 0.1, 0.25 for ri ε (see eq. ( 4)). As one can see, distances between consecutive points do not vary much for ε values much larger than zero.

Higher ε values result in more equally spaced interpolations, and the interpolation optimisation process is faster. However, higher ε result in puts a different emphasis on terms of the formula optimised (see eq. ( 19) and discussion on ε in Sec. 4), i.e. lengths start to be more important (hence the inter-point distances) than individual point realities.

On the other hand, lower ε values result in non-equally spaced points on the interpolation, which is easily visible in Fig. 11 , while the optimisation process is slower.

Empirically, we found ε = 0.1 to be optimal, and therefore it is used as default value. Experiments conducted on a Celeb-A dataset gave similar results. 4)).

:

In general, even if f is bounded, the obvious normalisation given by ri(x) = f (x)/ max f would not work, see Appendix A, Remark 1.

This step is essential when the initial interpolation passes through regions with density close to zero, which could result in a vanishing gradient problem, see Appendix C.

Figures and Tables:

Figure 1: (left to right) Interpolation (projected onto two selected dimensions) for a semicircle distribution: green -linear, grey -consecutive interpolation steps, red -final proposed interpolation; values of the proposed reality index ri for consecutive points in the linear interpolation; values of ri for points in the optimised interpolation.

Figure 2: Examples of results for linear interpolation (top row in each example) and our results (bottom row in each example) for Celeb-A dataset, with a semicircle prior trained DCGAN. Observe that since in the linear interpolation the middle point is far from the density of the data, we can often observe in it some artefacts.

Figure 3: Examples of results for linear interpolation (top rows of interpolation pairs) and our FIDscore based results (bottom rows of interpolation pairs) for Celeb-A dataset. The interpolation was optimised had 50 midpoints, and we show here 10 of them.

Figure 5: Interpolation points in DCGAN model trained with normal prior. In each pair the top represents the linear path and the bottom shows the results of our optimisation procedure. Each line consists of 10 equally spaced images selected from the 50 points that form the path.

Figure 8: Linear and proposed interpolation paths for the MNIST dataset (a semicircle latent prior model in left column, a multidimensional normal in right column) in a DCGAN model.

Figure 9: Comparison of interpolations in VAE (left) and WAE (right) models. The top line is the linear interpolation, the bottom one is the obtained with use of the density realism index approach.

Figure 10: Interpolations (projected defines by (27)) for a semicircle distribution: green -linear, grey -consecutive interpolation steps, red -final proposed interpolation. The picture on the left shows result of our method without 'Optimisation procedure', and right side with optimisation.

Figure11: The squared L 2 distances between consecutive points in the last example path from Fig.2for our interpolation and different values of ε for ri ε (see eq. (4)).

Peer review (iclr2020):
The paper proposes a density score, which is defined based on a density or some other score such as Fréchet Inception Distance (FID). While the given problem is an important one, there are some substantial problems with the proposed approach: 1) The paper does not mention the limitations of the FID which, as described in (Shmelkov et al, ECCV 2018. How good is my GAN?), are: i) it is based on the pre-trained Inception network and therefore does not exactly match the distributions over the data in other datasets; and ii) crude approximation of the scores by Gaussian distributions. Overall, this score is empirical and aims at circumventing the subjective analysis and it should be better reflected in the paper. 2) The justification of the normal density based index (Section 2) seems weak. While it is obvious that this score could be used, is it possible to make the empirical assessment? E.g. compare between two scores on an extensive amount of data. 3) In addition to the previous comment, there is a substantial problem in the experimental results that all the observations are qualitative and based only on a few images. Further analysis of the realism index on the real images which are not included into the training dataset could improve the analysis. 4) Following up on the previous comment, the experiments on the real images may need the values from the latent space corresponding to the real images. Currently, the model has been mostly assessed using DCGAN (with the assessment of VAE in Figure 4 but only for FID score), while it is stated after equation (1) that *This case covers both GAN-like and AE-like models.* This might be used for the assessment of the realism index on real images as stated before. If the proposed model were assessed with variational auto-encoders (VAE) and flow based models, it would make it possible to transform between the latent representation and the data themselves. From another perspective, experimental results on different types of models (VAE-type, flow based type such as GLOW) for different types of interpolation are needed for the sake of experimental completeness. It would help emphasise the limitations of the method and difference in interpolation results in different models. 4) In the optimisation section, the following statement is made: *However, to accelerate the process of minimisation, we alternate the gradient step with the following one: first choose two random numbers i < j from the set {0, . . . , k} and then consider the linear interpolation between xi and xj given with...*. Could the authors elaborate on why does this acceleration happen? It might be necessary to give some references on the experimental results or at least provide some line of support for this phrase. *** The following comments are not as critical but fall into the category of ‘nice to have*: 5) Although the reviewer is aware that there were some experiments in the appendix on the value epsilon, it might be a good idea to have more studies on the influence of this regularisation parameter for other datasets rather than just MNIST 6) While figure 1 appears in the beginning of the paper, on page two, it is discussed on page seven, in the experimental section. Placing the figures closer to the narrative would improve the reading experience.