FLAT MANIFOLD VAES
Authors: Nutan Chen
Alexej Klushyn
Francesco Ferroni
Justin Bayer
Patrick van der Smagt
Venue:  (2020)

Abstract: Latent-variable models represent observed data by mapping a prior distribution over some latent space to an observed space.  Often, the prior distribution is specified by the user to be very simple, effectively shifting the burden of a learning algorithm to the estimation of a highly non-linear likelihood function. This poses a problem for the calculation of a popular distance function, the geodesic between data points in the latent space, as this is often solved iteratively via numerical methods. These are less effective if the problem at hand is not well captured by first or second-order approximations. In this work, we propose less complex likelihood functions by allowing complex distributions and explicitly penalising the curvature of the decoder. This results in geodesics which are approximated well by the Euclidean distance in latent space, decreasing the runtime by a factor of 1,000 with little loss in accuracy. 


Under review as a conference paper at ICLR 2020
LEARNING FLAT LATENT MANIFOLDS WITH VAEs
Anonymous authors Paper under double-blind review
ABSTRACT
Latent-variable models represent observed data by mapping a prior distribution over some latent space to an observed space. Often, the prior distribution is specified by the user to be very simple, effectively shifting the burden of a learning algorithm to the estimation of a highly non-linear likelihood function. This poses problem for the calculation of a popular distance function- geodesic between data points in the latent space-aa this is often solved iteratively via numerical methods. These are less effective if the problem at hand is not well captured by first or second- order approximations. In this work, we propose less complex likelihood functions by allowing complex distributions and explicitly penalising the curvature of the decoder. This results in geodesics which are approximated well by the Euclidean distance in latent space, decreasing the runtime by factor of 1,000 with little loss in accuracy. Additionally, we apply our method to state-of-the-ee tracking algorithm using real world image data, showing that our unsupervised method performs similar to supervised learning methods.
INTRODUCTION
I. Latent-variable models (LVMs) are a viable tool in data analysis: a set of observations is explained by simpler set of latent variables in conjunction with a map from the latent space to the space of observation. Methods from this family (c.g.. principal component analysis (Wold et al., 1987). non-negative matrix factorisation (Lee & Seung 2001, generalised discriminant analysis (Baudat & Anouuar 2000). etc.) are standard tools, serving either as feature extractors for subsequent data processing pipelines, density estimators or dimensionality reducers for visualisation.
Despite the maturity of the field, research has far from halted. While kernel methods (KernelPCA (Schölkopf et al., 1997). KerneINMF (Li & Ding 2006). etc.) have been used to improve the applicability of LVMs to data inhibiting non-linear phenomena, neural formulations such as the variational autoencoder (VAE) (Kingma & Welling 2014: Rezende et al.) 2014) or the generative adversarial network (GAN) (Goodiellow et al.] 2014)) have become popular recently, especially due to their enormous success on modelling natural images. Here, a simple prior distribution (such as a multivariate standard normal means of a powerful deep neural network. A learning algorithm then finds weights for that neural network such that the data distribution is approximated well. In case of GANs, this is minimax game, while the evidence lower bound is maximised in the case of VAEs.
uniform distribution) is mapped to the space of observations by
If the data distribution is relatively complicated and the prior is relatively simple, the map from the latent to the observable space, the decoder, has to be sufficiently complex. In fact, it has to mimic the inverse CDF in parts. This results in highly non-linear neural networks. As an example, the separation of two modes in the probability landscape has to be implemented by flat CDF, which in turn requires an infinitely steep inverse CDF. Not only does this often pose difficulties for maudient-based learning. It also hinders the calculation of geodesics, the shortest paths from one point to another, as measured by the rate of change in the decoder along the path in latent space. Geodesics are often solved numerically, i.c. through a gradient-basse scenario, it is crucial to project the three-dimensional earth onto a two-dimensional map. When applying VAEs, the problem is summarised to map from two-dimensional Euclidean latent space (the map) to the three-dimensionl observation space (the earth surface). If, by construction, the decoder exhibits regions of high curvature, the stage is especially bad for such methods.
optimisation of first second order. In a cartography
We aim to improve the state of geodesics in deep latent-variable models with respect to runtime and the geodesic as a distance metric. We expect that short (approximate) geodesics under the learned

---

Under review as a conference paper at ICLR 2020
model indicate similarity of data points in question. If we assume that simple prior and simple decoder are insufficient to represent complex data distributions and a complex decoder is detrimental to the calculation of geodesics, we need the prior to be sufficiently complex to allow the decoder to be sufficiently simple. Our solution is the use of a powerful hierarchical prior representation in the context of VAFs and the simple penalisation of the curvature of the decoder. We name this approach flat manifold variational autoencoder since the Riemannian manifold of the decoder is isometric to Euclidean space. We show empirically that the resulting model features geodesics which are approximated very well by the Euclidean distance. This effectively removes the need for numerical optimisation, reducing the calculation of an approximate geodesic to that of a simple Euclidean distance calculation in latent space. This is accompanied by a speedup of several orders of magnitude, rendering the method practical for applications in real-time scenarios.
VARIATIONAL. AUTOENCODERS WITH RIEMANNIAN MANIFOLD REGULARISATION
2.1 BACKGROUND ON VAES WITH HIERARCHICAL PRIORS
Latent-variable models (LVMs) are defined as
p(x) - f p(x|z) p(z)
(1)
where z E R represents latent variables and x E R.N. the observable data. The integral in Eq. (1 usually intractable but it can be approximated by maximising the evidence lower bound (ELBO) (Kingma & Welling 2014; Rezende et al. (2014):
E px) [x 2 (x) (xs log po (x(22] KL (1s(z)x))
where (x) = N Σx= tion parameters of the approximate posterior go(z)x (x) and the likelihood p=(x)z) are represented by neural networks. The prior p(z) is usually defined as the standard normal distribution. This model is commonly referred to the variational autoencoder (VAE).
1 8(x- - x2) represents the empirical distribution of the data D. The distribu-
However, a standard normal prior often leads to an wwer-regularisttoon of the approximate posterior, which results in less informative learned latent representation of the data (Tomczak & Welling 2018
Klushyn et al. 2019). To enable the model to learn an informative latent representation, Klushyn
ct al. (2019) propose to use a flexible hierarchical prior pe (z) pe (z(() P()) dç, where p(() is the
standard normal distribution. Based on the insight that the optimal prior is the aggregated posterior
(Tomezak Welling 2018), the above integral approximated by an amportance-weigheee (IW)
bound (Burda et al. 2015) using samples from (z(2))) ()) This leads to model with two stochastic layers and the following upper bound on the Kullback-Leibe (KL) term:
(x)(x) KL (gxxz))) S F( θ 2)
po (2(()) P(G))
= Exp(()) Eg (x|x) lo/g)(z|x)
E mo
(S x) log K qo (G:1z) (z)
where K is the number of importance samples. Since has been shown that high ELBO values do not necessarily correlate with informative latent representations (Alemi et al., 2018 Higgins et al.) also the case for hierarchical models (Sønderby et al.) 2016) different optimisation approaches have been introduced (Bowman et al. 2016: Sønderby et al. 2016). Klushyn et al. (2019)
2017)
follow the line of argument in (Rezende & Viola 2018)) and reformulate the resulting ELBO as the Lagrangian of a constrained optimisation problem:
LVHP (0, o, O, A) = F(e. θ, Φ) + A(Epo(s) (xg() [Co z)] n²²).
(4)
with the optimisation objective F(ò, O Φ), the inequality constraint (x)()) (xix)) [C, z) <
x?? and the Lagrange multiplier x. Cx z) is defined as the coannsttection-ccorrrrrereee term in - log pa (x|z). Thus, we obtain the following optimisation problem:
min min max min LVHP(0, Φ, O, 2) s.t. A 2 0.
(5)
0, A
2

---

Under review as a conference paper at ICLR 2020
Building that, the authors propose an optimisation algorithm iggoritmm-includinnga X-update scheme- to achieve a tight lower bound on the log likelihood. This approach referred to as variational hierarchical prior VHP-VAE.
2.2 BACKGROUND ON RIEMANNIAN GEOMETRY FOR VAES
A manifold M is space which is differentiable and locally Euclidean. Given M, Riemannian manifold is (M, G), where G E R. xN, is the Riemannian metric tensor. At each point z E M in the latent space, the corresponding metric tensor G defines an inner product in the tangent space z' E T.M:
(z' 1,,2'xx = x G(z)z'.
Chen et al. (0188); Arvanitidis et al. (2018) define the latent space of a VAE as a Riemannian manifold, which allows for computing the observation space distance based on distances in the latent space. Given a smooth trajectory [0, 1] - NR in the Riemannian (latent) space and the corresponding Ny-dimensional Euclidean (observation) space, the Riemaninan distance of the curve can be written as
S (tt) dt, [(7) -
((t) = (1(0) = V
(7)
where G depends on y, @(t) denotes the Riemannian velocity and (t)) represents the time-derivative of the trajectory. In the VAE models, 7 is transformed by continuous function (()(()) (decoder) to x, and the metric tensor is defined as G J"J, where J is the Jacobian of the decoder. The geodesic obtained by minimising L(r).
The magnification factor (MF = /ot Gzz) (Bishop al. 1997) shows the sensitivity of the likelihood functions. When projecting from the Riemannian (latent) to the Euclidean (observation) space, the MF can considered scaling coefficient.
2.3 FLAT MANIFOLD VAE
In this work, we aim to compute geodesics directly in the latent space by measuring the Euclidean distance between encoded data points. For this purpose, the metric tensor, which describes our latent space, needs be G henee a Euclidean metric. This simplifies the computation of geodesics (Eq. (7)
L(7) - f (t) dt x |z((1) - z(0)]l-
(8)
We refer to a manifold M with this property as flat manifold (Lee) 2006). As a consequence, our model must be capable of learning such flat manifold latent spaces, which typically requires complex latent representations of the data (see experiments in Sec. 4. Therefore, we propose the following approach: (i) to enable prior, which learned by the model (empirical Bayes); and (ii) we penalise the curvature of the decoder such that G x 1.
model to learn complex latent representations, we introduce a flexible
For this purpose, we extend the VHP-VAL introduced in Sec. 2.1 by a Jacobian Jacobian-regularissinn regularisatuion term. Defining the regularisation term part of the constraint is in line with the constrained optimisation setting. The resulting objective function is
L = Flb O b) + X( (x)(x) [Co 2) + 31) 3
21 -
where B is a
determining the influence of the regularisation. c² is defined to be the mean over the batch samples and diagonal elements of J J. which we view as a normalisation process. Additionally, we use a stochastic approximation (first order Taylor expansion) of the Jacobian (Rifai et al. 2b) improve the computational efficiency:
J ((z)= = lim E Ep Egg (a x) [f +) f(()),
(10)
o-0
3

---

Under review as a conference paper at ICLR 2020
where <~ 00) This approximation method allows for faster computation of the gradient and avoids the reconddderivative problem of piece-wise linear layers (Chen et al. 2018a) during optimisation.
However, the regularisation term in Eq. ( only effects the decoder function in regions where data is available. To overcome this issue, we propose use mixup, data-augmentation method (Zhang let al. 2018), which was introduced in the context of supervised learning. We extend this method to the VAE framework (unsupervised learning) by applying aim is to augment data by interpolating between two encoded data points z; and Zj:
encoded data in the latent space. Our
Gang (Zi; 2,) = (1 a) 24 + 02 Z.j.
(11)
with X, (x), Z₁ (g(2)xx), Zj 4s (z|x,), and U(-ao, + an). In contrast to (Zhang et al. 2018), where E [0,1] limits the data augmentation to only convex combinations,
define cro to take into account the outer edge of the data manifold. We obtain the objective
function of our flat manifold VAE (FMVAE) by combining mixup (Eq.
(TI) with Eq. (
LYHP-PMVAE = LVHP + X3 Ex,xx-pp((x) Exingo (1))s))))))))))))))) (a, (x|xx)
G (gang (Ze, 2,)) - c²1 21
(12)
By using augmented data when minimising |G - c²1 we regularise G to be a scaled identity matrix for the entire latent space enclosed by our data manifold. Hence, our VAE learns scaled Euclidean latent space, where is the scale factor. Therefore, the function f(z) (decoder) us-up to the scale factor isometry/distaacce preeeervingee since (f f(z)) c (z zj), where D refers to the distance between two data points in the observation and latent space, respectively.
The decoder of the proposed approach satisfies the Lipschitz continuity condition. Given the Lipschitz continuity condition Dx (z,),, f(z)) S a Dz(Z; 2j), where a is the Lipschitz constant, we consider the decoder function, and hence the latent space as smooth if 3c S a.
RELATED WORK
Latent space of VAEs. In general, the latent space of VAEs is considered be Euclidean (e.g. Kingma et al. 2016 Higgins et al. (017), but they are not constrained to be Euclidean. This can be problematic if a precise metric is required in the latent space. Some recent works (Mathieu et al.) 2019. Grattarola et al. 2018) adapted the latent space to be non-Fuclidean to match the data structure. We solve the problem from another perspective by enforcing the latent space to be Euclidean.
Jacobian and Hessian regularisation. In (Rifai et al. 2011a). the authors propose to regularise the Jacobian and Hessian of the encoder. However, the encoders of VAEs are already regularised by the KL term. Furthermore, it is more difficult augment data in the observation space than in the latent space for data augmentation. Encoder regularisation enables the model to perform better in case of, e.g., object recognition from the latent space. By contrast, decoder regularisation enables the model to do tasks such as generating motions based on the latent space. In (Hadjeres et al., (017), the Jacobian of the decoder was regularised be as small as possible/zero. By contrast, we regularise the Jacobian to be constand, and hence the Hessian to be zero leading to correct metric in the latent space. Nie & Patel (201)) regularised the Jacobian with respect the weights of both the encoder and decoder for GANs. In terms of supervised learning, Jakubovitz & Giryes (2018) regularised the Jacobian to improve the robustness for classification.
Metric learning. Various metric learning approaches for both deep supervised and unsupervised models were proposed. For instance, deep metric learning (Hoffer & Ailon 2013) used a triplet network for supervised learning. |Karlletoos et al. (2016) introduced an unsupervised metric learning method, where a VAE is combined with triplets. However, a human oracle is still required. By contrast, approach is completely based on unsupervised learning, using the tangent space of the decoder as a distance metric. Our proposed method is similar to the metric learning methods such as Large Margin Nearest Neighbor (Weinberger & Saul 2009), which pulls target neighbours together and pushes imposters away, but our approach is an unsupervised method.
Constraints in latent space. Constraints time (e.g. Wang et al. 2007; Chen et al. 2016 2015) allow to obtain similar distance metrics in the latent space. However, our method can be used for general datasets without sequential data. Additionally, constraints time cannot guarantee that
4

---

Under review as a conference paper at ICLR 2020
the metric is correct in between of different time steps. By contrast, in case of sequential data, method can be used to obtain a correct metric through data augmentation.
Data augmentation. In regions without training data, the latent space is trained arbitrarily. Our method is able to augment data in the latent space, so that we can smoothly interpolate between two points even in case the data in between is missing in the training dataset using mixup. Various follow-up studies of mixup were developed, such as (Verma et al. 2018 [eckhha et al. 2019). GANs which generates fake data in the latent space a similar approach as our approach.
Geodesic. There have been some recent studies on geodesics for generative models using both
stochastic methods (e.g- [Tosi al., 2014 Arvanitidis et al., 2018 Hauberg 2018 Chen al. 2018b) and deterministic approaches (c.g.. Chen et al. 201 Sa 2019). The main difference is that the stochastic methods work for the regions without data, because the RBF layer generates high MF for those-however, it is less general. The uncertainty does not emerge from principled way (such as Bayesian model) but is instead driven by certain assumptions. The deterministic method requires other strategies to guarantee that the geodesic is within the data manifold. In our proposed method, we regularise the latent space to have a similar metric, so that we do not need to consider low density regions. In previous work, methods were introduced for computing/inding the geodesic in the latent space. However, is a novel upproach to use the geodesic/Riemanniaa distance for influencing the latent representation. Tenenbaum et al. (2000) projected the latent space new latent space where the geodesic is equivalent to the Euclidean interpolation. However, the two separate processes- VAFs and projection-probably cannot allow the model find the latent features autonomously.
EXPERIMENTS
We test method artificial pendulum images, human motion, the MNIST and the MOTI6 datasets. We measure the performance in terms of equidistances, interpolation smoothness and geodesics. Additionally, our method applied to real-world environment-a tracking system from the context of autonomous driving. Consequently, the tracking and re-identification capabilities are evaluated.
Riemannian metric tensor has many intrinsic properties of a manifold and measures local angles, length, surface area and volumes (Bronstein al. 2017). Therefore, the models are quantified using the Riemannian metric tensor- conditionn numbers and MFs. The condition number which shows the ratio of the most elongated to the least elongated direction is defined as iGG) Sme. (G), where (..
S an eigenvalue G. Since we cannot directly compare the MFs of different models, the MFs normalised through dividing by their mean. Accordingly, out from their mean. The model is more invariant with respect to the metric tensor the condition number is smaller and the normalised MF closer to one.
measure how the MFs spread
4.1 PENDULUM IMAGE DATASET
20
3000
3.0
magninaa factr log scale)
300
magmificatoo factor log scale
10
rotaton angle
1.5
3.5
4
200
200
2.0
-10
2.8
1010
-5
-00
-00
20
-5
5
21
R1
(a) Latent representation of VHP-FMVAE.
(b) Latent representation of VHP-VAE.
Figure 1: Equidistance the latent space of the pendulum dataset. The black curves are points of equal distance to center. The distance is computed using
5

---

Under review as a conference paper at ICLR 2020
200
4.5
6000
condition number
5000
normalised MF
1000
normalised
condition
100
2000
7009
1000
+
Vie PAriE
VWe UAC
vio RMNAE
wee ViE
vie FINNEE
WH VAE
vne neve
wp VME
methods
methods
methods
methods
(a) Pendulum.
(b) MNIST.
Figure Boxplot of the condition number and the normalised MF pendulum and MNIST datasets.
The pendulum and MNIST have 10,000 and 1,000 samples generated in from the latent space, respectively.
The pendulum dataset (Klushyn et al. 2019 Chen et al. 2018a) consists of 16 x 16-pixel images generated by pendulum simulator. We generated 15 10 images with joint angle the ranges of [0, 3600 degrees. Additionally, we added 0.05 Gaussian noise to each pixel to avoid overfitting.
Fig. shows the equidistance plots for five different encoded data points. VHP-FMVAE smoothens the MF, while VHP- has large area of high MF in the middle. Without regularisation, the contour of the equidistances are significant different from high MF areas low MF areas. Fig. 2a shows that the condition number of the VHP-FMVAL is smaller than that of the VAE- VHP in terms of the Riemannian tensor. Additionally, the normalised MF of the VHP-FMVAE is closer one.
In order to avoid bias visualisation in Fig. and we reset the range of the MF for plot- ting while not changing the MF values. In Fig. la
3a and 7 the upper range set to be max (grid, area)) :eean (data))) MF, and MF2 refer to the MF of VHP-FMVAE with/without mean Mel' (data))
Jacobian normalisation/mixup and VAE- VHP, respectively. MF (data)) and MF (grid_aara)) the
MF of the training data and the MF of the grid area, respectively.
4.2 HUMAN MOTION
To evaluate approach, CMU human motion dataset (http::/mocaa) cs emu eduu is used. Walking (subject 35), jogging (subject 35), balancing (subject 49), punching (subject 143) and kicking (subject 74) selected for the experiment. After data pre-procsssing, the input data is 50-dimensional vector the joint angles. The dataset is soallancdd-wlkinng dataset size is larger than that of jogging.
Table The length ratio of Euclidean interpolation geodesic. The Riemannian distance and the distance in the latent space are computed. We randomly sample 100 pairs of points and interpolate between each pair. The mean and the standard deviation of the ratios are listed below.
DATASET
METHOD
RIEMANNIAN
LATENT
HUMAN
1.02 1 0.06
0.93 1 0.03
VHP-VAE
1.23 + 0.20
0.82 + 0.
MNIST
VHP-FMVAB
1.01 + 0.08
0.92 + 0.05
VHP-VAE
1.13 10.22
0.70 + 0.31
Equidistance. We randomly select point from each class as the centre of the equidistance. As shown Fig. 33, the proposed method has more similar equidistance at different locations in the latent space, while in the model without regularisation, the equidistance contour distorted in the high MF

---

Under review as a conference paper at ICLR 2020
equidistance
geodesic walking
balancing jogging
punching kicking
Euclidean interpolation
magnification factor [log scale
magnificattin factor [log scale]
20
10
S.
5
3 2
-10
-20
(a) Latent representation of VHP-FMVAE.
(b) Latent representation of VAE-VHP.
Figure 3: Equidistance in the latent space of the human motion dataset. (a) Jogging a large-range movement compared with walking, so that jogging is reasonably distributed on a larger area in the latent space than that of walking. (b) In contrast, without regularisation, walking is larger than the jogging in the latent space. For FMVAE, the Euclidean interpolatioins are much closer to the geodesics.
Jeft light log head and torso left arm rright arm
condition number
narmalised MF
smoothness factor
VHP-FMVAE VHP-VAE
3x1
0.2
0.1
Vie PMNAA
Wish MAC
UH FuariE
cemottatio cottattoon
wre adlag
VAPURARRE
Whe Tanaa
Veen FRURE
10
20
30
40
50
joint index
methods
methods
Figure 5: Smoothness of the human dataset. The mean and standard deviation are shown. The smaller the value is, the smoother the model is.
(a) Condition number.
(b) Normalised MF.
Figure 4: Boxplot of the condition number and the normalised MF of human motion dataset.
(a) Linear interpolation of VHP-FMVAF.
(b) Linear interpolation VHP-VAF.
Figure 6: Generated movements of the human motion dataset. The abrupt motions marked by
blue boxes.
area. The latent space of VHP-FMVAE reflects the true distribution of the data (see more details in Fig. B. Moreover, compute the Riemannian tensors for 3,000 samples randomly generated from the latent space. different locations a latent space, the Riemannian tensors of VHP-FMVAE more invariant than that of VAE-VHP (see Fig. 4)

---

Under review as a conference paper at ICLR 2020
10
20
20
-20 -0
0
20
21
21
-5
magnification factor Plg scale]
7
magnification factor [log scale]
(a) Latent representation of VHP-FMVAE with- mixup. In the which has training data,
(b) Latent representation of VHP-FMVAE without the Jaco- hian normalisation. Although it does not have extreme sharp equidistance contour, the equidistance is still scaled in various locations of the latent space. Additionally, the distribution of walking in the latent space still larger than that of jogging.
the equidistance contour is smooth. However, the area of data missing, e.g.. between two move- ments, the MF' is high and the equidistances are distorted.
Figure 7: Influence of the data augmentation and the Jacobian normalisation. The movements are coloured the same as Fig.
Smoothness. We randomly sample 100 pair points and linearly interpolate between each pair. The second derivative of each trajectory is defined as the smoothness factor. Fig. 5 illlsttratss that VHP- FMVAE significantly outperforms the latent space of VAE-VHP terms of the smoothness. Fig. shows five examples of the interpolated trajectories.
Geodesic. We compare the proposed method with the graph-based geodesic approach (Chen et al. 2019) which approximates the geodesic using graph in generative model. The graph-based approach is much faster than previous geodesic search method such as (Chen et al. 2018a). The graph of the baseline has 14,400 nodes which are sampled in the latent space using uniform distribution. Each node has 12 neighbours. The regularisation of singular value decomposition (SVD) (Chen et al., 2018a) is 0.001 for VAE-VHP while it is adapted to VHP-FMVAE based the mean value of the square of the singular.
Table shows the ratios from Euclidean interpolations geodesics. If the ratio of the distance is close to one, the Euclidean interpolation is able to approximate the geodesic. Table demonstrates that the Euclidean interpolation of VHP-FMVAE is more close to geodesic, compared with VAE-VHP. Additionally. the proposed method is 1,000 times faster than the graph-based method in terms of searching for the geodesics. Fig. depicts five examples of the geodesics.
Influence of the data augmentation and the Jacobian normalisation. Fig. 7a shows the influence of the data augmentation. The samples of the regularisation term are the same as LVIP- Fig. 7b illustrates the influence of the Jacobian normalisation. We removed the normalisation, and conse- quently the regularisation term [| 2,))).. The c²1 term the regularisation necessary; otherwise it only has dissimilarity constraints, but cannot reduce the distance for points with high similarities. For instance, the walking is not squeezed in the latent space (see Fig. 4.. By contrast, regularising the Jacobian to be constant elongates the distance in the latent space with high MF areas while squeezing the distance with low MF areas.
4.3 MNIST
A fixed binarised version of the MNIST digit dataset (Larochelle & Murray 2011 is used to evaluate our approach. The dataset consists of 50,000 training and 10,000 test images of handwritten digits (zero to nine) with 28 x 28 pixels in size.
The equidistances of VHP-FMVAE are more invariant and smoother than that of VAE VHP (see Fig. 8 and Fig. (4) Similar the human motion dataset, the geodesic of our method is more similar to Euclidean interpolation, compared with VAE-VHP, which indicates that the latent space of the VHP-FMVAE is able to approximate geodesic (see Table
8

---

Under review as a conference paper at ICLR 2020
100
302
-00
(a) Latent representation of VHP-FMVAB.
(b) Latent representation of VHE-VAE.
Figure 8: Equidistance the latent space of MNIST dataset. (b) The data ranges on 21 and Zg of the VHP-VAE [-106.21, 369.38] and [-365.64, 164.08), respectively. For better visualisation, crop out the less dense areas.
4.4 TRACKING
We evaluate our approach on the MOTI6 object-tracking database (Milan et al. 2016), which is large-scale person re-identification dataset, containing both static and dynamic scenes from diverse cameras.
Table 2: Comparisons between different descriptors for the purposes of object tracking and re- identification (Ristani et al. 2016). The bold and the red numbers denote the best results among all methods and among non-supervised methods, respectively.
METHOD
TYPE
IDFit
IDPt
IDR
RECALL
PRECISIONT
FAR
MT1
SHIP-FMVAE-OE SORT
3 300 (OURS)
UNSUPERVISED
63.7
77.0
54.3
65.0
92.3
1.12
158
VHP-FMVAE-SORE
UNSUPERVISED
64.2
77.6
54.8
65.1
92.3
1.13
162
3000 (OURS)
VHP- VAP-SOT-O
UNSUPERVISED
60.5
72.3
52.1
65.8
91.4
1.28
170
SORT
N.A.
57.0
67.4
49.4
66.4
90.6
1.44
158
DEEPSORT
SUPERVISED
64.7
76.9
55.8
66.7
91.9
1.22
180
МETHOD
PT4
MLJ
FPI
FN4
IDs|
FMJ
MOTA
MOTP1
MOTAL
VHIP- HIP--MMAE-SORT
269
90
5950
38592
616
1143
59.1
81.8
59.7
300 (OURS)
VHP-FMVA FMVAE-SORE
265
90
6026
38515
598
1163
59.1
81.8
59.7
3000 (OURS)
-AP-SRT-SORT
266
81
6820
37739
693
1264
59.0
81.6
59.6
SORT
275
84
7643
37071
1486
1515
58.2
81.9
59.5
DEEPSORT
250
87
6.506
36747
585
1165
60.3
81.6
60.8
We compare with baselines: SORT (Bewley al. 2016)) and DeepSORT (Wojke al.) (017). SORT is a simple online and realtime tracking method, which uses bounding box intersection-over union (IOU) for associating detections between frames and Kalman filters for the track predictions. relies on good two-dimensional bounding box detections from separate detector, and suffers from ID switching when tracks overlap in the image. DeepSORT extends the original SORT algorithm to integrate appearance information based on deep appearance descriptor, which helps with re- identification in the case such overlaps or missed detections. The deep appearance descriptor is trained using supervised cosine metric learning approach (Wojke & Bewley 2018). The candidate object locations the pre-generated detections for both SORT, DeepSORT and our method are taken from Yu et al. 2016). Further details regarding the implementation be found in App. A.3

---

Under review as a conference paper at ICLR 2020
3421
(a) SORT.
(b) DeepSORT.
61
(c) VHP-VAE-SORT.
(d) VHP-FMVAE-SORT with , = 3000.
Figure 9: Example identity switches between overlapping tracks. For vanilla SORT, track 3260 gets occluded and when subsequently visible, it gets assigned new ID 3421. For deeSORT and "HP-VAE-SORT, the occluding track gets assigned the same ID as the track occludes (42/61). and subsequently keeps this (erroneous) track. For VHP-FMVAE-SORT, the track 42 gets occluded, but re-identified correctly when again visible.
We use the following metries for evaluation. indicates that the higher the score is, the better the performance is. On the contrary, + indicates that the lower the score is, the better the performance is.
IDF; (1); ID F1 Score
FN(L): False Negatives
IDP(T): ID Precision
IDs(4): Number of times an ID switches to
IDR(1):: ID Recall
different previously tracked object
:FAR()): False Alarm Ratio
FM(J): Fragmentations
MT(T):: Mostly Tracked Trajectory
MOTA(1): Multi-object tracking accuracy
PT(J): Partially Tracked Trajectory
MOTP(T): Multi-object tracking precision
ML()): Mostly Lost Trajectory
-MOTAL()): Log tracking accuracy
FP((): False Positives
Table shows that the performance of the proposed method is better than that of the model without Jacobian regularisation, and even close to the the performance of supervised learning. All methods depend on the same underlying detector for object candidates, and identical Kalmann filter parameters. Compared to baseline SORT which does not utilise any appearance information, DeepSORT has 2.54 times, VHP-VAE-SORT has 2.14 times, VHP-FMVAE-SORT (B 300) has 2.41 times and VHP- FMVAE-SORT (B 3000) has 2.48 times fewer ID switches. Whilst the supervised DeepSORT descriptor has the least, using unsupervised VAES with flat decoders has only 2.2% more switches, without the need for labels. Furthermore, by ensuring a quasi-Euclinn Euclidean latent space, one can query neirest-neighborrs efficiently via data-structures such as kDTrees. Fig. shows an example of the results. In other examples of the videos, the VHP-FMVAE-SOWE works similar as the deepSORT. Videos the results can be downloaded at: httt:://ttttyt bt.. ww/
CONCLUSION
In this paper, we have proposed novel approach, which we call flat manifold variational autoencoder. We have shown that-using this method-geodesie can be computed directly in the latent space by measuring the Euclidean distance between encoded data points. This realised by combining a powerful empirical Bayes prior with a Jacobian-regularistt latent space to be Euclidean. Consequently, geodesic can be approximated 1,000 times faster than comparable state-of of-th-art methods. Furthermore. using the approximated geodesic distance function, we have evaluated our approach on the MOT16 object-tracking database showing comparable performance in case of supervised learning.
method that constrains the learned
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Alexander A Alemi, Ben Poole, Ian Fischer, Joshua V Dillon, Rif A Saurous, and Kevin Murphy. Fixing broken ELBO. ICML, 2018.
Georgios Arvanitidis, Lars Kai Hansen, and Søren Hauberg. Latent space oddity: on the curvature of deep generative models. In ICLR, 2018.
Gaston Baudat and Fatiha Anouar. Generalized discriminant analysis using kernel approach. Neural computation, 12(10):2385-2400 2000.
Christopher Beckham, Sina Honari, Alex M Lamb, Vikas Verma, Farnoosh Ghadiri, R Devon Hjelm, Yoshua Bengio, and Christopher Pal. On adversarial mixup resynthesis. NeurlPS, 2019.
Alex Bewley, Zongyuan Ge, Lionel Ott, Fabio Ramos, and Ben Upcroft. Simple online and realtime tracking. In 2016 IEEE International Conference on Image Processing (ICIP), pp. 3464-3468. IEEE, 2016.
Christopher M Bishop, Markus Svens' and Christopher KI Williams. Magnification factors for the SOM and GTM algorithms. In Proceedings Workshop on Self- Orlaanrrgnizing Maps, 1997.
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from continuous space. CoNLL, 2016.
Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geometric deep learning: going beyond Euclidean data. IEEE Signal Processing Magazine, 34(4):18-42, 2017.
Yuri Burda, Roger B. Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. CoRR, 0bs/1509.005/ 19, 2015.
Nutan Chen, Justin Bayer, Sebastian Urban, and Patrick Van Der Smagt. Efficient movement representation by embedding dynamic movement primitives deep autoencoders. 2015 IEEE- RAS 15th International Conference on Humanoid Robots (Humanoids), pp. 434-440, IEEE, 2015.
Nutan Chen, Maximilian Karl, and Patrick van der Smagt. Dynamic movement primitives in latent space of time-dependent variational autoencoders. In 2016 IEEE-RAS 16th International Conference Humanoid Robots (Humanoids), pp. 629-636, 2016.
Nutan Chen, Alexej Klushyn, Richard Kurle, Xueyan Jiang, Justin Bayer, and Patrick van der Smagt. Metrics for deep generative models. AISTATS, pp. 1550-1550, 2018a.
Nutan Chen, Alexej Klushyn, Alexandros Paraschos, Djalel Benbouzid, and Patrick van der Smagt. Active learning based on data uncertainty and model sensitivity. IEEE/RS/ IROS, 2018b.
Nutan Chen, Francesco Ferroni, Alexej Klushyn, Alexandros Paraschos, Justin Bayer, and Patrick van der Smagt. Fast approximate geodesics for deep generative models. In ICANN, 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NIPS, pp. 2672-2680, 2014.
Daniele Grattarola, Daniele Zambon, Cesare Alippi, and Lorenzo Livi. Learning graph embed- dings on constant-curvature manifolds for change detection in graph streams. arXiv preprint arXiv:1805.06290 06299, 2018.
Gaëtan Hadjeres, Frank Nielsen, and François Pachet. GL.R--VA. geodesic latent space regulariza- tion for variational autoencoder architectures. In 2017 IEEE Symposium Series on Computational Intelligence (SSCI), pp. 2017.
Søren Hauberg. Only bayes should learn a manifold (on the estimation of differential geometric structure from data). arXiv, 2018
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. Beta- VAE: Learning basic visual concepts with a constrained variational framework. ICLR, 2017.

---

Under review as a conference paper at ICLR 2020
Elad Hoffer and Nir Ailon. Deep metric learning using triplet network. In International Workshop Similarity- Baseed Pattern Recognition, pp. 84-92. Springer, 2015.
Daniel Jakubovitz and Raja Giryes. Improving DNN robustness to adversarial attacks using Jacobian regularization. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 514-529, 2018.
Theofanis Karaletsos, Serge Belongie, and Gunnar Rätsch. Bayesian representation learning with oracle constraints. ICLR, 2016.
Diederik Kingma and Max Welling. Auto-cncoding variational Bayes. ICLR, 2014.
Diederik P. Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improving Variational Inference with Inverse Autoregressive Flow. NIPS, 2016.
Alexej Klushyn, Nutan Chen, Richard Kurle, Botond Cseke, and Patrick van der Smagt. Learning hierarchical priors in VAEs. NeurlPS, 2019.
Hugo Larochelle and lain Murray. The neural autoregressive distribution estimator. In International Conference on Artificial Intelligence and Statistics, pp. 29-37, 2011.
Daniel D Lee and H Sebastian Seung. Algorithms for non-negative matrix factorization. In Advances in neural information processing systems, pp. 556 562, 2001.
John M Lee. Riemannian manifolds: an introduction curvature, volume 176. Springer Science & Business Media, 2006.
Tao Li and Chris Ding. The relationships among various nonnegative matrix factorization methods for clustering. In International Conference on Data Mining, pp. 362-371. IEEE, 2006.
Emile Mathieu, Charline Le Lan, Chris J Maddison, Ryota Tomioka, and Yee Whye Teh. Hierarchical representations with Poincar e variational auto-encoders. arXiv preprint arXiv:1901.06033 2019.
Anton Milan, Laura Leal- Ian Reid, Stefan Roth, and Konrad Schindler. Mot16: A benchmark for multi-object tracking. arXiv preprint 1603.000 2016.
Weili Nie and Ankit Patel. Towards a better understanding and regularization of GAN training dynamics. In UAI, 2019.
Danilo Jimenez Rezende and Fabio Viola. Taming VAEs. arXiv preprint arXiv: 2018.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In ICML, volume 32, pp. 1278- 1278-1286, 2014.
Salah Rifai, Yann N Dauphin, Pascal Vincent, Yoshua Bengio, and Xavier Muller. The manifold tangent classifier. In Advances in Neural Information Processing Systems, pp. 2294-2302, 201 la.
Salah Rifai, Grégoire Mesnil, Pascal Vincent, Xavier Muller, Yoshua Bengio, Yann Dauphin, and Xavier Glorot. Higher order contractive auto-encoder. FCMMI-PDDD, pp. 645-660. Springer, 201
Ergys Ristani, Francesco Solera, Roger S. Zou, Rita Cucchiara, and Carlo Tomasi. Performance measures and a data set for multi-target, multi multi-camera tracking. CoRR, abs/ abbs/160900175. 2016.
Bernhard Schölkopf, Alexander Smola, and Klaus-Robert Müller. Kernel principal component analysis. In International conference on artificial neural networks, pp. 583-588. Springer. 1997.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. ICLR, 2015.
Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. NIPS, 2016.
Joshua Tenenbaum, Vin De Silva, and John C Langford. A global geometric framework for nonlinear dimensionality reduction. science, 29005500):2319-2222 2000.
12

---

Under review as a conference paper at ICLR 2020
Jakub M. Tomezak and Max Welling. VAE with vampprior. In International Conference Artificial Intelligence and Statistics, pp. 2214-223, 2018.
Alessandra Tosi, Søren Hauberg, Alfredo Vellido, and Neil D. Lawrence. Metrics for probabilistic geometries. In UAI, pp. 800-808, 2014.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, loannis Mitliagkas, Aaron Courville, David Lopez-Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. JCML, 2018.
Jack M Wang, David Fleet, and Aaron Hertzmann. Gaussian process dynamical models for human motion. IEEE transactions pattern analysis and machine intelligence, 30(2):283-295 2007.
Kilian Q Weinberger and Lawrence K Saul. Distance metric learning for large margin nearest neighbor classification. Journal of Machine Learning Research, 10(Feb):207-244, 2009,
Nicolai Wojke and Alex Bewley. Deep cosine metric learning for person re- re-identification.. In IEEE
Winter Conference on Applications of Computer Vision (WACV). pp- 748-756, 2018.
Nicolai Wojke, Alex Bewley, and Dietrich Paulus. Simple online and realtime tracking with a deep association metric. In IEEE International Conference Image Processing, pp. 3645-364) 2017.
Svante Wold, Kim Eshensen, and Paul Geladi. Principal component analysis. Chemometrics and intelligent laboratory systems, 2(1-3):37-52, 1987.
Fengwei Yu, Wenbo Li, Quanquan Li, Yu Liu. Xiaohua Shi, and Junjie Yan. POI: multiple object tracking with high performance detection and appearance feature. CoRR, abs/1610.06 0.066100,6,... 2016.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. International Conference Learning Representations, 2018.
13

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1 VECTOR FIELD
30
10
(a) VHP-FMVAE.
(b) VHP-VAL
Figure 10: Vector field of the human motion dataset. The vector field is vector of L2 the output Jacobian. The figures are corresponding to Fig. The vector field of VHP-FMVAE more regular than that of VAE-VHP.
A.2 MODEL ARCHITECTURES
Table 3: Model architectures. FC refers fully-connected layers. Conv2D and Conv2DT denote tow-D convolution layer and transposed two-D convolution layer, respectively. See the definition of in (Klushyn 2019).
DATASET
OPTIMISER
ARCHITECTURE
PENDULUM
ADAM
INPUT
16x
1e-4
LATENTS
[x(2|x)
FC 256, 256. RELU ACTIVATION.
pa(x(z)
256, 256. RELU ACTIVATION. GAUSSIAN.
(<(z))
FC 256, 256, RELU ACTIVATION.
pe (z
FC 256, 256, RELU ACTIVATION.
OTHERS
0.025, = K =1,6, 3 1000.
CMU HUMAN ADAM le-4
INPUT LATENTS
50
(z(x))
FC 256, 256, 256, 256. RELU ACTIVATION.
po(x
FC 256, 256, 256, 256. RELU ACTIVATION. GAUSSIAN.
2o (C(z))
FC 256, 256, 256, 256, RELU ACTIVATION.
Pa (2
FC 256, 256, 256, 256, RELU ACTIVATION.
OTHERS
k=0. v= 1, 32, 8000.
MNIST
ADAM 1e-4
INPUT LATENTS
28x28x1
{x(x)x)
FC 256, 256, 256, 256. RELU ACTIVATION.
px(x z)
FC 256, 256, 256, 256. RELU ACTIVATION, BERNOULLL
@o (<iz) pe (2
FC 256. 256, 256. 256. RELU ACTIVATION.
FC6, 256. 256. 256. RELU ACTIVATION.
OTHERS
= /
16, 8 8000.
MOTI6
ADAM 3e-5
INPUT LATENTS
64x64x3
128
(x(2)x)
VGG16 (SIMONYAN & ZISSERMAN
2015
Po (x(z)
CONV2D1+CONE
256, 128, 64, 32, T6.
RELU ACTIVATION. GAUSSIAN. FC 512, 512. RELU ACTIVATION. FC 512, 512. RELU ACTIVATION.
go (CIz)
Pa (z){)
OTHERS
K = 0.8 == 1, K 8, B = 300 or 3000.
14

---

Under review as a conference paper at ICLR 2020
A.3 IMPLEMENTATION OF VHP- W-A.-M--EEOOOO -SORT
We evaluate the performance of our model by replacing the appearance descriptor from DeepSORT with the latent space embedding from the various auto-encoders used, using the same size of 128. The hyperparameters used were held constant: the minimum detection confidence of 0.3, NMS max overlap of 0.7, max cosine distance 0.2, max appearance budget 100. We tested VHP-FMVAE, and our regularised VHP-FMVAE with B 300 and 3 = 3000.
15


Reviews:
[{"review_id": "rJxI645i_r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Notes: \n\n  -This paper suggests the use of VAEs with stronger priors along with more powerful regularization of the decoder (especially its curvature).  This lowering of curvature is seen as \"flattening\".  \n\n  -Paper uses the normal VAE ELBO.  \n\n  -The normal prior over-regularizes the approximate posterior.  One proposal is to use a \"hierarchical prior\": integral(p(z|zeta)*p(zeta), zeta) where zeta is a normal distribution.  So basically a function can transform the prior.  \n\n  -Importance weighting with q(z|x) has been proposed as a way to define a valid learning objective for this setting.  \n\n  -Another objective using lagrangian is called \"VHP-VAE\".  \n\n  -This paper extends VHP-VAE with jacobian regularization, which is approximated (the paper doesn't say so but I think it's a first order taylor expansion).  \n\n  -Paper also uses mixup in the latent space to provide regularization at points farther from the data.  \n\n  -With this mixup objective the mixing is also done to consider extrapolations in addition to interpolations.  \n\n  -The resulting latent space does indeed look much better (Figure 1).  \n\n  -The condition number is also way better (2a, 2b).  \n\n  -In figure 2, the background color indicates the degree of magnification (so the VAE-VHP has greater variability in distances?)  I found this figure a bit hard to itnerpret.  \n\nComments: \n\n  -This paper cites Mixup but there are two more papers to consider here: Manifold Mixup (ICML 2019) and Adversarial Mixup Resynthesis (Neurips 2019) which both considered mixing in a latent space.  AMR considered in an autoencoder, and Manifold Mixup is also relevant because its theoretical analysis explicitly considers flattening although in a somewhat difference sense (and both are different from what's done here).  \n\n  -The object tracking experiments don't seem very convincing to me (just looking at table 2 at least).  \n\nReview: \n\n  This paper considers augmenting the hierarchical VHP-VAE with a criteria in which the jacobian is approximately regularized at interpolations and extrapolations between different points in z space.  The experiments suggest this is an important problem with VHP-VAE and also that it's successfully addressed.  ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "SJeN-ZNl5r", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1.\tThe idea of explicitly forcing the encoding space to be flat by putting constraint on metric tensor is simple but neat.\n2.\tThe use of Jacobi regularization in Eq. (9) is effective but the choice of using interpolation to extend this in the entire decoding space is kind of adhoc. Can authors please justify?\n3.\tNot sure how authors put the Lipschitz continuity constraint on f. Please explain. \n4.\tThe title of “Flat manifold VAEs” is misleading as it potentially means VAEs for flat manifold \n5.\tI wonder what will happen if you put an unfolding constraint in the encoding space like LLE, ISOMAP etc.. The loss function is data driven so this should give atleast similar behavior.\n6.\tOverall I like the experimental setup, but the tracking experiment is kind of distracting. The authors may want to remove this experiment.\n7.\tIn Fig. 7, the authors have shown with and without  Jacobi normalization which I am really not convinced with, need better explanation.  \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJlXO5WwYr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary of paper:\nThe paper is concerned with the geometry of latent spaces in VAEs. In particular, it is argued that since geodesics (shortest paths) in the Riemannian interpretation of latent spaces are expensive to compute, then it might be beneficial to regularize the decoder (generator) to be flat, such that geodesics are straight lines. One such regularization is proposed.\n\nReview:\nI have several concerns with the paper:\n\n1) Geodesics are never motivated:\nThe paper provides no motivation for why geodesics are interesting objects in the first place, so it is not clear to me what the authors are even trying to approximate.\n\n2) Under the usual motivation, the work is flawed:\nThe usual motivation for geodesics is that they should follow the trend of the data (e.g. go through regions of high density). Since no other motivation is provided, I will assume this to be the motivation of the paper as well. The paper propose to use a flexible prior and then approximate geodesics by straight lines. Beyond the most simple linear models, then this cannot work. If the prior is flexible, then straight lines will hardly ever constitute paths through regions of high density. The core idea of the work, thus, seem to be in conflict with itself.\n\n3) A substantial bias is ignored:\nThe paper consider the Riemannian metric associated with the *mean* decoder. Due to regularization, holes in the data manifold will be smoothly interpolated by the mean decoder, such that geodesics under the associated metric will systematically be attracted to holes in the data manifold. Hauberg discuss this issue in great length here:\n\n  https://arxiv.org/abs/1806.04994\n\nHere it is also demonstrated that geodesics under the mean decoder tend to be straight lines (which is also what the authors observe). Taking the stochasticity of the VAE decoder into account drastically change the behavior of geodesics to naturally follow the trend of the data.\n\n4) Related work is mischaracterized:\nPrevious work on the geometry of latent spaces largely fall into two categories: those that treat the decoder as deterministic and those that treat it as being stochastic. In the cited papers Arvanitidis et al and Tosi et al consider stochastic decoders, while the other consider deterministic decoders. Given that geodesics have significantly different behavior in the two cases, it is odd that the difference is never discussed in the paper.\n\n5) It is not clear to me what the experiments actually show:\n\n-- I did not understand the sentence (page 5): \"The model is more invariant if the condition number is smaller...\" What does it mean to be \"more invariant\" ? And how is invariance (to what) related to the condition number of the metric?\n\n-- Figure 3 show example geodesics, but only geodesics going between clusters (I have no idea how such geodesics should look). If I look at the yellow cluster of Fig3a, then it seems clear  to me that geodesics really should be circular arcs, yet this is being approximated with straight lines. Are the ground truth geodesics circular? At the end, it seems like the shown examples are the least informative ones, and that intra-cluster geodesics would carry much more meaning.\n\n-- What am I supposed to learn from the \"Smoothness\" experiment (page 7) ? My only take-away is currently that the proposed regularization does what it is asked to do. It is not clear to me if what it aims to do is desirable? Does the experiment shed light on the desirability of the regularizer or is it more of a \"unit test\" that show that the regularizer is correctly implemented?\n\n-- In the \"Geodesic\" experiment (page 7) I don't agree with the choice of baseline. If I understand correctly, the baseline approximate geodesics with shortest paths over the neighbor graph (akin to Isomap). However, there is no reason to believe that the resulting paths bare any resemblance to geodesics under the studied Riemannian metric. The above-mentioned paper by Hauberg provide significant evidence that these baseline geodesics are not at all related to the actual geodesics of the studied metric. The only sensible baseline I can think of is the expensive optimization-based geodesics.\n\n== rebuttal ==\nI have read the rebuttal and discussed with the authors, and I retain my original score.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to regularize the decoder of the VAE to have a flat pull-back metric, with the goal of making Euclidean distances in the latent space correspond to geodesic distances. This, in turn, results in faster geodesic distance computation. I share the concern of R2 that this regularization towards a flat metric could result in "biased" geodesic distances in regions where data is scarce. I suggest the authors discuss in the next version of the paper if there are situations where this regularization might have drawbacks and if possible, conduct experiments (perhaps on toy data) to either rule out or highlight these points, particularly about scarce data regions. 
Decision: Reject