Year: 2019

Abstract:
Latent-variable models represent observed data by mapping a prior distribution over some latent space to an observed space. Often, the prior distribution is specified by the user to be very simple, effectively shifting the burden of a learning algorithm to the estimation of a highly non-linear likelihood function. This poses a problem for the calculation of a popular distance function—the geodesic between data points in the latent space—as this is often solved iteratively via numerical methods. These are less effective if the problem at hand is not well captured by first or secondorder approximations. In this work, we propose less complex likelihood functions by allowing complex distributions and explicitly penalising the curvature of the decoder. This results in geodesics which are approximated well by the Euclidean distance in latent space, decreasing the runtime by a factor of 1,000 with little loss in accuracy. Additionally, we apply our method to a state-of-the-art tracking algorithm using real world image data, showing that our unsupervised method performs similar to supervised learning methods.

heading: 1 INTRODUCTION
text: Latent-variable models (LVMs) are a viable tool in data analysis: a set of observations is explained by a simpler set of latent variables in conjunction with a map from the latent space to the space of observation. Methods from this family (e.g., principal component analysis (Wold et al., 1987), non-negative matrix factorisation (Lee & Seung, 2001), generalised discriminant analysis (Baudat & Anouar, 2000), etc.) are standard tools, serving either as feature extractors for subsequent data processing pipelines, density estimators or dimensionality reducers for visualisation.
Despite the maturity of the field, research has far from halted. While kernel methods (KernelPCA (Schölkopf et al., 1997), KernelNMF (Li & Ding, 2006), etc.) have been used to improve the applicability of LVMs to data inhibiting non-linear phenomena, neural formulations such as the variational autoencoder (VAE) (Kingma & Welling, 2014; Rezende et al., 2014) or the generative adversarial network (GAN) (Goodfellow et al., 2014) have become popular recently, especially due to their enormous success on modelling natural images. Here, a simple prior distribution (such as a multivariate standard normal or a uniform distribution) is mapped to the space of observations by means of a powerful deep neural network. A learning algorithm then finds weights for that neural network such that the data distribution is approximated well. In case of GANs, this is a minimax game, while the evidence lower bound is maximised in the case of VAEs.
If the data distribution is relatively complicated and the prior is relatively simple, the map from the latent to the observable space, the decoder, has to be sufficiently complex. In fact, it has to mimic the inverse CDF in parts. This results in highly non-linear neural networks. As an example, the separation of two modes in the probability landscape has to be implemented by a flat CDF, which in turn requires an infinitely steep inverse CDF. Not only does this often pose difficulties for gradient-based learning. It also hinders the calculation of geodesics, the shortest paths from one point to another, as measured by the rate of change in the decoder along the path in latent space. Geodesics are often solved numerically, i.e. through a gradient-based optimisation of first or second order. In a cartography scenario, it is crucial to project the three-dimensional earth onto a two-dimensional map. When applying VAEs, the problem is summarised to map from a two-dimensional Euclidean latent space (the map) to the three-dimensional observation space (the earth surface). If, by construction, the decoder exhibits regions of high curvature, the stage is especially bad for such methods.
We aim to improve the state of geodesics in deep latent-variable models with respect to runtime and use the geodesic as a distance metric. We expect that short (approximate) geodesics under the learned
model indicate similarity of data points in question. If we assume that a simple prior and a simple decoder are insufficient to represent complex data distributions and a complex decoder is detrimental to the calculation of geodesics, we need the prior to be sufficiently complex to allow the decoder to be sufficiently simple. Our solution is the use of a powerful hierarchical prior representation in the context of VAEs and the simple penalisation of the curvature of the decoder. We name this approach flat manifold variational autoencoder since the Riemannian manifold of the decoder is isometric to Euclidean space. We show empirically that the resulting model features geodesics which are approximated very well by the Euclidean distance. This effectively removes the need for numerical optimisation, reducing the calculation of an approximate geodesic to that of a simple Euclidean distance calculation in latent space. This is accompanied by a speedup of several orders of magnitude, rendering the method practical for applications in real-time scenarios.

heading: 2 VARIATIONAL AUTOENCODERS WITH RIEMANNIAN MANIFOLD REGULARISATION

heading: 2.1 BACKGROUND ON VAES WITH HIERARCHICAL PRIORS
text: Latent-variable models (LVMs) are defined as
p(x) = ∫ p(x|z) p(z) dz, (1)
where z ∈ RNz represents latent variables and x ∈ RNx the observable data. The integral in Eq. (1) is usually intractable but it can be approximated by maximising the evidence lower bound (ELBO) (Kingma & Welling, 2014; Rezende et al., 2014):
EpD(x) [ log pθ(x) ] ≥ EpD(x) [ Eqφ(z|x) [ log pθ(x|z) ] −KL ( qφ(z|x)‖ p(z) )] , (2)
where pD(x) = 1N ∑N i=1 δ(x− xi) represents the empirical distribution of the data D. The distribution parameters of the approximate posterior qφ(z|x) and the likelihood pθ(x|z) are represented by neural networks. The prior p(z) is usually defined as the standard normal distribution. This model is commonly referred to the variational autoencoder (VAE).
However, a standard normal prior often leads to an over-regularisation of the approximate posterior, which results in a less informative learned latent representation of the data (Tomczak & Welling, 2018; Klushyn et al., 2019). To enable the model to learn an informative latent representation, Klushyn et al. (2019) propose to use a flexible hierarchical prior pΘ(z) = ∫ pΘ(z|ζ) p(ζ) dζ , where p(ζ) is the standard normal distribution. Based on the insight that the optimal prior is the aggregated posterior (Tomczak & Welling, 2018), the above integral is approximated by an importance-weighted (IW) bound (Burda et al., 2015) using samples from qφ(z|x). This leads to a model with two stochastic layers and the following upper bound on the Kullback-Leibler (KL) term:
EpD(x) KL ( qφ(z|x)‖ p(z) ) ≤ F(φ,Θ,Φ)
≡ EpD(x) Eqφ(z|x) [ log qφ(z|x)− Eζ1:K∼qΦ(ζ|z) [ log 1
K K∑ i=1 pΘ(z|ζi) p(ζi) qΦ(ζi|z) ]] , (3)
where K is the number of importance samples. Since it has been shown that high ELBO values do not necessarily correlate with informative latent representations (Alemi et al., 2018; Higgins et al., 2017)—which is also the case for hierarchical models (Sønderby et al., 2016)—different optimisation approaches have been introduced (Bowman et al., 2016; Sønderby et al., 2016). Klushyn et al. (2019) follow the line of argument in (Rezende & Viola, 2018) and reformulate the resulting ELBO as the Lagrangian of a constrained optimisation problem:
LVHP(θ, φ,Θ,Φ;λ) ≡ F(φ,Θ,Φ) + λ ( EpD(x) Eqφ(z|x) [ Cθ(x, z) ] − κ2 ) , (4)
with the optimisation objective F(φ,Θ,Φ), the inequality constraint EpD(x) Eqφ(z|x) [ Cθ(x, z) ] ≤ κ2, and the Lagrange multiplier λ. Cθ(x, z) is defined as the reconstruction-error-related term in − log pθ(x|z). Thus, we obtain the following optimisation problem:
min Θ,Φ min θ max λ min φ LVHP(θ, φ,Θ,Φ;λ) s.t. λ ≥ 0. (5)
Building on that, the authors propose an optimisation algorithm—including a λ-update scheme— to achieve a tight lower bound on the log likelihood. This approach is referred to as variational hierarchical prior VHP-VAE.

heading: 2.2 BACKGROUND ON RIEMANNIAN GEOMETRY FOR VAES
text: A manifold M is a space which is differentiable and locally Euclidean. Given M , a Riemannian manifold is (M,G), where G ∈ RNz×Nz is the Riemannian metric tensor. At each point z ∈M in the latent space, the corresponding metric tensor G defines an inner product in the tangent space z′ ∈ TzM :
〈z′, z′〉z ≡ z′T G(z) z′. (6)
Chen et al. (2018a); Arvanitidis et al. (2018) define the latent space of a VAE as a Riemannian manifold, which allows for computing the observation space distance based on distances in the latent space. Given a smooth trajectory γ : [0, 1]→ RNz in the Riemannian (latent) space and the corresponding Nx-dimensional Euclidean (observation) space, the Riemaninan distance of the curve can be written as
L(γ) = ∫ 1 0 φ(t) dt, φ(t) ≡ √〈 γ′(t), γ′(t) 〉 γ(t) = √ γ′(t)TG(γ(t))γ′(t) , (7)
where G depends on γ, φ(t) denotes the Riemannian velocity and γ′(t) represents the time-derivative of the trajectory. In the VAE models, γ is transformed by a continuous function f(γ(t)) (decoder) to x, and the metric tensor is defined as G = JTJ, where J is the Jacobian of the decoder. The geodesic is obtained by minimising L(γ).
The magnification factor (MF(z) ≡ √
detG(z)) (Bishop et al., 1997) shows the sensitivity of the likelihood functions. When projecting from the Riemannian (latent) to the Euclidean (observation) space, the MF can be considered a scaling coefficient.

heading: 2.3 FLAT MANIFOLD VAE
text: In this work, we aim to compute geodesics directly in the latent space by measuring the Euclidean distance between encoded data points. For this purpose, the metric tensor, which describes our latent space, needs to be G ∝ 1—hence a Euclidean metric. This simplifies the computation of geodesics (Eq. (7)) to
L(γ) = ∫ 1 0 φ(t) dt ∝ ‖z(1)− z(0)‖. (8)
We refer to a manifold M with this property as flat manifold (Lee, 2006). As a consequence, our model must be capable of learning such flat manifold latent spaces, which typically requires complex latent representations of the data (see experiments in Sec. 4). Therefore, we propose the following approach: (i) to enable our model to learn complex latent representations, we introduce a flexible prior, which is learned by the model (empirical Bayes); and (ii) we penalise the curvature of the decoder such that G ∝ 1. For this purpose, we extend the VHP-VAE introduced in Sec. 2.1 by a Jacobian-regularisation term. Defining the regularisation term as part of the constraint is in line with the constrained optimisation setting. The resulting objective function is
L = F(φ,Θ,Φ) + λ ( EpD(x) Eqφ(z|x) [ Cθ(x, z) + β ∥∥J(z)TJ(z)− c21∥∥]− κ2), (9) where β is a hyper-parameter determining the influence of the regularisation. c2 is defined to be the mean over the batch samples and diagonal elements of JTJ, which we view as a normalisation process. Additionally, we use a stochastic approximation (first order Taylor expansion) of the Jacobian (Rifai et al., 2011b) to improve the computational efficiency:
J(z) = lim σ→0
1 σ EpD(x) Eqφ(z|x)[f(z + )− f(z)], (10)
where ∼ N (0, σ2I). This approximation method allows for a faster computation of the gradient and avoids the second-derivative problem of piece-wise linear layers (Chen et al., 2018a) during optimisation.
However, the regularisation term in Eq. (9) only effects the decoder function in regions where data is available. To overcome this issue, we propose to use mixup, a data-augmentation method (Zhang et al., 2018), which was introduced in the context of supervised learning. We extend this method to the VAE framework (unsupervised learning) by applying it to encoded data in the latent space. Our aim is to augment data by interpolating between two encoded data points zi and zj :
gaug(zi, zj) = (1− α) zi + α zj , (11) with xi,xj ∼ pD(x), zi ∼ qφ(z|xi), zj ∼ qφ(z|xj), and α ∼ U(−α0, 1 + α0). In contrast to (Zhang et al., 2018), where α ∈ [0, 1] limits the data augmentation to only convex combinations, we define α0 > 0 to take into account the outer edge of the data manifold. We obtain the objective function of our flat manifold VAE (FMVAE) by combining mixup (Eq. (11)) with Eq. (9):
LVHP-FMVAE = LVHP + λβ Exi,xj∼pD(x) Ezi∼qφ(z|xi),zj∼qφ(z|xj) [∥∥G(gaug(zi, zj))− c21∥∥].
(12)
By using augmented data when minimising ‖G − c21‖, we regularise G to be a scaled identity matrix for the entire latent space enclosed by our data manifold. Hence, our VAE learns a scaled Euclidean latent space, where c is the scale factor. Therefore, the function f(z) (decoder) is—up to the scale factor c—isometry/distance-preserving since Dx(f(zi), f(zj)) ≈ cDz(zi, zj), where D refers to the distance between two data points in the observation and latent space, respectively.
The decoder of the proposed approach satisfies the Lipschitz continuity condition. Given the Lipschitz continuity conditionDx(f(zi), f(zj)) ≤ aDz(zi, zj), where a is the Lipschitz constant, we consider the decoder function, and hence the latent space as smooth if ∃ c ≤ a.

heading: 3 RELATED WORK
text: Latent space of VAEs. In general, the latent space of VAEs is considered to be Euclidean (e.g. Kingma et al., 2016; Higgins et al., 2017), but they are not constrained to be Euclidean. This can be problematic if a precise metric is required in the latent space. Some recent works (Mathieu et al., 2019; Grattarola et al., 2018) adapted the latent space to be non-Euclidean to match the data structure. We solve the problem from another perspective by enforcing the latent space to be Euclidean.
Jacobian and Hessian regularisation. In (Rifai et al., 2011a), the authors propose to regularise the Jacobian and Hessian of the encoder. However, the encoders of VAEs are already regularised by the KL term. Furthermore, it is more difficult augment data in the observation space than in the latent space for data augmentation. Encoder regularisation enables the model to perform better in case of, e.g., object recognition from the latent space. By contrast, decoder regularisation enables the model to do tasks such as generating motions based on the latent space. In (Hadjeres et al., 2017), the Jacobian of the decoder was regularised to be as small as possible/zero. By contrast, we regularise the Jacobian to be constand, and hence the Hessian to be zero leading to a correct metric in the latent space. Nie & Patel (2019) regularised the Jacobian with respect to the weights of both the encoder and decoder for GANs. In terms of supervised learning, Jakubovitz & Giryes (2018) regularised the Jacobian to improve the robustness for classification.
Metric learning. Various metric learning approaches for both deep supervised and unsupervised models were proposed. For instance, deep metric learning (Hoffer & Ailon, 2015) used a triplet network for supervised learning. Karaletsos et al. (2016) introduced an unsupervised metric learning method, where a VAE is combined with triplets. However, a human oracle is still required. By contrast, our approach is completely based on unsupervised learning, using the tangent space of the decoder as a distance metric. Our proposed method is similar to the metric learning methods such as Large Margin Nearest Neighbor (Weinberger & Saul, 2009), which pulls target neighbours together and pushes imposters away, but our approach is an unsupervised method.
Constraints in latent space. Constraints on time (e.g. Wang et al., 2007; Chen et al., 2016; 2015) allow to obtain similar distance metrics in the latent space. However, our method can be used for general datasets without sequential data. Additionally, constraints on time cannot guarantee that
the metric is correct in between of different time steps. By contrast, in case of sequential data, our method can be used to obtain a correct metric through data augmentation.
Data augmentation. In regions without training data, the latent space is trained arbitrarily. Our method is able to augment data in the latent space, so that we can smoothly interpolate between two points even in case the data in between is missing in the training dataset using mixup. Various follow-up studies of mixup were developed, such as (Verma et al., 2018; Beckham et al., 2019). GANs which generates fake data in the latent space are a similar approach as our approach.
Geodesic. There have been some recent studies on geodesics for generative models using both stochastic methods (e.g. Tosi et al., 2014; Arvanitidis et al., 2018; Hauberg, 2018; Chen et al., 2018b) and deterministic approaches (e.g., Chen et al., 2018a; 2019). The main difference is that the stochastic methods work for the regions without data, because the RBF layer generates high MF for those—however, it is less general. The uncertainty does not emerge from a principled way (such as in a Bayesian model) but is instead driven by certain assumptions. The deterministic method requires other strategies to guarantee that the geodesic is within the data manifold. In our proposed method, we regularise the latent space to have a similar metric, so that we do not need to consider low density regions. In previous work, methods were introduced for computing/finding the geodesic in the latent space. However, it is a novel approach to use the geodesic/Riemannian distance for influencing the latent representation. Tenenbaum et al. (2000) projected the latent space to a new latent space where the geodesic is equivalent to the Euclidean interpolation. However, the two separate processes—VAEs and projection—probably cannot allow the model to find the latent features autonomously.

heading: 4 EXPERIMENTS
text: We test our method on artificial pendulum images, human motion, the MNIST and the MOT16 datasets. We measure the performance in terms of equidistances, interpolation smoothness and geodesics. Additionally, our method is applied to a real-world environment—a tracking system from the context of autonomous driving. Consequently, the tracking and re-identification capabilities are evaluated.
Riemannian metric tensor has many intrinsic properties of a manifold and measures local angles, length, surface area and volumes (Bronstein et al., 2017). Therefore, the models are quantified using the Riemannian metric tensor—condition numbers and MFs. The condition number which shows the ratio of the most elongated to the least elongated direction is defined as k(G) = Smax(G)Smin(G) , where S is an eigenvalue of G. Since we cannot directly compare the MFs of different models, the MFs are normalised through dividing by their mean. Accordingly, we can measure how the MFs spread out from their mean. The model is more invariant with respect to the metric tensor if the condition number is smaller and the normalised MF is closer to one.
4.1 PENDULUM IMAGE DATASET
The pendulum dataset (Klushyn et al., 2019; Chen et al., 2018a) consists of 16× 16-pixel images generated by a pendulum simulator. We generated 15 · 103 images with joint angle in the ranges of [0, 360) degrees. Additionally, we added 0.05 Gaussian noise to each pixel to avoid overfitting.
Fig. 1 shows the equidistance plots for five different encoded data points. VHP-FMVAE smoothens the MF, while VHP-VAE has large area of high MF in the middle. Without regularisation, the contour of the equidistances are significant different from high MF areas to low MF areas. Fig. 2a shows that the condition number of the VHP-FMVAE is smaller than that of the VAE-VHP in terms of the Riemannian tensor. Additionally, the normalised MF of the VHP-FMVAE is closer to one.
In order to avoid bias visualisation in Fig. 1, 3 and 7, we reset the range of the MF for plotting while not changing the MF values. In Fig. 1a, Fig. 3a and 7, the upper range is set to be max(MF2(grid_area))·mean(MF1(data))
mean(MF2(data)) . MF1 and MF2 refer to the MF of VHP-FMVAE with/without
Jacobian normalisation/mixup and VAE-VHP, respectively. MF(data) and MF(grid_area) are the MF of the training data and the MF of the grid area, respectively.

heading: 4.2 HUMAN MOTION
text: To evaluate our approach, CMU human motion dataset (http://mocap.cs.cmu.edu) is used. Walking (subject 35), jogging (subject 35), balancing (subject 49), punching (subject 143) and kicking (subject 74) are selected for the experiment. After data pre-processing, the input data is a 50-dimensional vector of the joint angles. The dataset is unbalanced—walking dataset size is larger than that of jogging.
Equidistance. We randomly select a point from each class as the centre of the equidistance. As shown in Fig. 3, the proposed method has more similar equidistance at different locations in the latent space, while in the model without regularisation, the equidistance contour is distorted in the high MF
methods
(a) Condition number.
methods
(b) Normalised MF.
Figure 4: Boxplot of the condition number and the normalised MF of human motion dataset.
(a) Linear interpolation of VHP-FMVAE. (b) Linear interpolation of VHP-VAE.
Figure 6: Generated movements of the human motion dataset. The abrupt motions are marked by blue boxes.
area. The latent space of VHP-FMVAE reflects the true distribution of the data (see more details in Fig. 3). Moreover, we compute the Riemannian tensors for 3,000 samples randomly generated from the latent space. In different locations of a latent space, the Riemannian tensors of VHP-FMVAE is more invariant than that of VAE-VHP (see Fig. 4).
−20 0 z1
−10
0
10 z 2
−5 −4 −3 −2 −1 magnification factor [log scale]
(a) Latent representation of VHP-FMVAE without mixup. In the area which has training data, the equidistance contour is smooth. However, in the area of data missing, e.g., between two movements, the MF is high and the equidistances are distorted.
20 0 20 z1
20
0
20
z 2
7 6 5 4 3 2 magnification factor [log scale]
(b) Latent representation of VHP-FMVAE without the Jacobian normalisation. Although it does not have extreme sharp equidistance contour, the equidistance is still scaled in various locations of the latent space. Additionally, the distribution of walking in the latent space is still larger than that of jogging.
Figure 7: Influence of the data augmentation and the Jacobian normalisation. The movements are coloured the same as Fig. 3.
Smoothness. We randomly sample 100 pair points and linearly interpolate between each pair. The second derivative of each trajectory is defined as the smoothness factor. Fig. 5 illustrates that VHPFMVAE significantly outperforms the latent space of VAE-VHP in terms of the smoothness. Fig. 6 shows five examples of the interpolated trajectories.
Geodesic. We compare the proposed method with the graph-based geodesic approach (Chen et al., 2019) which approximates the geodesic using a graph in a generative model. The graph-based approach is much faster than previous geodesic search method such as (Chen et al., 2018a). The graph of the baseline has 14,400 nodes which are sampled in the latent space using uniform distribution. Each node has 12 neighbours. The regularisation of singular value decomposition (SVD) (Chen et al., 2018a) is 0.001 for VAE-VHP while it is adapted to VHP-FMVAE based the mean value of the square of the singular.
Table 1 shows the ratios from Euclidean interpolations to geodesics. If the ratio of the distance is close to one, the Euclidean interpolation is able to approximate the geodesic. Table 1 demonstrates that the Euclidean interpolation of VHP-FMVAE is more close to geodesic, compared with VAE-VHP. Additionally, the proposed method is 1,000 times faster than the graph-based method in terms of searching for the geodesics. Fig. 3 depicts five examples of the geodesics.
Influence of the data augmentation and the Jacobian normalisation. Fig. 7a shows the influence of the data augmentation. The samples of the regularisation term are the same as LVHP. Fig. 7b illustrates the influence of the Jacobian normalisation. We removed the normalisation, and consequently the regularisation term is ‖G(gaug(zi, zj))‖. The c21 term in the regularisation is necessary; otherwise it only has dissimilarity constraints, but cannot reduce the distance for points with high similarities. For instance, the walking is not squeezed in the latent space (see Fig. 4). By contrast, regularising the Jacobian to be constant elongates the distance in the latent space with high MF areas while squeezing the distance with low MF areas.

heading: 4.3 MNIST
text: A fixed binarised version of the MNIST digit dataset (Larochelle & Murray, 2011) is used to evaluate our approach. The dataset consists of 50,000 training and 10,000 test images of handwritten digits (zero to nine) with 28× 28 pixels in size. The equidistances of VHP-FMVAE are more invariant and smoother than that of VAE-VHP (see Fig. 8 and Fig. 4). Similar as the human motion dataset, the geodesic of our method is more similar to Euclidean interpolation, compared with VAE-VHP, which indicates that the latent space of the VHP-FMVAE is able to approximate geodesic (see Table 1).

heading: 4.4 TRACKING
text: We evaluate our approach on the MOT16 object-tracking database (Milan et al., 2016), which is a large-scale person re-identification dataset, containing both static and dynamic scenes from diverse cameras.
We compare with two baselines: SORT (Bewley et al., 2016) and DeepSORT (Wojke et al., 2017). SORT is a simple online and realtime tracking method, which uses bounding box intersection-overunion (IOU) for associating detections between frames and Kalman filters for the track predictions. It relies on good two-dimensional bounding box detections from a separate detector, and suffers from ID switching when tracks overlap in the image. DeepSORT extends the original SORT algorithm to integrate appearance information based on a deep appearance descriptor, which helps with reidentification in the case of such overlaps or missed detections. The deep appearance descriptor is trained using a supervised cosine metric learning approach (Wojke & Bewley, 2018). The candidate object locations of the pre-generated detections for both SORT, DeepSORT and our method are taken from (Yu et al., 2016). Further details regarding the implementation can be found in App. A.3.
We use the following metrics for evaluation. ↑ indicates that the higher the score is, the better the performance is. On the contrary, ↓ indicates that the lower the score is, the better the performance is.
· IDF1(↑): ID F1 Score · IDP(↑): ID Precision · IDR(↑): ID Recall · FAR(↓): False Alarm Ratio ·MT(↑): Mostly Tracked Trajectory · PT(↓): Partially Tracked Trajectory ·ML(↓): Mostly Lost Trajectory · FP(↓): False Positives
· FN(↓): False Negatives · IDs(↓): Number of times an ID switches to a different previously tracked object · FM(↓): Fragmentations ·MOTA(↑): Multi-object tracking accuracy ·MOTP(↑): Multi-object tracking precision ·MOTAL(↑): Log tracking accuracy
Table 2 shows that the performance of the proposed method is better than that of the model without Jacobian regularisation, and even close to the the performance of supervised learning. All methods depend on the same underlying detector for object candidates, and identical Kalmann filter parameters. Compared to baseline SORT which does not utilise any appearance information, DeepSORT has 2.54 times, VHP-VAE-SORT has 2.14 times, VHP-FMVAE-SORT (β = 300) has 2.41 times and VHPFMVAE-SORT (β = 3000) has 2.48 times fewer ID switches. Whilst the supervised DeepSORT descriptor has the least, using unsupervised VAEs with flat decoders has only 2.2% more switches, without the need for labels. Furthermore, by ensuring a quasi-Euclidean latent space, one can query nearest-neighbours efficiently via data-structures such as kDTrees. Fig. 9 shows an example of the results. In other examples of the videos, the VHP-FMVAE-SORT works similar as the deepSORT. Videos of the results can be downloaded at: http://tiny.cc/0s71cz

heading: 5 CONCLUSION
text: In this paper, we have proposed a novel approach, which we call flat manifold variational autoencoder. We have shown that—using this method—geodesics can be computed directly in the latent space by measuring the Euclidean distance between encoded data points. This is realised by combining a powerful empirical Bayes prior with a Jacobian-regularisation method that constrains the learned latent space to be Euclidean. Consequently, geodesic can be approximated 1,000 times faster than comparable state-of-the-art methods. Furthermore, using the approximated geodesic as a distance function, we have evaluated our approach on the MOT16 object-tracking database showing comparable performance as in case of supervised learning.

heading: A APPENDIX
text: A.1 VECTOR FIELD
A.2 MODEL ARCHITECTURES
A.3 IMPLEMENTATION OF VHP-FMVAE-SORT
We evaluate the performance of our model by replacing the appearance descriptor from DeepSORT with the latent space embedding from the various auto-encoders used, using the same size of 128. The hyperparameters used were held constant: the minimum detection confidence of 0.3, NMS max overlap of 0.7, max cosine distance 0.2, max appearance budget 100. We tested a VHP-FMVAE, and our regularised VHP-FMVAE with β = 300 and β = 3000.

Review summaries:
The paper is concerned with the geometry of latent spaces in VAEs. In particular, it is argued that since geodesics (shortest paths) in the Riemannian interpretation of latent spaces are expensive to compute, then it might be beneficial to regularize the decoder (generator) to be flat, such that geodesics are straight lines. One such regularization is proposed.