Abstracts

On the Geometry of Latent Variable Models
Søren Hauberg
Latent variable models (LVMs) describe the distribution of data y ∈ Y = RDthrough a low-dimensional random variable x ∈ X = Rd , (d  D) and a (generally nonlinear) stochastic mapping f : X → Y. Here we discuss the random
Riemannian geometry induced by this stochastic mapping. The presented results
was first stated in [1, 2].
To make the discussion explicit, we consider a Gaussian Process (GP) LVM [3]where f has component-wise conditionally independent Gaussian process entries,
fi (x) ∼ GP(mi (x), k(x, x0 )),(1)∀i = 1, . . . , D.

Here mi and k are the mean and covariance functions of the ith GP. Note that weassume the same covariance function across all dimensions as this simplify future
calculations. The key presented results holds regardless of this simplification.
Assuming k is sufficiently smooth covariance then the image of a sample fromf is a smooth d-dimensional immersed manifold. Note that this manifold is only
locally diffeomorphic to d-dimensional Euclidean space, and it may globally selfintersect. It is then natural to consider the pull-back metric M = J> J over X ,where J ∈ RD×d is the Jacobian of f . This defines a Riemannian metric over X .
Since f is stochastic, M is a stochastic object as well.
Since Gaussian variables are closed under differentiation, then J follows a GP,(2) J ∼DY
j=1

N (µ(j, :), Σ) =DY−12>−1
N (∂K>x,∗ K̃x,x Y:,j , ∂ K∗,∗ − ∂K∗,x Kx,x ∂K∗,x ),j=1

where we use standard notation for GPs [4]. It then follows that M at a given
point is governed by a non-central Wishart distribution [5](3)M ∼ Wd (D, Σ, E[J]> E[J]).

The entire metric by definition follows a generalized Wishart process [6].
Since the metric is a stochastic variable, we cannot apply standard Riemannian
geometry to understand the space X (e.g. curvature is stochastic, geodesics aresolutions to a stochastic differential equation, etc.). We can, however, inspect theleading moments of the metric(4)(5)

E[M] = E[J> J] = E[J]> E[J] + D Σ>var[Mij ] = D(Σ2ij + Σii Σjj ) + µ>j Σµj + µi Σµi= O(D)
= O(D)

which we see both grow linearly with the dimension of Y. This motivate thequestion as to how the pull-back metric behaves in high dimensions, D → ∞. To
1

2

Oberwolfach Report 3/2018

ensure that the inner product of Y converges to the usual L2 inner product in thelimit D → ∞ we let
ZD
1 X
ha, biY =(6)ai bi −−−−→ at bt dt.
D→∞
D i=1
1 >
J J, which has moments
Then the natural pull-back becomes M̃ = D1 >1
E[M̃] = E(7)J J = E[J]> E[J] + Σ= O(1)DD 1 >111
Σµ+µΣµ=O(8)var[M̃ij ] = (Σ2ij + Σii Σjj ) + 2 µ>jiji
2DDDDIn the limit D → ∞ we, thus, see that the variance vanishes and the metricbecomes fully deterministic even if the underlying manifold is a stochastic object.
Implications and Extensions. This simple-to-prove
result is rather surprising: even if we only have stochastic information about the underlying data manifold, its
metric is deterministic. Furthermore, from Eq. 7 we seethat this deterministic metric correspond to the (usual)pull-back metric of the mean f plus an additional term
capturing the uncertainty of the manifold. This imply
that the metric is large in regions of low data density(where the manifold is uncertain), and consequently,that geodesics will tend to avoid such regions. One such example is shown in thefigure. Here human motion capture data y is used to estimate a two-dimensional
manifold [1]. In the figure white points correspond to low-dimensional representations of the data, the green curve is an example geodesic computed under theexpected metric, and the background color is proportional to the volume measureinduced by the expected metric. We see that the metric is “larger” in regions of
low data density and that geodesics consequently follow the structure of the data.
The latter is a useful property when analyzing real data as distance-based data
distribution will adapt well to the data [2].
From a practical point of view, geodesics can be computed in X by numericallysolving the usual system of ordinary differential equations under the expectedmetric. The solution will be a curve in X , which correspond to a GP in Y. Assuch, geodesics remain stochastic objects, but they can be determined by solving
a set of deterministic equations.
The presented derivations rely on the dimensions of f (X ) being conditionallyindependent, which is a common assumption. It can be eased upon: if the dimensions are (imperfectly) correlated, then the variance will still decrease, albeit ata slower rate than D−1 . Consequently, as a general rule of thumb, the stochastic
pull-back metric of an uncertain manifold immersed in a high-dimensional space
is well approximated by the (deterministic) expected metric.

Statistics for Data with Geometric Structure

3Acknowledgments. SH was supported by a research grant (15334) from VILLUM FONDEN. This project has received funding from the European ResearchCouncil (ERC) under the European Union’s Horizon 2020 research and innovation
programme (grant agreement no 757360).
References[1] A. Tosi, S. Hauberg, A. Vellido, and N. Lawrence, Metrics for Probabilistic Geometries, In
The Conference on Uncertainty in Artificial Intelligence (2014).
[2] G. Arvanitidis, LK. Hansen, and S. Hauberg, Latent Space Oddity: on the Curvature of Deep
Generative Models, In the International Conference on Learning Representations (2018).
[3] N. Lawrence, Probabilistic non-linear principal component analysis with Gaussian processlatent variable models, Journal of machine learning research 6. Nov (2005): 1783-1816.
[4] CE. Rasmussen, and CKI. Williams, Gaussian Processes for Machine Learning, University
Press Group Limited (2006).
[5] RJ. Muirhead, Aspects of Multivariate Statistical Theory, John Wiley & Sons (2005).
[6] AG. Wilson, and Z. Ghahramani, Generalised Wishart Processes, In The Conference on
Uncertainty in Artificial Intelligence (2011).