The Laplacian Eigenmaps Latent Variable Model

We introduce the Laplacian Eigenmaps Latent Variable Model (LELVM), a probabilistic method for nonlinear dimensionality reduction that combines the advantages of spectral methods—global optimisation and ability to learn convoluted manifolds of high intrinsic dimensionality—with those of latent variable models—dimensionality reduction and reconstruction mappings and a density model. We derive LELVM by defining a natural out-of-sample mapping for Laplacian eigenmaps using a semi-supervised learning argument. LELVM is simple, nonparametric and computationally not very costly, and is shown to perform well with motion-capture data. Consider the problem of dimensionality reduction, where we observe data y ∈ R assumed to lie on a manifold of dimension L < D, and want to model this data using latent variables x ∈ R. One class of dimensionality reduction methods are continuous latent variable models (LVMs; see Carreira-Perpiñán, 2001 for review); examples include factor analysis (linear) and the generative topographic mapping (GTM, nonlinear; Bishop et al., 1998). LVMs have two important properties: (1) they define a joint probability density p(x,y), thus densities p(x) and p(y) for the latent and observed variables, respectively. This means they can deal with missing data naturally, they can be used to model prior distributions (e.g. in tracking), and they can also be combined into mixtures of LVMs. (2) They define mappings for dimensionality reduction x = F(y) and reconstruction y = f(x) which can be applied to unseen data. However, they also have two important practical problems: (1) it is difficult to use more than 2 latent variables if using nonlinear models because the computational cost grows exponentially with L (or, if using mixtures of local linear models, aligning the component spaces is difficult). (2) Parameter estimation (by maximum likelihood) can easily result in bad local optima which yield entangled mappings. For some important applications, for example tracking 3D articulated pose from monocular video (Urtasun et al., 2005), it is crucial to have nonlinear, global, differentiable mappings and densities that use a latent (state) space of dimension possibly quite higher than 2. Another class of dimensionality reduction methods of interest in recent years are spectral methods, such as Isomap (Tenenbaum et al., 2000), LLE (Roweis and Saul, 2000) or Laplacian eigenmaps (Belkin and Niyogi, 2003). These methods provide a set of correspondences (xn,yn) of latent and observed points for points in the training set, and this correspondence is often able to learn complex manifolds that are very challenging for LVMs. Their estimation (by solving an eigenproblem) has no local optima and can easily yield latent spaces of dimension more than 2. However, they define neither mappings for out-of-sample points nor a probability density. Thus, LVMs and spectral methods have somewhat complementary advantages and disadvantages, so it would be desirable to combine them to get the best of both worlds. One way of partially doing this is to use the correspondences produced by a spectral method to initialise a LVM, or to fit a dimensionality reduction or reconstruction mapping (sec. 4). However, up to now there has not been a formal relation between both types of methods. We address this question in this paper by proposing a natural extension of Laplacian eigenmaps to out-of-sample points that yields both dimensionality reduction and reconstruction mappings, and a joint density p(x,y), thus defining a LVM that we call Laplacian eigenmaps latent variable model (LELVM). Before describing this extension in section 3, we first review LVMs (in particular GTM) and spectral methods (in particular Laplacian eigenmaps) in sections 1–2. We demonstrate the method in toy problems and real-world problems (in particular motion-capture data) in section 4, and discuss related work in section 5. 1 Latent variable models: GTM A LVM is defined by a prior distribution p(x) in latent space (x ∈ R), a (reconstruction) mapping f : x → y, and a noise model p(y|x) = p(y|f(x)) in observed space (y ∈ R, D > L). The density in observed space is then obtained by marginalising the joint density p(x,y) in latent space: p(y) = ∫ p(y|x)p(x) dx. A dimensionality reduction mapping can be defined from the posterior distribution in latent space p(x|y) as either the mean or, for multimodal distributions, the modes. Estimating the parameters in p(x), f and p(y|x) is achieved by maximum likelihood given a sample {yn}n=1 in observed space, often by an EM algorithm (where the latent variables are the missing data). Mixtures of LVMs p(y|m) for m = 1, . ,M are readily defined as ∑M m=1 p(m)p(y|m). A linear example of LVM is factor analysis, where p(x) is Gaussian, f is linear and p(y|x) is diagonal Gaussian; other linear LVMs include probabilistic PCA and independent component analysis. A nonlinear example is GTM (Bishop et al., 1998), where p(x) = ∑K k=1 δ(x − xk) (discretised uniform with points xk arranged on a square grid for visualisation), f = WΦ(x) is a radial basis function mapping (with fixed basis functions Φ = (φ1, . . . , φF ) and tunable weights W), and p(y|x) = N (f(x), σI) is isotropic Gaussian. The observed density p(y) is then a Gaussian mixture with centres constrained by f . The discretisation of x is a computational device to obtain a closed-form marginal p(y) in observed space, but the number of grid points K grows exponentially with the latent dimension L, limiting it in practice to L . 3. When modelling complex, convoluted manifolds such as a spiral, initialising GTM’s training from either random parameter values or the PCA solution almost always results in a tangled manifold (see fig. 1). The problem is not one of modelling ability but of search: the log-likelihood function is full of bad optima. The recently proposed Gaussian Process Latent Variable Model (GPLVM) (Lawrence, 2005) marginalises over functions f rather than over latent variables x. Essentially, it yields a Gaussian process reconstruction mapping f and a conditional probability p(y|x), but neither a joint density p(x,y) nor a posterior distribution p(x|y) in latent space. Unlike in GTM, in GPLVM the latent points {xn}n=1 associated with the data points are tunable parameters, which makes initialisation from a LE solution direct (and so can succeed with convoluted mappings), and also allows practical use of L > 2. However, its log-likelihood function also has many bad local optima and training is computationally very costly: each gradient iteration is O(N ) (since the covariance matrix involved is not sparse), though approximations based on using few points and so small N exist to accelerate it. 2 Spectral methods: Laplacian eigenmaps (LE) Define a neighbourhood graph on the sample data {yn}n=1, such as a k-nearest-neighbour or -ball graph, or a full mesh, and weigh each edge yn ∼ ym by a symmetric affinity function K(yn,ym) = wnm, typically Gaussian: wnm = exp (− 1 2 ‖(yn − ym)/σ‖ 2 ). Given this weighted graph, in Laplacian eigenmaps (LE) (Belkin and Niyogi, 2003) we seek latent points {xn}n=1 ⊂ R that are the solution of the optimisation problem: