Topologically-Constrained Latent Variable Models
Keywords: Gaussian processes, Latent Variable Models, Topology, Dimensionality Reduction, Human Motion
Raquel Urtasun
UC Berkeley EECS & ICSI; CSAIL MIT

rurtasun@csail.mit.edu

David J. Fleet
University of Toronto

fleet@cs.toronto.edu

Andreas Geiger
Karlsruhe Institute of Technology

geiger@mrt.uka.de

Jovan Popović
CSAIL MIT

jovan@csail.mit.edu

Trevor J. Darrell
UC Berkeley EECS & ICSI; CSAIL MIT
Neil D. Lawrence
University of Manchester

Abstract
In dimensionality reduction approaches, thedata are typically embedded in a Euclidean
latent space. However for some data sets thisis inappropriate. For example, in human motion data we expect latent spaces that are
cylindrical or a toroidal, that are poorly captured with a Euclidean space. In this paper,
we present a range of approaches for embedding data in a non-Euclidean latent space.
Our focus is the Gaussian Process latent variable model. In the context of human motion
modeling this allows us to (a) learn models
with interpretable latent directions enabling,for example, style/content separation, and(b) generalise beyond the data set enabling
us to learn transitions between motion styles
even though such transitions are not present
in the data.

1. Introduction
Dimensionality reduction is a popular approach todealing with high dimensional data sets. It is often the case that linear dimensionality reduction, suchas principal component analysis (PCA) does not adAppearing in Proceedings of the 25 th International Conference on Machine Learning, Helsinki, Finland, 2008. Copyright 2008 by the author(s)/owner(s).

trevor@eecs.berkeley.edu

Neil.Lawrence@manchester.ac.uk

equately capture the structure of the data. For thisreason there has been considerable interest in the machine learning community in non-linear dimensionality
reduction. Approaches such as locally linear embedding (LLE), Isomap and maximum variance unfolding (MVU) (Roweis & Saul, 2000; Tenenbaum et al.,
2000; Weinberger et al., 2004) all define a topology
through interconnections between points in the data
space. However, if a given data set is relatively sparseor particularly noisy, these interconnections can straybeyond the ‘true’ local neighbourhood and the resulting embedding can be poor.
Probabilistic formulations of latent variable models do
not usually include explicit constraints on the embedding and therefore the natural topology of the data
manifold is not always respected 1 . Even with the correct topology and dimension of the latent space, thelearning might get stuck in local minima if the initialization of the model is poor. Moreover, the maximum
likelihood solution may not be a good model, due e.g.,
to the sparseness of the data. To get better models insuch cases, more constraints on the model are needed.
This paper shows how explicit topological constraints
can be imposed within the context of probabilistic la1
An exception is the back-constrained GP-LVM(Lawrence & Quiñonero-Candela, 2006) where a constrained maximum likelihood algorithm is used to enforce
these constraints.

Topologically-Constrained Latent Variable Models

tent variable models. We describe two approaches,both within the context of the Gaussian process latent variable model (GP-LVM) (Lawrence, 2005). The
first uses prior distributions on the latent space thatencourage a given topology. The second influences
the latent space and optimisation through constrained
maximum likelihood.
Our approach is motivated by the problem of modeling human pose and motion for character animation.
Human motion is an interesting domain because, while
there is an increasing amount of motion capture data
available, the diversity of human motion means that
we will necessarily have to incorporate a large amountof prior knowledge to learn probabilistic models that
can accurately reconstruct a wide range of motions.
Despite this, most existing methods for learning poseand motion models (Elgammal & Lee, 2004; Grochow
et al., 2004; Urtasun et al., 2006) do not fully exploit
useful prior information, and many are limited to modeling a single human activity (e.g., walking with a particular style).
This paper describes how prior information can beused effectively to learn models with specific topologies
that reflect the nature of human motion. Importantly,
with this information we can also model multiple activities, including transitions between them (e.g. fromwalking to running), even when such transitions are
not present in the training data. As a consequence,
we can now learn latent variable models with training
motions comprising multiple subjects with stylistic diversity, as well as multiple activities, such as runningand walking. We demonstrate the effectiveness of ourapproach in a character animation application, where
the user specifies a set of constraints (e.g., foot locations), and the remaining kinematic degrees of freedom
are infered.

2. Gaussian Process Latent Variable
Models (GP-LVM)
We begin with a brief review of the GP-LVM(Lawrence, 2005). The GP-LVM represents a highdimensional data set, Y, through a low dimensional
latent space, X, and a Gaussian process mappingfrom the latent space to the data space. Let Y =[y1 , ..., yN ]T be a matrix in which each row is a single
training datum, yi ∈ ℜD . Let X = [x1 , ..., xN ]T denote the matrix whose rows represent the corresponding positions in latent space, xi ∈ ℜd . Given a covariance function for the Gaussian process, kY (x, x′ ), thelikelihood of the data given the latent positions is,p(Y | X, β̄) =




1
1
T
,
exp − tr K−1
YY
Y
Z1
2

(1)

where Z1 is a normalization factor, KY is known as
the kernel matrix, and β̄ denotes the kernel hyperparameters. The elements of the kernel matrix are defined by the covariance function, (KY )i,j = kY (xi , xj ).
A common choice is the radial basis function (RBF),δ ′kY (x, x′ ) = β1 exp(− β22 ||x − x′ ||2 ) + x,x
β3 , where thekernel hyperparameters β̄ = {β1 , β2 , β3 } determine theoutput variance, the RBF support width, and the variance of the additive noise. Learning in the GP-LVM
consists of maximizing (1) with respect to the latent
positions, X, and the hyperparameters, β̄.
When one has time-series data, Y represents a sequence of observations, and it is natural to augment the GP-LVM with an explicit dynamical model.
For example, the Gaussian Process Dynamical Model(GPDM) models the sequence as a latent stochastic
process with a Gaussian process prior (Wang et al.,2008) , i.e.,1p(x1 )
T(2)
exp − tr K−1
X
Xp(X | ᾱ) =out out
X
Z2
2
where Z2 is a normalization factor, Xout =[x2 , ..., xN ]T , KX ∈ ℜ(N−1)×(N−1) is the kernel matrix
constructed from Xin = [x1 , ..., xN −1 ], x1 is given an
isotropic Gaussian prior and ᾱ are the kernel hyperparameters for KX ; below we use an RBF kernel for
KX . Like the GP-LVM the GPDM provides a generative model for the data, but additionally it provides one for the dynamics. One can therefore predict
future observation sequences given past observations,and simulate new sequences.

3. Top Down Imposition of Topology
The smooth mapping in the GP-LVM ensures that
distant points in data space remain distant in latent space. However, as discussed in (Lawrence &Quiñonero-Candela, 2006), the mapping in the opposite direction is not required to be smooth. While
the GPDM may mitigate this effect, it often produces
models that are neither smooth nor generalize well
(Urtasun et al., 2006; Wang et al., 2008).
To help ensure smoother, well-behaved models,(Lawrence & Quiñonero-Candela, 2006) suggested theuse of back-constraints, where each point in the latent
space is a smooth function of its corresponding point
in data space, xij = gj (yi ; aj ), where {aj }1≤j≤d isthe set of parameters of the mappings. One possible
mapping is a kernel-based regression model, where regression on a kernel induced feature space provides the
mapping,NX
ajm k(yi , ym ) .
(3)
xij =
m=1

Topologically-Constrained Latent Variable Models

(a)

(b)

(c)

(d)

Figure 1. When training data contain large stylistic variations and multiple motions, the generic GPDM (a) and the
back-constrained GPDM (b) do not produce useful models. Simulations of both models here do not look realistic. (c,d)
Hybrid model learned using local linearities for smoothness (i.e., style) and backconstraints for topologies (i.e., content).
The training data is composed of 9 walks and 10 runs performed by different subjects and speeds. (c) Likelihood for thereconstruction of the latent points (d) 3D view of the latent trajectories for the training data in blue, and the automaticallygenerated motions of Figs. 3 and 4 in green and red respectively.

This approach is known as the back-constrained GPLVM. When learning the back-constrained GP-LVM,
one needs to determine the hyperparameters of the kernel matrices (for the back-constraints and the covariance of the GP), as well as the mapping weights, {aj }.
(Lawrence & Quiñonero-Candela, 2006) fixed the hyperparameters of the back-constraint’s kernel matrix,
optimizing over the remaining parameters.
Nevertheless, when learning human motion data withlarge stylistic variations or different motions, neither GPDM nor back-constrained GP-LVM producesmooth models that generalize well. Fig. 1 depicts
three 3–D models learned from 9 walks and 10 runs.
The GPDM (Fig. 1(a)) and the back-constrainted
GPDM2 (Fig. 1 (b)) do not generalize well to new runs
and walks, nor do they produce realistic animations.
In this paper we show that with a well designed
set of back-constraints good models can be learned(Fig. 1(c)). We also consider an alternative approachto the hard constraints on the latent space arisingfrom gj (yi ; aj ). We introduce topological constraints
through a prior distribution in the latent space, basedon a neighborhood structure learned through a generalized local linear embedding (LLE) (Roweis & Saul,
2000). We then show how to incorporate domainspecific prior knowledge, which allows us to develop
motion models with specific topologies that incorporate different activities within a single latent space andtransitions between them.
3.1. Locally Linear GP-LVM
The locally linear embedding (LLE) (Roweis & Saul,
2000) preserves topological constraints by finding a
representation based on reconstruction in a low dimensional space with an optimized set of local weightings.
Here we show how the LLE objective can be combined2

We use an RBF kernel for the inverse mapping in (3).

with the GP-LVM, yielding a locally linear GP-LVM(LL-GPLVM).
The locally linear embedding assumes that each data
point and its neighbors lie on, or close to, a locallylinear patch on the data manifold. The local geometry of these patches can then be characterized by linear coefficients that reconstruct each data point fromits neighbors. This is done in a three step procedure: (1) the K nearest neighbors, {yj }j∈ηi , of eachpoint, yi , are computed using Euclidean distance in
the input space, dij = ||yi − yj ||2 ; (2) the weights
w = {wij } that best reconstruct each data pointfrom its neighbors are obtained by minimizing Φ(w) =PN
P
2
i=1 ||yi −j∈ηi wij yj || ; and (3) the latent positions
xi best reconstructed by the weights wij are computed
PN
Pby minimizing Φ(X) = i=1 ||xi − j∈ηi wij xj ||2 .

In the LLE, the weight matrix w is sparse (only a small
number of neighbors is used), and the two minimizations can be computed in closed form. In particular,
computing the weights can be done by solving, ∀j ∈ ηi ,the following system,Xsim simCkj
wij = 1 ,
(4)
k

sim
where Ckj
= (yi − yk )T (yi − yj ) if j, k ∈ ηi , and 0
otherwise. OnceP
the weights are computed, they arerescaled so that j wij = 1.

The LLE energy function can be interpreted, for agiven set of weights w, as a prior that forces eachlatent point to be locally reconstructedby its neighbors,i.e., p(X|w) = Z1 exp − σ12 Φ(X) , where Z isa normalization constant, and σ 2 represents a global
scaling of the prior. Note that strictly speaking this is
not a proper prior as it is conditioned on the weights
which depend on the training data. Following (Roweis& Saul, 2000), we first compute the neighbors basedon the Euclidean distance. For each training point yi ,

Topologically-Constrained Latent Variable Models

we then compute the weights solving Eq. (4).
Learning the LL-GPLVM is then equivalent to minimizing the negative log posterior of the model, 3 i.e.,LS=log p(Y|X, β̄) p(β̄) p(X|w)
X

D
1
T
=
ln |KY | + tr K−1
+
ln βi
Y YY
2
2
i
+

(a)

(d)

(b)

(e)

N
N
d
1 XX k X k k 2
wij xj k + C , (5)
kx
−
i
σ2
j=1
i=1
k=1

where C is a constant, and xki is the k-th componentof xi . Note that we have extended the LLE to havea different prior for each dimension. This will be useful below as we incorporate different sources of priorknowledge. Fig. 2 (a) shows a model of 2 walks and 2
runs learned with the locally linear GPDM. Note howsmooth the latent trajectories are.
We now have general tools to influence the structureof the models. In what follows we generalize the topdown imposition of topology strategies (i.e. backconstraints and locally linear GP-LVM) to incorporate
domain specific prior knowledge.

4. Reflecting Knowledge in Latent
Space Structure
A problem for modeling human motion data is thesparsity of the data relative to the diversity of naturally plausible motions. For example, while we might
have a data set comprising different motions, such as
runs, walks etc., the data may not contain transitions
between motions. In practice however, we know thatthese motions will be approximately cyclic and thattransitions can only physically occur at specific points
in the cycle. How can we encourage a model to respect such topological constraints which arise fromprior knowledge?
We consider two alternatives to solve this problem.
First, we show how one can adjust the distance metricused in the locally linear embedding to better reflect
different types of prior knowledge. We then show how
one can define similarity measures for use with the
back-constrained GP-LVM. Both these approaches encourage the latent space to construct a representationthat reflects our prior knowledge. They are complementary and can be combined to learn better models.
3

When learning a locally linear GPDM, the dynamics
and the locally linear prior are combined as a product of potentials. The objective function becomes LS + d2 ln |KX | +
`
´ P
T
1
tr K−1
X Xout Xout +i ln αi , with LS defined as in (5).
2

(c)
(f)
Figure 2. First two dimensions of 3–D models learned
using (a) LL-GPDM (b) LL-GPDM with topology (c)
LL-GPDM with topology and transitions. (d) Backconstrained GPDM with an RBF mapping. (e) GPDM
with topology through backconstraints. (f) GPDM withbackconstraints for the topology and transitions. For themodels using topology, the cyclic structure is imposed in
the last 2 dimensions. The two types of transition points(left and right leg contact points) are shown in red andgreen, and are used as prior knowledge in (c,f).

4.1. Prior Knowledge through Local
Linearities
We now turn to consider how one might incorporate
prior knowledge in the LL-GPLVM framework. This isaccomplished by replacing the local Euclidean distancemeasures used in Section 3.1 with other similarity measures. That is, we can modify the covariance used tocompute the weights in Eq. (4) to reflect our priorknowledge in the latent space. We consider two examples: the first involves transitions between activities;
with the second we show how topological constraints
can be placed on the form of the latent space.
Covariance for Transitions Modeling transitions
between motions is important in character animation.
Transitions can be infered automatically based on similarity between poses (Kovar et al., 2002) or at points
of non-linearity of the dynamics (Bissacco, 2005), andthey can be used for learning. For example, for motions as walking or running, two types of transitions
can be identified: left and right foot ground contacts.

Topologically-Constrained Latent Variable Models

To model such transitions, we define an index on theframes of the motion sequence, {ti }N
i=1 . We then define
subsets of this set, {t̂i }M
i=1 , which represents frames
where transitions are possible. To capture transitionsin the latent model we define the elements for the covariance matrix as follows,


trans
Ckj
= 1 − δkj exp(−ζ(tk − tj )2 )(6)
with ζ a constant, and δij = 1 if ti and tj are in thesame set {t̂k }M
k=1 , and otherwise δij = 0. This covariance encourages the latent points at which transitions
are physically possible to be close together.

Covariance for Topologies We now consider covariances that encourage the latent space to have aparticular topology. Specifically we are interested insuitable topologies for walking and running data. Because the data are approximately periodic, it seems
appropriate to have a non-Cartesian topology. To thisend one can extract the phase of the motion4 , φ, anduse it with a covariance to encourage the latent points
to exhibit a periodic topological structure within a
Cartesian space. As an example we consider a cylindrical topology within a 3–D latent space by constraining
two of the latent dimensions with the phase. In particular, to represent the cyclic motion we construct a distance function on the unit circle, where a latent point
corresponding to phase φ is represented with coordinates (cos(φ), sin(φ)). To force a cylindrical topology
on the latent space, we specify different covariances foreach latent dimension
cos
Ck,j= (cos(φi ) − cos(φk )) (cos(φi ) − cos(φj )) (7)sinCk,j= (sin(φi ) − sin(φk )) (sin(φi ) − sin(φj )) , (8)with k, j ∈ ηi . The covariance for the remaining dimension is constructed as usual, based on Euclidean
distance in the data space. Fig. 2 (b) shows a GPDM
constrained in this way, and in Fig. 2 (c) the covariance is augmented with transitions.
Note that the use of different distance measures foreach dimension of the latent space implies that theneighborhood and the weights in the locally linear
prior will also be different for each dimension. Here,three different locally linear embeddings form the priordistribution.
4.2. Prior Knowledge with Back Constraints
As explained above, we can also design backconstraints to influence the topology and learn useful4

The phase can be easily extracted from the data by
Fourier analysis or by detecting key postures and interpolating the phases between them. Another idea, not further
explored here, would be to optimize the GP-LVM with respect to the phase.

transitions. This can be done by replacing the kernel of Eq. (3). Many kernels have interpretations as
similarity measures. In particular, any similarity measure that leads to a positive semi-definite matrix canbe interpreted as a kernel. Here, just as we define
covariance matrices above, we extend the original formulation of back constraints by constructing similarity
measures (i.e., kernels) to reflect prior knowledge.
Similarity for Transitions To capture transitions
between two motions, we wish to design a kernel that
expresses strong similarity between points in the respective motions where transitions may occur. We canencourage transition points of different sequences tobe proximal with the following kernel matrix for the
back-constraint mapping:XX
k trans (ti , tj ) =δml k(ti , t̂m )k(tj , t̂l )
(9)
m

l

where k(ti , t̂l ) is an RBF centered at t̂l , and δml = 1
if t̂m and t̂l are in the same set. The influence of the
back-constraints is controlled by the support width of
the RBF kernel.
Topologically Constrained Latent Spaces We
now consider kernels that force the latent space to havea particular topology. To force a cylindrical topology
on the latent space, we can introduce similarity measures based on the phase, specifying different similarity
measures for each latent dimension. As before we construct a distance function in the unit circle, that takes
into account the phase. A periodic mapping can beconstructed from a kernel matrix as follows,
N
X
cos
acos
xn,1 =m k(cos(φn ), cos(φm )) + a0 δn,m ,
m=1

xn,2 =

N
X

sin
asin
m k(sin(φn ), sin(φm )) + a0 δn,m ,

m=1

where k is an RBF kernel function, and xn,i is the ith
coordinate of the nth latent point. These two mappings project onto two dimensions of the latent space,forcing them to have a periodic structure (which comes
about through the sinusoidal dependence of the kernel
on phase). Fig. 2 (e) shows a model learned using
GPDM with the last two dimensions constrained inthis way (the third dimension is out of plane). The
first dimension is constrained by an RBF mapping on
the input space. Each dimension’s kernel matrix canthen be augmented by adding the transition similarity
of Eq.(9), resulting in the model shown in Fig. 2 (f).
4.3. Model Combination
One advantage of our framework is that covariance matrices can be combined in a principled manner to form

Topologically-Constrained Latent Variable Models

new covariance matrices. Covariances can be multiplied (on an element by element basis) or added together. Similarly, similarities can be combined. Multiplication has, loosely speaking, an ‘AND gate effect’,i.e. both similarity measures must agree that an object is similar for their product to express similarity.
Adding them produces more of an ‘OR gate effect’, i.e.
if either representation expresses similarity the resulting measure will also express similarity.
The two sections above have shown how to incorporate prior knowledge in the GP-LVM by means of 1)
local linearities and 2) back-constraints. In general,the latter should be used when the manifold has awell-defined topology, since it has more influence on
the learning. When the topology is not so well defined(e.g., due to noise) one should use local linearities.
Both techniques are complementary and can be combined straightforwardly by including priors over somedimensions, and constraining the others through backconstraint mappings. Fig. 1 shows a model learned
with LL-GPDM for smoothness and back-constraints
for topology.

5. Results
We demonstrate the effectiveness of our approach with
two applications. First we show how models of multiple activities can be learned, and realistic animations
can be produced by drawing samples from the model.
We then show an interactive character animation application, where the user specifies a set of sparse constraints and the remaining kinematic degrees of freedom are infered.
5.1. Learning multiple activities

4.4. Multiple Activities and Transitions

We first considered a small training set comprised of
4 gait cycles (2 walks and 2 runs) performed by one
subject at different speeds. Fig. 2 shows the latent
spaces learned under different prior constraints. All
the models are learned using two independent dynamical models, one for walking and one for running. Note
how the phases are aligned when imposing a cylindrical
topology, and how the LL-GPDM is smooth. Notice
the difference between the LL-GPDM (Fig. 2 (c)) andthe backconstrained GPDM (Fig. 2 (f)) when transition constraints are included. Neverthess, both models ensure that the transition points (shown in red andgreen) are proximal.

Once we know how to ensure that transition points areclose together and that the latent structure has thedesired topology, we still need to address two issues.
How do we learn models that have very different dynamics? How can we simulate dynamical models thatlie somewhere between the different training motions?
Our goal in this section is to show how latent models for different motions can be learned independently,
but in a shared latent space that facilitates transitions
between activities with different dynamics.

Fig. 1 (c) shows a hybrid model learned using LLGPDM for smoothness, and back-constraints for topology. The larger training set comprises approximately
one gait cycle from each of 9 walking and 10 runningmotions performed by different subjects at different
speeds (3 km/h for walking, 6–12 km/h for running).
Colors in Fig. 1 (a) represent the variance of the GP
as a function of latent position. Only points close to
the surface of the cylinder produce poses with highcertainty.

T T] denote training data for M
Let Y = [Y1T , ..., YM
different activities. Each Ym comprises several different motions. Let X = [XT1 , ..., XTM ]T denote the corresponding latent positions. When dealing with multiple
activities, a single dynamical model cannot cope withthe complexity of the different dynamics. Instead, we
consider a model where the dynamics of each activity
are modeled independently5 . This has the advantagethat a different kernel can be used for each activity.

To enable interpolation between motions with different
dynamics, we combined these independent dynamical
models in the form of a mixture model. This allows us
to produce motions that gracefully transition between
different styles and motion types (Figs. 3 and 4).
5Another interpretation is that we have a block diagonal
kernel matrix for the GP that governs the dynamics.

We now illustrate the model’s ability to simulate different motions and transitions. Given an initial latent position x0 , we generate new motions by sampling the mixture model, and using mean prediction
for the reconstruction. Choosing different initial conditions results in very different simulations (Fig. 1 (d)).
The training data are shown in blue. For the firstsimulation (depicted in green), the model is initialized to a running pose with a latent position not far
from walking data. The system transitions to walking
quite naturally. The resulting animation is depicted inFig. 3. For the second example (in red), we initialize
the simulation to a latent position far from walking
data. The system evolves to different running styles
and speeds (Fig. 4). Note how the dynamics, and thestrike length, change considerably during simulation.

Topologically-Constrained Latent Variable Models

Figure 3. Transition from running to walking: The system transitions from running to walking in a smooth andrealistic way. The transition is encouraged by incorporating prior knowledge in the model. The latent trajectories areshown in green in Fig. 1 (d).

Figure 4. Different running styles and speeds: The system is able to simulate a motion with considerably changes
in speed and style. The latent trajectories are shown in red in Fig. 1 (d).

Rather than approximating the entire posterior, we use
hill-climbing to find MAP estimates. Assuming thatthe user constraints are noise-free, the minimization
can be expressed as
min
φ1:T

subject toFigure 5. Single activity 3D latent models learned from(left) 5 jumps of 2 different subjects using local linearities,(b) 7 walking cycles of one subject using back-constraints.

5.2. Character animation from constraints
A key problem in the film and game industry is thelack of tools to allow designers to easily generate animations. Traditional techniques such as keyframing
are time consuming; an expert can expend days ingenerating a few seconds of animation. A very useful
tool would provide the user with a simple way of generating motions from constraints that she/he defined.
Typical constraints are keyframes (i.e., specificationof the position of the full body in a particular time
instant), or joint trajectories. Here we use the topologically constrained motion models as priors over the
space of possible motions.
Our motion estimation formulation is based on a statespace model with a GPDM prior over pose and motion.
Given the state, φt = (yt , xt ), the goal is to estimate
the state sequence φ1:T = (φ1 , · · · , φT ) that satisfies
the user constraints u1:J . Inference is performed in a
Batch mode, so that the state is infered all at once.
The posterior can be expressed asp(φ1:T |u1:J , M) ∝ p(u1:J |φ1:T )p(φ1:T |M)(10)

where we assumed that p(u1:J ) is uniformily distributed; all the user constraints are equally probable.
The prediction distribution p(φ1:T |M) can be further
factored as follows
TYp(yt |xt , M)(11)p(φ1:T |M) = p(x1:T |M)
t=1

Lpose + Ldyn + Lsmooth||u − f (yψ(u) )|| = 0(12a)where f is a forward kinematics function (i.e., afunction that maps joint angles to positions in the
3D world), ψ(u) is a function that outputs theframe
P where each constraint uj is defined, Lpose =− i ln p(yt |xt , M) and Ldyn = − ln p(x1:T |M ) are
the pose and dynamics likelihood from the GPDM
prior (Urtasun et al., 2006), and Lsmooth =PT −1 PP 1 j
j 2
1
t=1
j=1 σ 2 (yt+1 − yt ) is a term that encourσ2sjage smooth motions, where ytj is the j-th component of
yt , and σj2 is a constant that encounters from the factthat each degree of freedom has a different variance.

Initialization is important since a large number of variables need to be optimised and our objective function
is non-convex. In particular, we sample the model
starting at each training point and use as initialization the sample that is closest to the user constraints.
To demonstrate the effectiveness of our approach welearned models of two different motions, walking andjumping (Fig. 5). We impose smoothness and cyclic
topologies using back-constraints for the walking andlocal linearities for the jumping. We demonstrate theability of the model to generalize to unseen styles.
We first show how the model can produce realistic animations from a very small set of user defined constraints. The user specifies the contact points of thefoot with the ground (first row of Fig. 6) for walking
and the foot trajectories for the jumping (third rowof Fig. 6), and the rest of the degrees of freedom are
infered producing very realistic animations.
The model can also generalize to styles very different

Topologically-Constrained Latent Variable Models
20

15

15
y

25

20

15
y

25

20

15
y

25

20

15
y

25

20

y

25

10

10

10

10

5

5

5

5

0

0

0

0

−35

−35
−30

−35

−20

−20
−15

−20
−15

−10

−5

−5
0

−10
−5

−5

0

5

0

5
15
10

−15
−10

−5

0

15

10

z

0

5
10

15

5

5
15

10

z

10

15

5

25

x

−25
−20

−15
−10

25

−30
−25

−20
−10

15

−35
−30

−25

−15

z

0

−30
−25

10

5

−35

−30
−25

10

z

5
10

15

5

25

x

15

10

25

x

15

15

15

15

10

10

10

10

5

5

5

5

0

0

0

0

0

−40

−40

−40

−40

−40

−20
−10

0

0
10

15
10

z
x

10

z

5

35

35

35

30

30

30

30

25

25

25

25

20

20

20

20

5

30

25

20

35

35

35

30

30

30

25

25

25

20

20

20

y

y

x

15

15

15

15

15

15

15

15

10

10

10

10

10

10

10

10

5

5

5

5

5

5

5

−25

−20

−15

−10

0

−5

−30

−25

−20

z

−15

−10

−5

0
−30

−25

−20

z

−10

−5

−25

−20

−15

25

25

20

20

20

−30

−25

−20

z

−15

−10

−30

−5

−25

−20

−15

−10

5

0
−30

−5

−25

−20

30

25

20

−15

−10

−5

−30

−25

−20

z

z

z

−15

−10

−5

z

30

30

30

25

25

25

20

20

20

y

y

25

−5

y

30

y

30

−10

0

0

0

−30

z

30

y

−15

y

0
−30

y

0

15
10

z
x

y

y

y

10
5

x

35

y

10

15

z

5

x

35

0
10

15

y

10

10

−10

0

15
5

−20
−10

y

0

z

−30

−20
−10

10

5

−30

−20
−10

x

10

−30

−20

5

y

15

y

20

y

20

y

20

y

20

−30

10

15

25

x

20

−30

15

10

z

5

15

15

15

15

15

15

15

10

10

10

10

10

10

10

5

5

5

5

5

5

0

0

0

0

0

0

−35

−30

−25

−20

−15

−10

−5

0

z

−35

−30

−25

−20

−15
z

−10

−5

0

−35

−30

−25

−20

−15

−10

−5

0

−35

−30

−25

z

−20

−15
z

−10

−5

0

−35

−30

−25

−20

−15
z

−10

−5

0

5

0
−35

−30

−25

−20

−15
z

−10

−5

0

−35

−30

−25

−20

−15

−10

−5

0

z

Figure 6. Animations generated from a set of foot constraints (green). First row: Normal walk. Second row: Generalization a different style by changing the user constraints to be separate in the coronal plane. Third row: Short jump.
Last row: Longer stylistic jump. See video at http://people.csail.mit.edu/rurtasun

from the ones in the training set, by imposing constraints that can be satisfied only by motions very different from the training data. In particular, the user
placed the foot constraints far in the coronal plane forwalking. Consequently the character opens the legs tosatisfy the constraints (second row of Fig. 6). In thelast row of Fig. 6 the user places the foot trajectories
to create a jump with a style very different from thetraning data (the character opens his legs and bends
his body and arms in an exaggerated way).

6. Conclusions
In this paper we have proposed a general frameworkof probabilistic models that learn smooth latent variable models of different activities within a shared latent space. We have introduced a principled way toinclude prior knowledge, that allow us to learn specific topologies and transitions between the different
motions. Although we have learned models composedof walking, running and jumping, our framework is
general, being applicable in any data sets where there
is a large degree of prior knowledge for the problem
domain, but the data availability is relatively sparsecompared to its complexity.

References
Bissacco, A. (2005). Modeling and Learning Contact Dynamics in Human Motion In CVPR. (pp. 421–428).

Elgammal, A., & Lee, C. (2004). Inferring 3D Body Posefrom Silhouettes using Activity Manifold Learning. In
CVPR (pp. 681–688).
Grochow, K., Martin, S., Hertzmann, A., & Popovic, Z.
(2004). Style-based inverse kinematics In SIGGRAPH.
Kovar, L., Gleicher, M., & Pighin, F. (2002).
Graphs In SIGGRAPH, (pp. 473–482).

MotionLawrence, N. (2005). Probabilistic non-linear principal
component analysis with gaussian process latent variable models. JMLR, 6, 1783–1816.
Lawrence, N. D., & Quiñonero-Candela, J. (2006). Local
distance preservation in the GP-LVM through back constraints In ICML (pp. 96–103).
Rasmussen, C. E., & Williams, C. K. (2006). Gaussian
process for machine learning. MIT Press.
Roweis, S., & Saul, L. (2000). Nonlinear dimensionality
reduction by locally linear embedding. Science, 290.
Tenenbaum, J., de Silva, V., & Langford, J. (2000). A
global geometric framework for nonlinear dimensionality
reduction. Science, 290, 2319–2323.
Urtasun, R., Fleet, D. J., & Fua, P. (2006). 3d people
tracking with gaussian process dynamical models. In
CVPR, (pp. 932–938).
Wang, J., Fleet, D. J., & Hertzman, A. (2008). Gaussian
process dynamical models . In PAMI 30(2), 283–298 .
Weinberger, K. Q., Sha, F., & Saul, L. K. (2004). Learninga kernel matrix for nonlinear dimensionality reduction.
In ICML, (pp. 106–113).