Model Transport: Towards Scalable Transfer Learning on Manifolds
Oren Freifeld
MIT
Cambridge, MA, USA

Søren Hauberg
DTU Compute
Lyngby, Denmark

Michael J. BlackMPI for Intelligent Systems
Tübingen, Germany

freifeld@csail.mit.edu

sohau@dtu.dk

black@tue.mpg.de

Abstract
We consider the intersection of two research fields:
transfer learning and statistics on manifolds. In particular, we consider, for manifold-valued data, transfer learning of tangent-space models such as Gaussians distributions, PCA, regression, or classifiers. Though one would
hope to simply use ordinary Rn -transfer learning ideas, themanifold structure prevents it. We overcome this by basing
our method on inner-product-preserving parallel transport,a well-known tool widely used in other problems of statistics on manifolds in computer vision. At first, this straightforward idea seems to suffer from an obvious shortcoming: Transporting large datasets is prohibitively expensive,
hindering scalability. Fortunately, with our approach, wenever transport data. Rather, we show how the statisticalmodels themselves can be transported, and prove that for
the tangent-space models above, the transport “commutes”
with learning. Consequently, our compact framework, applicable to a large class of manifolds, is not restricted by
the size of either the training or test sets. We demonstrate
the approach by transferring PCA and logistic-regression
models of real-world data involving 3D shapes and image
descriptors.

(a) Data on a manifold(b) Data models(c) Ordinary translation(d) Model transport

Figure 1: Model Transport for covariance estimation. On
nonlinear manifolds, statistics of one class (red) are transported to improve a statistical model of another class (blue).
While ordinary translation is undefined (c), and data transport is expensive, a model can be inexpensively transported(green) while preserving data statistics (d).

1. Introduction
In computer vision, manifold-valued data arise often.
The advantages of representing such data explicitly on amanifold include a compact encoding of constraints, distance measures that are usually superior to ones from Rn ,and consistency. For such data, statistical modeling on the
manifold is generally better than statistical modeling in a
Euclidean space [12, 18, 29, 39]. Here we consider the first
scalable generalization, from Rn to Riemannian manifolds,of certain types of transfer learning (TL). In particular, weconsider TL in the context of several popular tangent-spacemodels such as Gaussian distributions (Fig. 1b), PCA, classifiers, and simple linear regression. In so doing, we recast
TL on manifolds as TL between tangent spaces. This gen-eralizes those Rn -TL tasks where models learned in one region of Rn are utilized in another. Note, however, that we
do not claim that all Rn -TL tasks have this form.
Let M denote an n-dimensional manifold and let Tp M
and Tq M denote two tangent spaces to M , at points p, q ∈M . One cannot simply apply models learned in Tp M todata in Tq M as these, despite both being isomorphic to Rn ,are two different spaces: A model on Tp M is usually noteven defined in Tq M ; see Fig. 1c. Such obstacles, caused by
the curvature of M , do not arise in Rn -TL. To address thiswe could parallel transport (PT) [7] the data from Tp M to
Tq M , learn a model of the transported data in Tq M , and usethis model in Tq M . In fact, this brings us back to the settingof ordinary Rn -TL whose methods are now transparently
applicable. The question whether this idea is statisticallyuseful depends on the application and the data of interest;
our experiments show that the answer is often positive.
Unfortunately, this solution scales poorly: Transporting
data is expensive. This is particularly true for large datasetsor when the PT has no closed form. Also, the value of q
(which determines Tq M ) is sometimes only available at test
time, and neither the transport nor the learning can be done
offline. Our key contribution is to prove that, for a fairly
large class of important models, one never has to transport
the data. Rather, not only is it possible to transport themodel but we also prove that the resulting model is identical to what would have been gained from the expensive
data-transport-based approach mentioned above. Loosely
speaking, we prove that learning commutes with paralleltransport. Consequently, the computational complexity of
our framework is not proportional to the size of the dataset;rather, it is proportional to the complexity of the model. It
also frees us from having to store or access the training data
during test time.
Our model transport (MT) framework is straightforward
to apply yet it allows us to solve seemingly difficult problems with surprising results; e.g., we find that on a manifoldof mesh deformations [12], observations of people with normal Body-Mass Index (BMI) help modeling high-BMI people. Similarly, we show that the transported shape modelof women, learned from many examples, improves a shapemodel of men learned from only few examples. We also
show that MT is useful for classifiers of image descriptors encoded as Symmetric Positive-Definite (SPD) matrices [40]. To conclude, we present a framework that will enable researchers to apply Rn -TL ideas to manifold-valued
data while being transparently-scalable to large datasets.

2. Previous Work
Manifold-valued data are ubiquitous in computer vision. The SE(3) and SO(3) groups are omnipresent while
spherical data appear in omnidirectional images [26], normalized features [20], pre-shape spaces [21] and surface
normals [37]. Articulated poses are represented by rotations or length constraints [18, 27]. A square root of aprobability density function is a point on the unit sphere[35]. Anatomical variability is captured via certain Liegroups [15]. Diffusion-tensor images yield orientation distribution functions (e.g. [8]) or SPD matrices (e.g., [1]). The
latter are also used as image descriptors [40]. Flow patterns
are described by matrix Lie groups [23, 42] while infinitedimensional diffeomorphisms are used for nonrigid image
deformations (e.g. [2]). Kendall’s shape spaces are also
manifolds [21]. The Grassmannian captures affine shapes[3, 34, 39]. Mesh deformations enable a tractable gener-

ative framework for deformable triangular surfaces [12].
See [36, 44] for additional shape spaces.
Statistics on manifolds is often done via parametricmodels on tangent spaces, e.g. [11, 12, 17, 28, 33, 37, 39, 41],
but alternatives (not considered here) exist, e.g. [4, 10, 19,38]. Statistics on manifolds (our work included) is different from manifold learning, where a low-dimensional latent
manifold is learned from data in Rn . In contrast, here M is
known and the goal is to model statistics of M -valued data.
We do not advocate a specific manifold or a specific Riemannian metric and we do not seek to define a new tangentspace model. Rather, we strive for an efficient TL framework that applies to many manifolds and many models. We
base our method on PT but emphasize we are not the firstto use PT in statistical computer-vision tasks: PT has beenused in other, non-TL, applications; e.g. [24,25,30,39]. Our
approach differs from such work in, not only the application, but also how PT is used: We PT models, not data.
Transfer Learning: Using Rn -valued data from one
dataset in an inferential task on another is a classical TL
problem (e.g. [5]). A model learned from one set can be aprior for a model learned from the second [22]. Models canalso be learned independently and then be combined [32].
One may also use two mixture models, one per set, whichshare a component [6]. There are many other examples of
TL; a full review is beyond our scope. Surprisingly, despite
the success of TL in Rn , the ubiquity of manifold-valued
data, and the effectiveness of statistics on manifolds, Rn -TL
solutions have yet to be generalized to manifolds. Note that
our M -valued data setting should not be confused with thatof [14], where, for Rn -valued data, they exploit the geometry of a space of models. Also, in manifold alignment [16],TL is done via estimating a latent low-dimensional manifold shared by two Rn -valued datasets, while in our setting
the data lie on a known manifold, and our goal is TL acrossit. Wei et al. [42] share information across a Lie group using
PT. Particularly, they PT bases that span linear subspaces of
the Lie algebra. Unfortunately, their choice of PT does notpreserve inner-products so second-order statistics are distorted. Thus, they cannot transport covariances, and theirtransported PCA models are distorted. Our approach does
not have this problem, is applicable to more models (e.g.,
regression and classification) and does not require M to bea Lie group. Hauberg et al. [17] transport covariances ina tracking application. Our method differs in the application, applies to a broader class of models, and also sheds
new light on their procedure, by showing it can be justified
in a concrete statistical sense. Closely related to ours, is avery recent work by Xie et al. [43] who use PT for PCA
models in a shape space (whose elements are shapes, notshape transformations). Their work is not focused on MT
in general, and while they apply PT to PCA eigenvectors,they do not justify why this is optimal. Our work does not
only fill this theoretical gap but also treats PT of regressions/classifiers which are not covered in [43].

3. Mathematical Background
Assuming familiarity with basic Riemannian geometry (see [7] for an introduction), we sketch the additional
background required for our method. Henceforth, M is a
geodesically-complete Riemannian manifold of dimension
n, and d : M × M → R+ is a geodesic distance on M . If
p ∈ M , we let Expp and Logp denote its associated Riemannian exponential and logarithmic maps.

3.1. Statistics on Manifolds
We review relevant notions in statistics on manifolds [11,28]. Various concepts from classical statistics in Rn canbe generalized to M . A popular approach utilizes tanNgent spaces. Let {pi }i=1 ⊂ M . The Fréchet function,
PN+
Q : M → R : p 7→ i=1 d(p, pi )2 , generalizes the Rn notion of sum-of-squared-distances. A local minimizer ofQ (called the Karcher mean) can usually be efficiently computed [28]. Let µ denote the Karcher mean. A covarianceis defined via the covariance of the data as expressed in
1 PN
TTµ M : Cov({pi }N
i=1 Logµ (pi )Logµ (pi ) .
i=1 ) , N −1
As Tµ M ∼= Rn , PCA can be done on {Logµ (pi )}N
i=1 [11].
If the pi ’s are labeled, and {Logµ (pi )}N
i=1 are seen as theindependent variable, then we can define a simple linear regression (for R-valued labels) or a logistic regression (forbinary labels)1 . Our treatment of both cases is similar andwe may assume, without loss of generality, that the labels
are real. Such problems should not be confused with thosewhere labels are M -valued, but the independent variable isreal (e.g. [10]).

3.2. Parallel Transport
In Rn , data and models can be moved from one region
to another via ordinary translation. On M , this fails; e.g. if
x ∈ Tp M then x+q −p is rarely in Tq M . Similarly, models
cannot be naively translated; e.g. a covariance, which canbe viewed as a bilinear form over one tangent space, cannot
be simply translated to another (Fig. 1c). The same applies
for linear regression (a linear functional over a particulartangent space) and other models. However, vectors can be
moved from Tp M to Tq M using a well-known tool called
parallel transport (PT), to be defined as follows2 . Suppose
every smooth curvec : [0, 1] → M is associated with a collection of maps, Γcs,t : Tc(s) M → Tc(t) M |s, t ∈ [0, 1] ,such that: (1) Γcs,s is the identity map on Tc(s) M ; (2)Γcu,t ◦ Γcs,u = Γcs,t ; (3) Γcs,t depends smoothly on s andt. In which case, if x ∈ Tc(s) M , we call Γcs,t (x) the PT
1 Additional models, e.g. SVM [39], can be defined similarly.
2 Another way to define it is via a connection [7].

of x along c to Tc(t) M . PT provides a principled way
to move data (when expressed as tangent vectors) across
M [24, 25, 39]. In what follows, c, s, and t are either clear
from the context or their particular values are immaterial tothe discussion, and thus we use Γ as short for Γcs,t ; when it
is also clear which Γ is used, we write xe instead of Γ(x).
Criteria (1-3) are met by various collections of maps andso there exist many types of PT. A metric parallel transport (MPT) is one that preserves inner-products: hx, yip =hex, yeiq for every x, y ∈ Tp M . Thus, orthogonality and distance between vectors are preserved. This criterion too canbe met in various ways and so there exist various types of
MPT. Note that an MPT preserves second-order statistics:
when applied to Tp M -valued random variables, their variances and correlations are unchanged. Also, every MPT is
an invertible linear map; cf . [13]. Finally, given the MPT of
choice, there still remains the technical issue of how to compute it. The solution may be analytical or only numerical,
depending on the case. See also discussion in [24, 25].

4. Parallel Transport for Transfer Learning
4.1. The Euclidean Setting
Before the manifold setting, we start with a simpler Euclidean one. Consider the following typical TL tasks.
Task I: The training set consists of two Rn -valued
S NS
L
L
datasets: {xLi }N
i=1 ; {xj }j=1 ; NL > NS . Let p(x ) andSp(x ) be the unknown generating distributions. Our interest is in modeling {xSj } using either a normal distribution or
PCA. Suppose NS is sufficiently large to estimate the meanof p(xS ), but that it is too small to yield a reliable estimateof a covariance or a PCA subspace.
Task II: The training set consists of two Rn -valued
NA
B NB
datasets, {xAi }i=1 and {xj }j=1 , the xi ’s being labeled; NB
need not be smaller than NA . With yiA = label (xAi ), theyiA ’s are either binary (more generally, categorical) or real3 .
The task is either classification or regression; i.e., to predict
B
A A
B B
B
{yjB }N
1 , the labels of xj . We let p(y |x ) and p(y |x )denote the unknown conditional distributions. A possible
approach is to employ a logistic or linear regression; as themathematical treatment is similar we focus on the latter.
Solutions: In both tasks, we wish to leverage statistics
of one dataset when solving an inferential task for another.
If p(xL ) and p(xS ) (respectively, p(y A |xA ) and p(y B |xB ))are unrelated, then this would fail. Suppose, however, that
there does exist an unknown relation; e.g., p(xL ) and p(xS )may be well-modeled by two Gaussians of different means,
but with similar covariances. Likewise, the optimal linearregression models implied by p(y A |xA ) and p(y B |xB ) may
differ mostly in their intercepts while the associated hyperplanes are similar4 . Thus, in Task I, following an appropri3 A close variant is if we have some y B ’s, but their number is smaller.
j
4 In both cases, by “similar” we do not necessarily mean “identical”.

ate offset implied by the means, we may fuse the associatedGaussian or PCA models. In Task II, we may learn a modelAfrom the (xAi , yi ) pairs, and then shift the associated hyperplane to the region of Rn where the xB
j ’s reside (we may
also follow this by some adaptation).

the columns of the n × n identity matrix. If L denotes thematrix of the transported basis vectors, then the MPT can becomputed by a multiplication by L (i.e. a change of basis).
This has cost nC as we must transport n basis vectors (plusthe cost of the multiplication by an n × N matrix).

4.2. The Manifold SettingThis work considers analogous tasks but in the moregeneral and challenging manifold setting. Specifically, we
express each M -valued dataset in the tangent space at its
Karcher mean. In Task I, given two M -valued datasets,NSL{pi }Ni=1 and {qj }j=1 , letting p and q denote the respectivemeans, set xLi = Logp (pi ) and xSj = Logq (qj ). In Task
NA
A
B
and {qj }NII, given {pi }i=1
j=1 in M , set xi = Logp (pi )and xBj = Logq (qj ), reusing the symbols p and q to denotethe means (the labels are still either binary or real). The
goals are as before, but now the space of interest is Tq M ,while the statistics we hope to leverage are in Tp M . Thus
we cannot immediately use the latter in the the former.

4.3. Solutions Can Be Based on Parallel Transport– But What Objects Should We Transport?
Any type of PT can be used to move data from Tp M to
Tq M but for TL we restrict the choice to the MPT class inorder to preserve second-order statistics. More generally,
there are many ways, including non-parallel, to move datafrom Tp M to Tq M , but all create some distortion due tothe curvature of M . Without prior knowledge it is sensible
to pick the approach which distorts the data the least. Thisis provided by a particular choice of MPT, called the LeviCivita (LC) PT [7] which is widely used in computer vision. When prior information is available it may, however,be better to pick another MPT that utilizes it (see [24, 30]
for examples in image registration tasks). Our frameworkholds for all choices of MPT. Fix an MPT Γ, and let C denote the computational cost associated with transporting asingle vector. C increases with dim(M ) = n and also depends on M and Γ (particularly, it is higher when Γ is not
given in closed form). The solutions below differ in theircosts as they apply Γ to different numbers of vectors.
Solution 1 – data transport. The most direct solution
is to transport the data from Tp M to Tq M using an MPT.
This places the data from Tp M in Tq M while preserving
second-order statistics of the data. Unfortunately, the computational cost can be high. If N is the size of the dataset,L
the cost is N C; e.g., a large NL makes Γ({xLi }N1 ) expen−1
S NS
sive. Likewise, using Γ ({xj }1 ), intending to learn afused model in Tp M , will not help if a large test set in Tq M
has to be transported to Tp M in order to apply the model.
Solution 2 – basis transport. If the number of data
points N is larger than the manifold dimensionality n we
can lower the computational cost by transporting the basisof Tp M . This is done by transporting the natural basis, i.e.

4.3.1

A Third Solution – Model Transport

In practice, both solutions are typically computationally expensive. Fortunately, often there is no need to transport either the data or the basis: it suffices to transport the model.
In which case, the number of transported vectors is determined by the model complexity, and is fixed w.r.t. n and N .
Below we show how this is done. For Tasks I and II, Propositions 4.1 and 4.2 imply that after being learned in Tp M , amodel can be then transported to Tq M and we will get exactly the same results as with the expensive solutions above.
We provide the proofs in the supplemental material [13].
Proposition 4.1 (Covariance/PCA Transport). Let n0 ≥ n
0and assume M is embedded in Rn with a Riemannian met0
ric induced by Rn . Let p, q ∈ M and let {xi }N
i=1 ⊂Tp M . Let V S 2 V T be the eigen-decomposition of XX T
where X , [x1 , . . . , xN ] and let V SU T be the SVD of X,nwith {Si,i }i=1 , the diagonal entries of S, sorted in a nonincreasing order. Let [v1 , . . . , vn ] denote the columns of V .
If x ∈ Tp M , let x̃ ∈ Tq M denote the transport of x according to some MPT along a smooth curve in M from p to q.
Let X̃ , [x̃1 , . . . , x̃N ] and let Ṽ , [ṽ1 , . . . , ṽn ]. Then:(a) Ṽ SU T is the SVD of X̃. while Ṽ S 2 Ṽ T is the eigendecomposition of X̃ X̃ T .
(b) If k < n, then the k-dimensional PCA model of
k
{e
xi }N
i=1 ⊂ Tq M is given√by (the eigenvectors) {ṽi }i=1k
and (eigenvalues) {Si,i / N − 1}i=1 .
In words: (a) means that X and X̃ share singular values and right-singular vectors, but the left-singular vectors of X̃ are exactly the parallel-transported left-singular
vectors of X. The model in (b) is equivalent to firstcomputing a k-dimensional PCA model (in Tp M ) ofk{xi }Ni=1 , given√ by thek vectors {vi }i=1 and standard deviations {Si,i / N − 1}i=1 , and then transporting the {vi }ki=1
while keeping the standard deviations unchanged.
Proposition 4.1 substantially extends a previous result [17], derived in the context of the Kalman filter, whichstates that the parallel transport of a Tp M -covariance yields
some valid Tq M -covariance. Thus, Proposition 4.1 not only
provides the basis for a compact TL framework for transporting covariances and PCA models on manifolds, butalso sheds a new light on an existing tracking algorithm;
note it also provides further mathematical justification to
the method used in [43].

Our next proposition provides a similar result for simple
linear regression. But first, we need some preliminaries.
Let p and q be in an n-dimensional geodesically-complete
Riemannian manifold M . Let the inner-products on Tp M
and Tq M (implied by the Riemannian metric) be defined byh·, ·ip : Tp M × Tp M → R : (x, y) 7→ xT Ap yTh·, ·iq : Tq M × Tq M → R : (x, y) 7→ x Aq y

(1)
(2)

where Ap and Aq are SPD. Let {xi }N
i=1 ⊂ Tp M , let{yi }N⊂R
denotetheirlabels,andlet
L : Tp M 7→ Tq M
i=1denote the linear transformation associated with some MPT
along a smooth curve in M from p to q. A simple linear regression model Tp M → R has the following form:
x 7→ xT α + α0 = hx, A−1
p αip + α0 . Here α0 ∈ R,
5
while α and A−1αareregarded
as elements of Tp M . Letpli : R → R+ be a loss function associated with yi (e.g.,li : yˆi 7→ (yˆi − yi )2 is a square loss).
Proposition 4.2 (Simple-Linear-Regression Transport). Let(β, β0 ) =arg minN
X

α∈Tp M ,α0 ∈R i=1

li (xTi α + α0 ) ,(3)and set γ = Aq LA−1p β (note that γ ∈ Tq M ). Thenγ = arg minNXli ((Lxi )T δ + β0 ) .

(4)δ∈Tq M i=1

An equivalent version of Proposition 4.2 holds for alogistic-regression model (transported using the same expression we used here for γ); we omit the details. Unlike
in Proposition 4.1, in Proposition 4.2 we do not require the0presence of an ambient space Rn nor do we impose a particular metric. In fact, it is easy to prove a slightly different version of Proposition 4.1 where these restrictions are
removed. However, showing this requires more notational
clutter which might obscure the main idea.
Note that for transporting PCA, the cost is kC (k vectors are transported); for linear/logistic regression, it is C (1vector). By now it should be apparent that similar results
can be derived for many other models. The desired commutativity of learning and transport holds for any model where
the data enter the equations only via inner products; e.g., inan SVM model one may transport the support vectors.

4.4. Applying Transported Models
Henceforth we will assume Γ : Tp M → Tq M is doneLalong a geodesic curve. Let ΣL = cov({pi }Ni ) and set,by abuse of notation, ΣΓ = Γ(ΣL ). We can use ΣΓ as a
5 Formally, α and A−1 α are elements of the dual space of T M .
p
p

(a)

(b)

(c)

Figure 2: Mean shapes. These correspond to the Karcher
means computed from 1000 female shapes (a), 50 maleshapes (b), and 50 shapes of women with high-BMI (c).
Scovariance on Tq M or, letting ΣS = cov({qj }Nj ), we may
combine them; e.g. using shrinkage estimation [32]:Σλ , λΣΓ + (1 − λ)ΣS , 0 ≤ λ ≤ 1 .

(5)

For PCA, let VL ⊂ Tp M and VS ⊂ Tq M denote thefirst kL eigenvectors of ΣL (where kL ≤ NL ) and first kS
eigenvectors of ΣS respectively (we take kS = NS ), andset VΓ = Γ(VL ) ⊂ Tq M . Similarly, let σL2 ∈ RkL andσS2 ∈ RkS denote the eigenvalues. We can now use (VΓ , σL )as a PCA model in Tq M , or combine it with VS ; e.g., letVF ⊂ Tq M denote an orthonormalized version of [VΓ , VS ].
VF , which we regard as a fused model, contains kL + kS
vectors and is able to generalize better than VS as it also
contains the transported variation from Tp M . To enable a
direct comparison with VL or VΓ , we can also restrict the
combined model to have the same dimensionality by using
only kL vectors. Let X = [Logq (q1 ), .., Logq (qNS )], define
Xλ , [λVΓ σL , (1 − λ)X] , 0 ≤ λ ≤ 1 ,(6)and let Vλ represent the kL -dimensional PCA subspace of
Xλ . This is simply weighted PCA, where we treat vectors
in VΓ as examples weighted by the standard deviations. In
both Eqs. (5) and (6), the larger λ is, the stronger is theLinfluence of {pi }N
i . We think of this influence as regularization. The value of λ may be chosen by cross-validation.
Transported regression models or classifiers can be utilized in a similar way. More generally, a transported model
can either be applied as is in Tq M , or it can be adapted (orfused with a Tq M -model) using Rn -TL methods.

5. Results
MT is applicable on many manifolds; here we experiment with two of these, using only real, non-synthetic data.

5.1. PCA Transport and Shape Deformations
For Task I, let M be the manifold of triangular-mesh deformations, proposed in [12]. Points on M are deformations of 3D shapes from a template mesh. M is a Lie group(though it is not a requirement for MT) made out of 21550

Ground
Truth

Figure 3: Summary for shape experiments. Left: Gender.
Right: BMI. The bars represent the overall reconstruction
error for VL , VS , VΓ , and VF . For a given model, the heightof the bar represents the reconstruction error measured interms of SGE averaged over the entire test dataset as well
as all of the mesh triangles.

VL(Women)VS(Men)(a) VL(b) VS(c) VΓ(d) VF(e) VλFigure 4: Model mean error: Genders. Blue and red indicate small and large errors respectively. The heat maps
are overlaid over the points of tangency associated with themodels: p for (a), and q for (b-e). See text for details.

VΓ
(PT)

copies of a 6-dimensional Lie group, which is isomorphic
to the product of three smaller ones, including SO(3); thus,n = 129300. While here we do not advocate a particular
manifold nor does our work focus on shape spaces, this M
enables us to easily demonstrate the MT framework. The
data consist of aligned6 3D scans of real people [31]. On
this M , the LC PT is computed as follows: For the SO(3)components of M , a closed-form solution is available [9],while for the rest we use Schild’s ladder (see, e.g., [17,24]).
From Venus to Mars. We first illustrate the surprising power of MT. The training data contains NL = 1000
shapes of women (Fig. 1a, red; shown here on a 2D manifold for illustration) but only NS = 50 shapes of men(blue), where all shapes are represented as points on M .
As it is reasonable to expect some aspect of shape variation
among women may apply to men as well, we model the
shape variation of men while leveraging that of women. We
first compute the Karcher means for women and men denoted p and q, respectively (Fig. 2a–2b). We then compute
their PCA models, VL ⊂ Tp M and VS ⊂ Tq M (kL = 200
and kS = 50), as well as VΓ = Γ(VL ). For an animatedillustration see [13]. We also compute VF and Vλ using
the procedures from Sec. 4.4. We evaluate performance on

VF(Fuse)

6 MT also applies to some shape spaces that do not require alignment.

Figure 5: Selected results: Gender. Each column represents
a different test body. The heat maps are overlaid on thereconstructions using different models.

(a) VL(b) VS(c) VΓ(d) VF(e) VλFigure 6: Model mean error: BMI. Analogous to Fig. 4.

1000 test male shapes, whose deformations serve as groundtruth. Let V ∈ {VL , VS , VΓ , VF , Vλ }. Let µ denote the pointof tangency; i.e., p for VS and q otherwise. Let zi ∈ Mdenote the true deformation of test example i. Its reconstruction is Expµ (V V T Logµ (zi )) ∈ M . We then com-pute, for each triangle, the Squared Geodesic Error (SGE)
between the reconstruction and the true deformation. Fixing i, SGE is averaged over all body triangles, yielding theMean SGE (MSGE) of the ith body. Overall performanceof V is defined by averaging MSGE over all test examples.
MSGE results are summarized in Fig. 3 (left). To visualize,
we average the SGE, per triangle, over all test examples,and display these per-triangle errors over the mesh associated with µ (Fig. 4). Figure 4a shows that VL performs very
poorly; a shape model of women fails to model men. Whilethe errors for VS are much lower (Fig. 4b), there are still
noticeable errors due to poor generalization. The surpriseis Fig. 4c, which shows the result for VΓ : the PT dramatically improves the female model (Fig. 4a) to the point it
fares comparably with the male model (Fig. 4b), although
the only information used from the male data is the mean.
Combining transported and local models lets us do even better. Figure 4d shows that VF significantly improves over VSor VΓ . Figure 4e shows the regularized model, Vλ , which
has the same dimensionality as VL and still performs well.
Figure 5 shows selected results for test bodies; see [13] for
additional results and reconstructions.
From Normal-Weight to Obesity. A good statisticalshape model of obese women is important for fashion andhealth applications but is difficult to build since the data arescarce as reflected by their paucity in existing body shapedatasets [31]. This experiment is similar to the previous
one, but both the data and the results are of different nature. Here, we have 1000 shapes of women with BMI ≤ 30
but only 50 shapes of women with BMI > 30. We compute means and subspaces as before. Figure 2c shows q, thehigh-BMI mean; p, the normal-BMI mean, is not shown as
it is very similar to p from the gender experiment. Figures
3 (right) and 6 summarize the results. Compared with thegender experiment there are two main differences: 1) HereVΓ is already much better than VS so fusion only makes asmall difference. 2) Error bars (Fig. 3, right) are larger than
before (Fig. 3, left) due to the limited amount of test data
available for high-BMI women; this is truly a small-sample
class: we were able to obtain only 50 test examples. Compared with using VS , reconstruction is noticeably improved
using our method (VF ). In both experiments, results for Vλlook nearly identical to VF , and are not shown. See [13] forindividual reconstruction results.

5.2. Classification Transport and Image Descriptors
7

For Task II, our data consist of facial images and thegoal is binary facial-expression classification. Images aredescribed by SPD matrices that encode normalized correlations of pixel-wise features [40]. Each quarter of an image is described by a 5 × 5 SPD matrix, yielding an image descriptor in M = SP D(5)4 . PT is computable by
7 From www.wisdom.weizmann.ac.il/

˜vision/FaceBase

Figure 7: Classifier-transport example. Select images. Top:First data set. Bottom: Second data set. In each row, examples from class 1 (left) and class 2 (right) are shown.
Schild’s ladder, M is not a Lie group8 and n = 60. The
B
A
reflect two different viewingand {qj }Ndatasets {pi }N
j
i
A
are
directions; NA = NB = 168. The labels of {pi }NiNBknown, those of {qj }j withheld. See Fig. 7 for examples.
We compute p and q, the means of the datasets. Then, usA⊂ Tp M , we learn a logistic-regression
ing {Logp (pi )}Nimodel. This classifier, defined on Tp M , is correct 59% ofB
the time when applied to {Logp (qj )}N⊂ Tp M . Applyj
NB
ing the transported model to {Logq (qj )}j ⊂ Tq M improves performance to 67%. Thus, for the same unannoBtated {qj }N
j , MT improves over the baseline. Note we had
to PT only one vector; even for such a small dataset the
speed gain is already significant.

6. Conclusion
Our work is the first to suggest a framework for generalizing transfer learning (TL) to manifold-valued data. As
is well-known, parallel transport (PT) provides a principled
way to move data across a manifold. We follow this reasoning in our TL tasks, but rather than transporting data wetransport models – so the cost does not depend on the size ofthe data – and show that for many models the approaches areequivalent. Thus, our framework naturally scales to large
datasets. Our experiments show that not only is this mathematically sound and computationally inexpensive but alsothat in practice it can be useful for modeling real data.
Acknowledgments This work is supported in part by
NIH-NINDS EUREKA (R01-NS066311). S.H. is supported in part by the Villum Foundation and the Danish
Council for Independent Research (Natural Sciences).

References[1] V. Arsigny, P. Fillard, X. Pennec, and N. Ayache. Logeuclidean metrics for fast and simple calculus on diffusion
tensors. MR in medicine, 56(2):411–421, 2006. 2[2] M. F. Beg, M. I. Miller, A. Trouvé, and L. Younes. Computing large deformation metric mappings via geodesic flows ofdiffeomorphisms. IJCV, 61(2):139–157, 2005. 2
8 Since SPD is not a matrix Lie group. While not used here, some Liegroup structure can still be imposed to get a nonstandard matrix Lie group;i.e., the binary operation will not be the matrix product.

[3] E. Begelfor and M. Werman. Affine invariance revisited.
CVPR, 2:2087–2094, 2006. 2[4] A. Bhattacharya and R. Bhattacharya. Nonparametric inference on manifolds: with applications to shape spaces. Cambridge University Press, 2012. 2[5] H. Daumé III. Frustratingly easy domain adaptation. In ACL,
256–263, 2007. 2[6] H. Daumé III and D. Marcu. Domain adaptation for statistical classifiers. JAIR, 26(1):101–126, 2006. 2[7] M. Do Carmo. Riemannian geometry. Birkhäuser Boston,
1992. 1, 3, 4
[8] J. Du, A. Goh, S. Kushnarev, and A. Qiu. Geodesic regression on orientation distribution functions with its application
to an aging study. NeuroImage, 2013. 2[9] A. Edelman, T. Arias, and S. Smith. The geometry of algorithms with orthogonality constraints. SIMAX, 20(2):303–
353, 1998. 6[10] P. Fletcher. Geodesic regression and the theory of leastsquares on Riemannian manifolds. IJCV, 1–15, 2012 2, 3
[11] P. Fletcher, C. Lu, S. Pizer, and S. Joshi. Principal geodesic
analysis for the study of nonlinear statistics of shape. IEEE
TMI, 23(8):995–1005, 2004. 2, 3
[12] O. Freifeld and M. Black. Lie bodies: A manifold representation of 3D human shape. ECCV, 1–14, 2012. 1, 2, 5
[13] O. Freifeld, S. Hauberg, and M. Black. Model Transport:
Towards Scalable Transfer Learning on Manifolds – Supplemental material. MPI-IS-TR-009, 2014. 3, 4, 6, 7
[14] B. Gong, Y. Shi, F. Sha, and K. Grauman. Geodesic flow
kernel for unsupervised domain adaptation. CVPR, 2066–
2073, 2012. 2[15] U. Grenander and M. Miller. Computational anatomy: An
emerging discipline. QAM, 56(4):617–694, 1998. 2[16] J. Ham, D. Lee, and L. Saul. Semisupervised alignment of
manifolds. In UAI, 10:120–127, 2005. 2[17] S. Hauberg, F. Lauze, and K. Pedersen. Unscented Kalmanfiltering on Riemannian manifolds. JMIV, 1–18, 2012. 2, 4,
6[18] S. Hauberg, S. Sommer, and K. S. Pedersen. Natural metrics and least-committed priors for articulated tracking. IVC,
30(6-7):453–461, 2012. 1, 2
[19] S. Huckemann, T. Hotz, and A. Munk. Intrinsic shape analysis: Geodesic PCA for Riemannian manifolds modulo isometric Lie group actions. Stat. Sinica, 20:1–100, 2010. 2[20] Y. Ke and R. Sukthankar. PCA-SIFT: A more distinctive
representation for local image descriptors. CVPR, 2:506–
513, 2004. 2[21] D. Kendall. Shape manifolds, Procrustean metrics, and complex projective spaces. BLMS, 16(2):81–121, 1984. 2[22] X. Li and J. Bilmes. A bayesian divergence prior for classifier adaptation. In AISTATS, 2007. 2[23] D. Lin, E. Grimson, and J. Fisher III. Learning visual flows:
A Lie algebraic approach. CVPR 747–754, 2009. 2[24] M. Lorenzi, N. Ayache, and X. Pennec. Schild’s ladder forthe parallel transport of deformations in time series of images. In IPMI, 463–474, 2011. 2, 3, 4, 6

[25] M. Lorenzi and X. Pennec. Geodesics, parallel transport &one-parameter subgroups for diffeomorphic image registration. IJCV, 1–17, 2012. 2, 3
[26] A. Makadia and K. Daniilidis. Direct 3d-rotation estimation from spherical images via a generalized shift theorem.
CVPR, 2003. 2[27] R. Murray, Z. Li, and S. Sastry. A mathematical introduction
to robotic manipulation. CRC, 1994. 2[28] X. Pennec. Probabilities and statistics on Riemannian manifolds: Basic tools for geometric measurements. In NSIP,
194–198, 1999. 2, 3[29] X. Pennec, P. Fillard, and N. Ayache. A riemannian framework for tensor computing. IJCV, 66(1):41–66, 2006. 1[30] X. Pennec and M. Lorenzi. Which parallel transport for thestatistical analysis of longitudinal deformations? In Colloque GRETSI’11, 2011. 2, 4[31] K. Robinette, S. Blackwell, H. Daanen, M. Boehmer,
S. Fleming, T. Brill, D. Hoeferlin, and D. Burnsides. Civilian
American and European Surface Anthropometry Resource.
AFRL-HE-WP-TR-2002-0169, US AFRL, 2002. 6, 7
[32] J. Schäfer and K. Strimmer. A shrinkage approach to largescale covariance matrix estimation and implications for functional genomics. SAGMB, 4(1), 2005. 2, 5[33] S. Sommer, F. Lauze, S. Hauberg, and M. Nielsen. Manifold
valued statistics, exact principal geodesic analysis and theeffect of linear approximation. ECCV, 43–56, 2010. 2[34] G. Sparr. Structure and motion from kinetic depth. In The
Sophus Lie International Workshop on CVAM, 1995. 2[35] A. Srivastava, I. Jermyn, and S. Joshi. Riemannian analysis
of probability density functions with applications in vision.
CVPR, 1–8, 2007. 2[36] A. Srivastava, S. H. Joshi, W. Mio, and X. Liu. Statistical shape analysis: Clustering, learning, and testing. IEEE
PAMI, 27(4):590–602, 2005. 2[37] J. Straub, G. Rosman, O. Freifeld, J. Leonard, and
J. Fisher III. A mixture of Manhattan frames: Beyond the
Manhattan world. CVPR, 2014. 2[38] R. Subbarao and P. Meer. Nonlinear mean shift over riemannian manifolds. IJCV, 84(1):1–20, 2009. 2[39] S. Taheri, P. Turaga, and R. Chellappa. Towards viewinvariant expression analysis using analytic shape manifolds.
In AFGR, 306–313, 2011. 1, 2, 3[40] O. Tuzel, F. Porikli, and P. Meer. Region covariance: A fast
descriptor for detection and classification. ECCV, 589–600,
2006. 2, 7
[41] M. Vaillant, M. I. Miller, L. Younes, and A. Trouvé. Statistics on diffeomorphisms via tangent space representations.
NeuroImage, 23:S161–S169, 2004. 2[42] D. Wei, D. Lin, and J. Fisher III. Learning deformations withparallel transport. ECCV, 287–300, 2012. 2[43] Q. Xie, S. Kurtek, H. Le, and A. Srivastava. Parallel transportof deformations in shape space of elastic surfaces. ICCV,2013. 2, 3, 4[44] L. Younes. Spaces and manifolds of shapes in computer vision: An overview. IVC, 30(6):389–397, 2012. 2