Learning a Manifold of Fonts
Supplemental MaterialA

Neill D.F. Campbell1

Jan Kautz1,2

1 University College London

2 NVIDIA Research

Fonts Used
Andale Mono
Aparajita
Calibri
ConsolasCordia New
EbrimaGautamiKalinga
MS PMincho
Microsoft Tai Le
MoolBoran
PMingLiU
Plantagenet Cherokee
SylfaenTraditional Arabic
Verdana

Andalus
Arial
Cambria
Constantia
DaunPenh
Euphemia
Georgia
Lao Sangam MN
Microsoft Himalaya
MiriamNarkisim
Palatino Linotype
Shonar Bangla
TahomaTrebuchet MS

Angsana New
Browallia New
CandaraCorbel
David
Franklin Gothic Book
Gill Sans MT
Lucida Sans Unicode
Microsoft Sans SerifMongolian Baiti
Nyala
Perpetua
SimSun-ExtB
Times New Roman
VaniTable 1: The fonts used in our experiments.
We took the ‘ttf’ files from the fonts folder on a ‘Windows 7’ installation and removed any non-latin or symbol based fonts. We
also removed a few fonts that were topologically inconsistent overmany letters, for example handwriting based fonts; if only a few
letters are inconsistent, e.g.’g’, we omitted the inconsistent characterand embedded the font with missing data so that the manifold fills
in unmatched character [Navaratnam et al. 2007]. Our training set
includes examples of many different categories of typefaces (e.g.
Humanist, Garalde, Didone/Modern, Slab, Lineale/Grotesque, andGlyphic) for a total of 46 fonts listed in Table 1.

B

Details on the GP-LVM

Here we provide a more in-depth introduction to Gaussian Processes,illustrating their use for regression. We then extend this techniqueto present the Gaussian Process Latent Variable Model (GP-LVM)
that we use to construct probabilistic manifolds. Finally, we provide
the details about how to learn the true dimensionality of a manifoldfrom the data.

B.1

Gaussian Processes

Gaussian Processes (GPs) are a non-parametric model that consistof distributions over functions. They can be used as a powerful
way to model regression, that producing a smooth mapping function from an input space to an output space (both of which maybe
multi-dimensional vector spaces); essentially they act as a prior thatencourages functions to be smooth. A full treatment is beyond thescope of this work and instead we provide a basic illustrative example; there are many excellent texts on GPs including the definitive
book by Rasmussen and Williams [2006]. We begin by considering
using GPs for the problem of regression and then show how we maytreat the problem of finding a manifold (or probabilistic dimensionality reduction) as a new take on the regression task, forming theGP-LVM.
For the illustration of the regression task we will use
the following notation. Suppose that we have a set of training
data pairs {x j , y j } that associate an input vector x j ∈ R Q with its
corresponding output vector y j ∈ R D with respective dimensions
Notationof Q and D. If we need to consider multiple sets of these vectors
we stack them together row-wise to produce a matrix, for example
X = [· · · x j · · · ]T and Y = [· · · y j · · · ]T .
In the task of regression, we wish to identify afunction f (·) that maps the input to the output space so that we cantake a new input x∗ and find the corresponding output as y∗ = f (x∗ ).
Now, if we take a GP it has the property that any discrete set ofsamples taken from the GP will be normally distributed. Thus if
the GP is evaluated at a series of values X = [x1 · · · xJ ]T from ourinput space, the corresponding output values Y = [y1 · · · yJ ]T will be
distributed asGP Regression

P (Y |X) = N (Y |M(X),C(X, X))(B-1)

where
11
N (x|µ, Σ) = k2πΣk− 2 exp − (x − µ)T Σ−1 (x − µ)2(B-2)

is the standard multivariate Gaussian distribution with M(·) as amean function and C(·, ·) as a covariance function. The mean function generates a vector from the input X and the covariance function
creates a (positive semi-definite) covariance matrix, these vectors
each have the same number of dimensions as there were examples
given (in this case J). Thus the GP handles an infinite object (afunction defined over a continuous space) by considering only afinite set of input values at a time (J values in our example) andreturning the corresponding finite set of output values.
Now we know about the distribution
between the input and output spaces of a GP we can make it usefulfor regression by conditioning the GP on our training data. That isto say that we add the constraint that if any of the training inputs Xtr
are presented to the GP, the corresponding training output Ytr should
be produced. Thus if we consider a new test input x∗ , we can findy∗ = f (x∗ ) as 
 

Ytr
M(Xtr )
C(Xtr , Xtr ) C(Xtr , x∗ )
∼N,.
(B-3)
∗∗∗∗∗yM(x )C(x , Xtr ) C(x , x )Condition on Training DataWe simplify the situation by subtracting the mean from our training
input and output so that the mean function becomes a zero-vector.
Standard manipulation can convert this to an output function as
f (x∗ ) ∼ N C(x∗ , Xtr )C(Xtr , Xtr )−1Ytr , Σ∗(B-4)
with the output covariance (‘error bars’) asΣ∗ = C(x∗ , x∗ ) −C(x∗ , Xtr )C(Xtr , Xtr )−1C(x∗ , Xtr ) .

(B-5)

Therefore, the conditioned GP provides the regression function
to predict output vectors for new input vectors with the predictedvalue as the mean of (B-4) and the covariance value Σ∗ providing ameasure of confidence in the prediction; a low standard deviation
would indicate a very precise estimate. Please see [Rasmussen andWilliams 2006] for further details.
The final part of the GP regression
model is then to specify the covariance function C(·, ·). This function must accept one or more samples from the input space as eachargument and return a covariance value between each pairwise combination of samples. The function can depend on a small numberof ‘hyperparameters’ that we shall denote with θ . The elements aparticular covariance matrix given by a set of inputs are indexed
The Covariance Function

using the notation for representing a set of vectors described above.
ThushiC([· · · xi · · · ]T , [· · · x j · · · ]T |θ )= c xi , x j |θ
(B-6)
i, j

where [·]i, j denotes the element at the ith row and jth column of thematrix and c(xi , x j |θ ) denotes the covariance between two inputvectors.
Perhaps the most typical covariance function would be a radial basis
function (RBF)
1c xi , x j |θ = α exp − ψ kxi − x j k2(B-7)
2where α and ψ are the hyperparameters and therefore θ = [α, ψ].
We can see that this covariance function will produce a smooth
mapping from the input space to the output space since nearby input
vectors will have a high covariance and therefore the output vectors
will be correlated. Similarly, if the input vectors are far apart (relativeto the length -scale parameter ψ) then there will be a low covarianceand the corresponding output vectors will be independent of one
another.
The training of a GP consists of learning the values for these hyperparameters by maximizing the log-likelihood (B-1) of the training
data (Xtr ,Ytr ) to give an optimal setting θ ∗ . Since the covariancefunction can be a powerful, non-linear function, as in the case ofthe RBF, this training cannot be performed analytically and is instead performed using gradient based methods such as Conjugate
Gradients.

B.2

and we require a trade-off between producing a perfect reconstruction of the data and a smooth and useful manifold. With this in mind,
a noise model is usually added to the covariance matrix during training to account for any slight noise in the high dimensional vectors.
We therefore replace the basic covariance function of (B-7) with
1c0 xi , x j |θ = α exp − ψ kxi − x j k2 + σ 2(B-8)
2where σ is the standard deviation of the noise. The noise term is only
present during training and is not used afterwards when generatinghigh dimensional vectors using (B-4).
In our experiments, the noise variation was found to be very small,suggesting that the font outlines do indeed lie on a low dimensional
manifold.
The training process for the GP-LVM considers the likelihood of the high dimensional data Y asTrainingMP(Y | X, θ ) = ∏ N ym | 0,C(X, X|θ ) + σ 2 I(B-9)

m=1where there are M training examples (one for each font) and I is theidentity matrix. Whereas, for the GP model we simply maximized
the log-likelihood over the hyperparameters θ , for the GP-LVM training we must maximize jointly over the latent vectors X = [· · · x j · · · ]Tas well as θ such thatX ∗ , θ ∗ = arg max log [P(Y | X, θ )] .
X,θ(B-10)The GP-LVM

We can think of the task of dimensionality reduction for a set ofhigh dimensional vectors as identifying a set of low dimensional
vectors that recreate the corresponding high dimensional vectors
under a basic mapping. More powerfully, we could then go furtherto consider a manifold model that defines a generative mapping
function, as well as the low dimensional vectors, such that newlocations in the low dimensional space will be mapped to novel but
plausible output values.
With this in mind, we returnto the GP model that we have established as a powerful, non-linear
regression technique. Suppose that no longer have pairs of input andoutput training examples, but instead only a set of high dimensional
vectors; we move from supervised learning to unsupervised learning.
If we treat these high dimensional vectors (the font outlines in ourcase) as the output of a GP regression model, then we observe thatfinding a manifold model is the same as finding a set of corresponding, low dimensional input vectors and the hyperparameters for thecovariance function (the points on the manifold and the mapping).
Manifold Modelling as Regression

The GP-LVM, therefore, may be thought of as a GP regression modelwith the regression input and output operating the other way around.
The input to the GP-LVM model is a set of high dimensional data{y j } ∈ R D (the regression output) and we find corresponding low
dimensional data {x j } ∈ R Q (the regression input), with Q  D,and the hyperparameters θ . The low dimensional space is also
termed the ‘latent’ space since it is unobserved and must be inferred.
Once we have performed this learning task, the manifold model thenbecomes a GP regression process, as in (B-4) where any locationin the low dimensional space (a point on the font manifold) will
be mapped into a new high dimensional vector (a new font outline)with the variance value indicating the likelihood of the generated
high dimensional vector in terms of the training fonts provided.
Representing a set of very high dimensional vectors
by a low dimensional manifold is obviously a challenging process
Noise ModelAgain, the covariance function is a complex, non-linear function,
both in terms of its relationship to X and to θ ; however, gradients
may be found against both so training may be performed using
Conjugate Gradients [Lawrence 2005]. Such methods require initialvalues for both X and θ . As is standard [Lawrence 2005], we
initialize the latent values with a linear PCA reduction of the highdimensional vectors Y and set the hyperparameters to conservativevalues with a wide uninformative prior.

B.3

Manifold Dimensionality

For the purpose of visualization, we have limited the manifold dimensionally in the figures (and the interactive javascript viewer) to
Q = 2. In reality the dimensionality of the manifold may be higher.
In order to identify the true dimensionality of the manifold, we useda more advanced derivate of the GP-LVM, the Bayesian GP-LVMof Titsias and Lawrence [2010]. Their approach uses the ‘Automatic
Relevance Determination’ (ARD) covariance function!
1 Q2
card xi , x j |θ = α exp − ∑ ψq (xi,q − x j,q )(B-11)
2 q=1
instead of the standard RBF kernel of (B-7). Comparing the RBF
and ARD kernels we see that the ARD kernel has a different lengthscale parameters for each dimension, {ψq }. The Bayesian GP-LVM
marginalizes out, rather than optimize, the latent space X. This
allows us to set Q to a large value, run the learning process, and thenlook at the values of each ψq . If a dimension is unnecessary, wewill have the corresponding value of ψq → 0. Thus, we count thenumber of non-zero length-scales to determine how many dimensions are useful. For the joint manifold, shown in Figure 1, the best
setting is obtained with Q = 4 manifold; this explains the separation
into islands when constrained to Q = 2. Manifolds for individualcharacters may differ in intrinsic dimensionality, for example thecharacter ‘g’ has Q = 3.

References
L AWRENCE , N. 2005. Probabilistic non-linear principal component
analysis with gaussian process latent variable models. The Journal ofMachine Learning Research 6, 1783–1816.
NAVARATNAM , R., F ITZGIBBON , A., AND C IPOLLA , R. 2007. The
joint manifold model for semi-supervised multi-valued regression. InIEEE 11th International Conference on Computer Vision.
R ASMUSSEN , C. E., AND W ILLIAMS , C. 2006. Gaussian Processes
for Machine Learning. MIT Press.
T ITSIAS , M., AND L AWRENCE , N. 2010. Bayesian gaussian processlatent variable model. In 13th International Workshop on AI and Stats.