GlyphGAN: Style-Consistent Font Generation Based on
Generative Adversarial Networks
Hideaki Hayashia,∗, Kohtaro Abea , Seiichi UchidaaaDepartment of Advanced Information Technology, Kyushu University, Fukuoka, Japan

Abstract
In this paper, we propose GlyphGAN: style-consistent font generation basedon generative adversarial networks (GANs). GANs are a framework for learning a generative model using a system of two neural networks competing witheach other. One network generates synthetic images from random input vectors, and the other discriminates between synthetic and real images. The
motivation of this study is to create new fonts using the GAN framework
while maintaining style consistency over all characters. In GlyphGAN, theinput vector for the generator network consists of two vectors: characterclass vector and style vector. The former is a one-hot vector and is associated with the character class of each sample image during training. The
latter is a uniform random vector without supervised information. In thisway, GlyphGAN can generate an infinite variety of fonts with the characterand style independently controlled. Experimental results showed that fonts
generated by GlyphGAN have style consistency and diversity different fromthe training images without losing their legibility.
Keywords: Font generation, generative adversarial networks, styleconsistency, deep convolutional neural network
1. Introduction
There is a variety of fonts in the world. As shown in Fig. 1, fonts arecharacterized by various components such as the thickness of lines, decoration, and serifs. There are also handwritten-like fonts, fonts made of outlines,∗

Corresponding author
Email address: hayashi@ait.kyushu-u.ac.jp (Hideaki Hayashi)

Preprint submitted to Knowledge-Based Systems

August 6, 2019

Figure 1: Examples of various fonts. Texts denote the name of the font.

fonts with lowercase letters capitalized, and so on. Among these fonts, thebest ones will be chosen according to the medium such as books, newspapers, signboards, and web pages. Even for the same medium, different fonts
can be used depending on the title, text, and speakers. In response to thesedemands, a large number of fonts have been created.
This study aims at the automatic design of fonts; a computer automatically generates various fonts instead of a human designing fonts individually.
There are two reasons why we aim at automatic design even though a large
number of fonts already exists.
The first aim is to reduce the labor for creating a new font. Even today,
new fonts are still being created. When a font is created, a large number ofcharacters with the same style should be designed. In the case of alphabetic
fonts, not only 52 upper/lowercase letters but symbols also are designed.
For Japanese fonts, the labor increases because the Japanese language has alarge number of letters including hiragana, katakana, and kanji. Therefore,
automatic font design can potentially reduce this labor by a large extent.
The second aim is to understand designers’ tacit knowledge via a constructive approach. Basically, fonts are created by designers individually.
This know-how is fundamentally cultivated by the designer’s experience and2

is not easily systematized. Reproduction of the process where a designer
becomes able to create new fonts will lead to new knowledge of character
design.
To realize the above aims, the following approaches can be considered.
1. Designing all characters from a few samples: After manually designinga few examples as templates, the system automatically designs theremaining characters using these templates. This approach is effective,
particularly for character sets with many of the same parts such as
Chinese characters.
2. Transformation and interpolation: A new font is made of existing fonts
via operations such as changing the thickness of lines, adding decorations, and calculating an interpolation of two fonts. This approach hasdifficulty in designing a completely novel font because the generated
font depends on the original font.
3. Generating fonts automatically using machine learning: Utilizing alarge number of fonts, a computer is trained to learn the design principle. If the computer can learn the designer’s know-how, which isdifficult to describe explicitly, then automatic font design with a highdegree of freedom is realized.
Studies related to the above approaches are described in the next section.
This study focuses on approach #3, i.e., machine learning-based font generation. This approach mainly includes two methods: transformation-based
and generative model-based methods. In the former, a font is generated byadding style information to the existing font [2, 28, 21]. The latter estimatesthe manifold that the existing fonts compose in the image space, and thengenerates new font images by sampling data from the estimated manifold[4, 5]. This second method has the potential to generate more diverse fonts
although there are challenges in the manifold estimation and generation stability.
Related to the generative model-based method, generative adversarial
networks (GANs) [10] have attracted much attention in terms of image generation. GANs are a framework for learning a generative model using a systemof two neural networks competing with each other. One network generates
synthetic images from a random input, and the other discriminates between
synthetic and real images, thereby allowing the generation of highly realistic
images. However, it is basically difficult to control the characteristics of thegenerated images using GANs because GANs generate images from random
3

input. Considering the application to font generation, the generated fontshould have the same style for all characters.
In this paper, we propose a font generation method based on GANs,
which is named GlyphGAN. In GlyphGAN, the input vector for the generator network consists of two vectors: character class vector and style vector.
The former is a one-hot vector and is associated with the character class ofeach sample during training. The latter is a uniform random vector without supervised information. In this way, GlyphGAN can generate an infinite
variety of fonts with the character and style independently controlled.
The main features of the proposed GlyphGAN are as follows:• Style consistency: GlyphGAN can generate a font that has the samestyle over all characters.
• Legibility: The generated fonts are legible compared with the othermethods.
• Diversity: The generated fonts have diversity different from the training images.
2. Related Work2.1. Example-based Font Generation
Various attempts have been made in previous studies on automatic fontdesign. One of the classical methods is example-based font generation [27,15, 23, 31]. For example, “A” is generated from human-designed “B,” “C,”. . ., “Z.” Tenenbaum and Freeman [27] used several font sets containing allalphabets to separate them into standard shapes of individual alphabets andfont styles. Then, the style of example patterns was extracted by using thestandard shape and applied to the other alphabets. Miyazaki et al. [23]proposed an automatic typographic font generation method based on theextraction of strokes from a subset of characters. Yang et al. [31] proposeda patch-based method to transfer heavy decoration from an example image
to others. Phan et al. [25] developed a system called FlexyFont for synthesizing a complete typeface with a small set of glyphs using a part-assembly
approach.

4

2.2. Transformation and Interpolation
Some studies attempted font generation based on the transformation andinterpolation of existing fonts. Wang et al. [30] employed stroke-level transformation for generating Chinese letters. A new font can also be generated byinterpolating multiple fonts [6, 29]. Campbell and Kautz [6] obtained a manifold of fonts by learning nonlinear mapping that can be used to smoothly interpolate between existing fonts. Uchida et al. [29] analyzed the distributionof fonts using a large-scale relative neighborhood graph, and then generated
new fonts by using a contour-based interpolation between neighboring fonts.
2.3. Machine Learning
Nowadays, various machine learning techniques are used for generating
fonts. A recent trend is GAN-based methods, which will be reviewed subsequently. Neural font style transfer [2] is an example of font generation using
deep learning. This method transfers the style of one image to another input
image using the features extracted from the intermediate layers of a convolutional neural network (CNN), inspired by the idea of neural style transfer[9]. In the neural font style transfer, various types of images are used as astyle image such as textures and fonts of different languages from the input,
thus expanding the possibility of font design by deep learning.
Other machine learning techniques have also been used for font or character pattern generation. Lake et al. [15] proposed an interesting way togenerate handwritten patterns by Bayesian program learning. This approachinfers a rule to draw an example pattern and then applies the rule to generate
new patterns. Baluja [3] used a CNN-like neural network that is originallytrained for font type discrimination. The neural network outputs a single
letter or all alphabet letters from a limited number of examples. Lian et al.
[17] proposed a system that automatically synthesizes personal handwriting
for all characters in the font library by learning style from a small numberof handwritten samples. Li et al. [16] used Adaboost composed of support
vector regressions for evaluating topological features in their method for synthesizing Chinese calligraphy. Gao et al. [8] proposed a learning system forcalligraphy robots to autonomously learn to write Chinese strokes using a
convolutional auto-encoder and differential evolution.
2.4. Font Generation by GANs
Recently, there are several attempts that utilize GANs for font generation. The zi2zi [28] is a method that converts a certain font pattern into5

a target font pattern based on a combination of pix2pix [13] and AC-GAN[24]. Although the generated fonts have sharp outlines and include a variety
of styles, the target font is restricted to those having a large number of character types such as Japanese, Chinese, and Korean. Therefore, it is difficult
to apply this method to alphabets composed of few letters.
Many other methods have also been proposed. They claimed that their
method is easier in balancing the loss functions than zi2zi. Lyu et al. [21]used a GAN along with a supervised network, which is an autoencoder that
captures the style of a target calligrapher. Lin et al. [18] proposed a strokebased font generation method where two trained styles can be interpolated
by controlling a weight. Liu et al. [19] proposed a multi-scale multi-classgenerative adversarial network for Chinese handwritten characters generation.
The main difference between the above GAN-based font generation methods and the proposed GlyphGAN lies in the way of providing input. The
above methods are based on image-to-image transformation, where a new
font is generated by adding style information to character class information
extracted from a reference character image given as an input. This approachallows for a large number of character classes, whereas the generated fontpotentially depends on the shape of the input image; hence the generationof a completely novel font is difficult. Different from such an approach,
GlyphGAN employs only abstracted inputs as vectors, thereby allowing thegeneration of fonts not seen in the training image. Although it is difficult
to maintain legibility and style consistency in this approach, we manage toimprove these important natures by embedding both the character ID andstyle information into the latent vector and introducing the loss function of
the Wasserstein-GAN gradient penalty.
3. Preliminary Knowledge of Generative Adversarial Networks
3.1. Overview
GANs are a framework for estimating a generative model composed of
two neural networks called the generator G and the discriminator D. The
generator takes a vector of random numbers z as an input, and produces datawith the same dimensions as the training data. Meanwhile, the discriminator
discriminates between samples from real data and data generated by thegenerator. The original version proposed by Goodfellow et al. [10] is called
the vanilla GAN.
6

In the training, G and D play the minimax game with the value function
V defined as follows:
min max V (D, G) = Ex∼pdata (x) [log D(x)]+Ez ∼pz (z ) [log(1−D(G(z)))], (1)GD

where pdata (x) and pz (z) are the distributions of the training data and z,
respectively. The discriminator output D(x) denotes the probability that x
came from the real data distribution, and G(z) represents a mapping from z
to data space. This can be reformulated as the minimization of the Jensen–
Shannon divergence between the real data and generated data distributions.
Following the proposal of the vanilla GAN, various derivations have been
proposed. Major examples related to this study are described below.
3.2. Deep convolutional GAN
The deep convolutional GAN (DCGAN) [26] is a class of architectures of
GANs based on convolutional neural networks (CNNs) mostly used for image
generation tasks. In DCGAN, G generates an image by repeating fractionallystrided convolutions with a random number input z. The discriminator D
uses a CNN to infer whether the given image came from training data ordata generated by G.
3.3. Wasserstein GAN
The Wasserstein GAN (WGAN) [1] is a variation of GANs that uses a
metric different from the vanilla GAN. WGAN defines the distance between
the distribution of training patterns and that of generated patterns based on
the Wasserstein distance, and then minimizes it via training. This approachhas the merit of stable learning with less mode collapse. In WGAN training,the minimax game is represented as follows:
min max V (D, G) = Ex∼pdata (x) [D(x)] − Ez ∼pz (z ) [D(G(z))] ,
G

D∈D

(2)

where D is a set of Lipschitz continuous functions. To satisfy the constraintthat D needs to be a Lipschitz function, D is parameterized with weights
lying in a compact space. Practically, the weights are clamped to a fixed box
after each gradient update. For convenience, we call this method WGANClipping in the rest of this paper.
The WGAN-gradient penalty (WGAN-GP) [12] is an improved versionof the WGAN. The weight clipping performed in WGAN is an approximated7

approach as mentioned in the original paper. This approach often causes
problems such as difficulty in adapting to a complicated distribution andinefficient learning with biased parameters. In WGAN-GP, to solve these
problems, a gradient penalty is employed in the value function. The trainingof WGAN-GP is expressed as follows:
min max V (D, G) = Ex∼pdata (x) [D(x)] − Ez ∼pz (z ) [D(G(z))]
GD− λEx̂∼px̂ (x̂) (k∇x̂ D(x̂)k2 − 1)2 ,(3)where x̂ = x + (1 − )G(z) and  ∼ U [0, 1]. The WGAN-GP employs anew penalty term to make D a Lipschitz function, thereby allowing moreaccurate and efficient learning compared with the original WGAN.
3.4. GANs with controlled output
In ordinary GANs, it is difficult to predict what type of pattern will be
generated from a certain input z via the generator network. Many studies
have therefore investigated the control of GANs’ output. Mirza and Osindero [22] proposed the conditional GAN that can control the class of thegenerated image by adding class information encoded as a one-hot vector to
the generator’s input and a channel representing the class to the discriminator’s input. Many applications have been studied using a conditional GAN
such as image de-raining [33], oversampling [7], terrain authoring [11], andimage denoising [32]. Odena et al. [24] proposed AC-GAN with a strong
constraint such that the class discrimination is also conducted in the discriminator by adding class information to the inputs of both the generator
and discriminator. Lv et al. [20] proposed TarGAN for unsupervised domain
adaptation that disentangles the class code and the style code and generates
target samples with given class labels. Zhang et al. [34] proposed StackGAN
that synthesizes high-resolution photo-realistic images using text description
as a conditioning variable.
4. GlyphGAN: Style-Consistent Font Generation
Figure 2 shows an overview of GlyphGAN. The major differences fromthe ordinary GANs are as follows.
• The input vector z of the generator G consists of a style vector z s anda character class vector z c .
8

1 iteration

char-ID=1

2

26

3

1

…
Encode char-ID
1 into a one-hot vector
char-ID=10
0…

Real sample queue( )]~ data ( | ) [⋮0

Uniform
distribution[-1, 1]100

Wasserstein
distance
0.12
-0.76
-0.39

26

⋮

100

0.51

(

)
~

( )

( )[

( ) ]

( )

Figure 2: Overview of GlyphGAN. The generator G, which generates synthetic fonts similar to manually designed ones, and the discriminator D, which discriminates between
generated and real fonts, are adversarially trained. In GlyphGAN, the input vector z for
G consists of the style vector z s and character class vector z c . The style vector z s is a
uniform random vector, and the character class vector z c is a one-hot vector associatedwith the character class of training patterns. The discriminator D calculates the distance between distributions of generated data and training data based on the Wasserstein
distance. Through learning, the parameters of G are optimized to minimize this distance.

• During training, the character class vector z c is associated with thecharacter class of the training pattern.
4.1. Input vector
Let z be the input of the generator G. In GlyphGAN, z consists of astyle vector z s and a character class vector z c . By independently preparing
input vectors for the style and character class, various character images canbe generated with the style fixed, and vice versa.
Let the style vector z s be a 100-dimensional random number sampledfrom a uniform distribution. This is the same setting as the ordinary GANs.
Let the character class vector z c be a one-hot vector corresponding to
the character class. Taking the alphabet for example as shown in Fig. 2, thecharacter IDs such as 1, 2, 3, . . . associated with the character classes “A,”
“B,” “C,” . . . are encoded to the one-hot format. The number of dimensionsof z c is the total number of characters used for learning. For example, it is
26 for upper-case Latin alphabets.
94.2. Network architecture
GlyphGAN basically employs DCGAN’s network architecture [26]. The
generator G takes a random vector as an input and then outputs an image
with the same size as the training images. Each layer of G is a fractionallystrided convolution. ReLU activation is used except for the output layerthat uses Sigmoid. The discriminator D takes an image and outputs a scalarvalue. Each layer is a strided convolution, instead of using a pooling layer
with an ordinary convolution layer. LeakyReLU was applied to each layer of
D. Different from the original DCGAN, GlyphGAN does not employ batchnormalization, following the recommendation in [12].
4.3. Training algorithm
Algorithm 1 shows training of GlyphGAN. The training algorithm ofGlyphGAN follows that of WGAN-GP. Different from WGAN-GP’s algorithm, a one-hot vector z c representing the character class is embedded into
the latent vector, and is associated with the character ID of the training
data. Given a set of font images x with a character class c = 1, . . . , 26 foreach image, the networks are trained. First, with a fixed z c , only the corresponding characters are used. For example, Fig. 2 illustrates the stage oflearning “A.” The style vector z s is sampled from a uniform distribution andthen concatenated with z c to make the generator’s input z. The networks
G and D are trained using z and the images of the corresponding characterclass. In this stage, we use only a batch of images randomly selected fromall of the training images.
After learning with respect to one character class, we move on to thelearning of the next character class. In the example of Fig. 2, “B” becomesthe next target. After that, we proceed to the learning of “C,” “D,” “E,”..., “Z,” continuously, and then return to “A.” By learning repeatedly foreach character class in this way, we prevent the network from overfitting toa specific character class. A series of learning for all of the character classes
is counted as an epoch.
In the training of GlyphGAN, the WGAN-GP-based value function [12]is used. Since the data sampling procedure and the input vector are different
from the original WGAN-GP, the minimax game is reformulated as follows:
min max V (D, G) = Ex∼pdata (x|c) [D(x)] − Ez s ∼pz (z s ) [D(G(z))]
GD− λEx̂∼px̂ (x̂) (k∇x̂ D(x̂)k2 − 1)2 ,10(4)Algorithm 1: Training of GlyphGAN
Require: Coefficient λ, number of discriminator iterations per
generator iteration Ndisc , batch size M , and Adam
hyperparameters α, β1 , β2Initialize discriminator’s parameters w and generator’s parameters θfor number of training epochs do
for c = 1, . . . , 26 do
Set one-hot vector z c corresponding to character class c;
for t = 1, . . . , Ndisc do
for i = 1, . . . , M do
Sample real data x ∼ pdata (x|c), style vectorz s ∼ pz (z s ), a random number  ∼ U [0, 1];z ← (z sT , z cT )T ;x̂ ← x + (1 − )Gθ (z);L(i) ← Dw (Gθ (z)) − Dw (x) + λ(k∇x̂ Dw (x̂)k2 − 1)2 ;end
PM (i)w ← Adam(∇w M1
i=1 L , w, α, β1 , β2 )endsSample a batch of style vectors {z si }M
i=1 ∼ pz (z );cT TsTz i ← (z i , z ) ;PMθ ← Adam(∇θ M1i=1 −Dw (Gθ (z i )), θ, α, β1 , β2 )end
end
where z = (z sT , z cT )T , c is the character class corresponding to z c , x̂ =
x + (1 − )G(z), and  ∼ U [0, 1].
5. Font Generation Experiment
To evaluate the capability of the proposed method, we conducted a fontgeneration experiment. We evaluated the generated fonts from the following
viewpoints:• Legibility: We verify that the generated font has legibility via a character recognition experiment using a pretrained CNN.
• Diversity: We validate whether the generated font set has diversity
different from the training data.
11• Style consistency: We qualitatively verify that the generated font has
style consistency via visual observation, and then quantitatively evaluate the effect of a training data shortage on style consistency.
5.1. Dataset
Figure 3 shows 30 examples randomly selected from the dataset. For thedataset, we prepared 26 uppercase alphabet letters from 6,561 different fonts.
Each image was a grayscale image with a size of 64 × 64. Although the sizesof the prepared fonts differed slightly from each other even if we set the same
number of points, we used them without normalization, regarding it as oneof the font features.
5.2. Details of the Network Structure and Parameter Settings
Figure 4 shows the structures of G and D used in the experiment. These
networks are the same as DCGAN [26] with the image size adjusted. For theactivation functions of G, we used ReLU except for the final layer that employed Sigmoid. LeakyReLU was applied to each layer of D. The algorithmof the gradient descent method and its hyperparameters were determinedaccording to [12]. For weight updating, we used Adam [14] with the parameters of α = 0.0002, β1 = 0.5, and β2 = 0.99. Batch normalization was notapplied. The number of discriminator iterations per generator iteration was
set as Ndisc = 5. The number of learning iterations was 2,500. The batch
size was set as 1,024.
5.3. Generation Results
Figure 5 shows the generation results; examples of generated fonts withrandomly selected style vectors z s . The results generated by changing character class vector z c with a fixed style vector z s are aligned horizontally.
The results with different style vectors and a fixed character class vector arealigned vertically. In each line, the generated letters have a similar font styleconsistently over all characters even though they are independently generated
with the same z s . In addition, by changing z s , GlyphGAN generated fontswith various types of styles such as serif, sans-serif, thickness, roundness, andsize, even including a font made of outlines.
Figure 6 shows the letter “A” generated with a continuously changing z s .
In this result, 128 points were randomly selected from the z s space. The vector z s was then moved along eight points that linearly interpolated every two
points out of the 128 points. The style of generated font smoothly changed
12Figure 3: Examples of training patterns used in the experiment.

according to the move of style vector z s , demonstrating the possibility of finecontrol of generated styles. This result shows the capability of generating anintermediate font between the existing two fonts.
5.4. Legibility Evaluation
To evaluate the legibility of the generated fonts, a recognition experiment
was performed using a multi-font character recognition CNN. Legibility is
indispensable in font generation and is also one of the important indicatorsin this research.
Figure 7 shows the structure of the multi-font character recognition CNN
13

3
64
128
256
512
4

32

8

64

16

4

8

FRACTIONALLYSTRIDED CONV

16

32

FRACTIONALLYSTRIDED CONV64

FRACTIONALLYSTRIDED CONVFRACTIONALLYSTRIDED CONV(a) Generator
3
Stride 264128Stride 2256512Stride 2
Stride 2

64

4

16

32

8

4
8

32
64

CONV

16

CONV

CONV

CONV

(b) Discriminator

Figure 4: Structures of the generator and the discriminator.

used in this experiment. This CNN consisted of four convolutional layers anda fully-connected layer. ReLU was employed as an activation function after
each layer. In addition, batch normalization and dropout were applied after
the fully-connected layer.
We confirmed the basic ability of this CNN for character recognition using
existing fonts. By dividing 6,561 fonts of 26 alphabet capital letters into
the training and testing sets with the ratio of 9:1, the CNN was trained toclassify 26 letters. Consequently, the training accuracy and testing accuracy
were 98.27 % and 93.26 %, respectively.
In the evaluation, we generated 10,000 fonts with 26 uppercase lettersusing GlyphGAN. For comparison, we replaced the value function for the14

Figure 5: Thirty randomly selected examples from the generated font sets. It should beemphasized that letters have similar font style consistently over all alphabets (“A” to “Z”),
although they are generated independently but with the same z s .

learning of GlyphGAN with those of DCGAN [26] and WGAN-Clipping [1],and generated 10,000 fonts for each comparative method.
Figure 8 shows examples of the comparative methods. Table 1 shows theresults of the character recognition. Compared with DCGAN and WGANClipping, the recognition accuracy for the generated fonts by GlyphGAN was
higher, thereby showing the effectiveness of learning GlyphGAN for improving the legibility of the generated fonts.

15

Figure 6: Generated results with a continuously changing z s for the letter “A”

5.5. Diversity Evaluation
In this evaluation, we validated whether the generated font set had diversity, that is, the generated fonts were different from the training patterns.
The generated fonts are sometimes similar to the font used as a training
pattern. In a sense, it is reasonable to have similar fonts because the goalof GANs is to reproduce the training patterns. Meanwhile, there can be unknown patterns that are not seen in the training patterns if GlyphGAN canestimate the mapping from the distribution of z s onto the manifold that isconstructed by the training patterns.
Figure 9 shows an outline of the analysis method used in this evaluation.
We analyzed the tendency of the generated patterns by measuring the dis16

3

64

64
128

128

256
64
32

64

64

64

26

32

64

64

CONV

CONV

32

32

CONV
POOL

CONV

FC

Figure 7: Structure of the multi-font character recognition CNN used in the legibility
evaluation.
Table 1: Results of character recognition for legibility evaluation.

MethodAccuracy [%]
DCGAN
3.90
WGAN-Clipping
32.92
GlyphGAN (ours) 83.90
tance between the generated patterns and training patterns. Using 10,000
× 26 generated patterns which are the same as in the legibility evaluation,
we calculated the minimum value among pseudo-Hamming distances [29]between each generated pattern and training patterns in the corresponding
character class. We then define the distance between the generated patterns
and the nearest training patterns for each style as the average of the minimum
values over all of the character classes. We also defined the most similar fontas an existing font to which the minimum value was most frequently assigned.
Figure 10 shows a histogram of the distances between generated patterns
and the nearest training patterns. The minimum and maximum distanceswere 40.30 and 2942.86 (0.98 % and 71.84 % of the total pixel number),respectively. In addition, the generated patterns with a distance of less than
500 accounted for 87.51 %.
Figure 11 shows examples of the generated patterns and the most similar
font in the training patterns. In the examples with small distances such as
in Fig. 11(a), fonts that look similar to the training pattern are observed.
By contrast, in the examples with large distances, the generated patterns
are greatly different from the training patterns. These examples can be
regarded as styles not seen in the training patterns. Although such patterns
17(a) DCGAN(b) WGAN Clipping

Figure 8: Generated fonts using different learning algorithms

are relatively few (the ratio of samples with a distance greater than 500 is
about 10 % of the total), the font set generated by the proposed method has
diversity different from the training patterns.
5.6. Effect of Training Data Shortage on Style Consistency
We explore the effect of a training data shortage on style consistency.
From the original 6,561 fonts, we gradually decreased the number of fonts inthe training dataset as 1,000, 100, 10 by randomly selecting fonts from the18

Nearest patterns
Most similar font
Training patterns
6.561 × 26…
Min=97

Min=115

Min=109…

Generated patterns
10,000 × 26…

Min=86
Averaged min-distance= 101.75

…

…

…

…

Figure 9: Outline of the measurement method of the distance between generated andtraining patterns. The distance between each generated and training pattern is calculated
based on the pseudo-Hamming distance. The training pattern nearest to the generated
pattern is surrounded by a blue dotted circle. The distance between generated patterns
and the nearest training patterns is defined as an average of the minimum distances over
all the character classes. The red circles indicate the most similar font, which is an existing
font to which the minimum distance is most frequently assigned.

(c)
(d)

2000

# of generated patterns

1750

(e)

1500
1250

(f)
(g)
(h)

1000
750

(a)

(i) (j)500 (b)(k) (l)(m)(n) (o)

1500

2000

(p)

250
0

0

500

1000

2500

3000

Pseudo-Hamming distance
Figure 10: Histogram of distances between generated patterns and the nearest training
patterns calculated based on the pseudo-Hamming distance. The points indicated by thearrows correspond to subfigures in Figure 11.

19

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

(p)

Figure 11: Comparison of generated patterns and the most similar font in the training
patterns. In each subfigure, the top shows generated ones and the bottom shows training
ones.

original font set. After training GlyphGAN with each selected font set, wequantitatively evaluated the style consistency of the generated font images.

20Figure 12: Generated results with a training dataset composed of only 10 fonts.
Table 2: Effect of the number of training fonts on style consistency.

Number of training fonts Cs
10
1.12
100
0.51
1000
0.47
6561
0.46
For quantitative evaluation, we defined the metric of style consistency byNC
1 X 1 XCs =(dn,c − d¯n )2 ,¯
N C n=1 dn c=1(5)where N is the number of generated images (we used N = 10, 000 in thisexperiment), C is the number of character classes, i.e., C = 26, dn,c is the
distance between the generated font and the nearest real font used in Section
5.5, and d¯n is the average of dn,c over c. The metric Cs is the averaged
coefficient of variation of dn,c . It represents an intra-style variation of thegenerated font images. The lower Cs is, the higher the style consistency is.
As an example of the generated fonts with a limited training font set,
generated results with a training dataset composed of only 10 fonts are presented in Figure 12. Compared with the results in Fig. 5, style consistency
is not maintained. Table 2 shows the relationships between the number oftraining font and the metric for style consistency Cs . The metric Cs increases
according to the decrease in the number of training fonts. These results suggest that a sufficient number of styles is required for the training data to21

Figure 13: Generated results using Deep-fonts.

guarantee style consistency.
5.7. Quantitative Comparison with the Existing Method
We conducted a quantitative comparison with Deep-fonts [4, 5], a neuralnetwork-based generative model for font images. As with GlyphGAN, Deepfonts takes the concatenated vector of a random vector representing a styleand a one-hot vector representing a character class as inputs, and it outputs
a font image. The main differences between Deep-fonts and GlyphGAN are
as follows:• Network structure: A multilayer perceptron-based network is employed
instead of a CNN.
• Loss function: Generated font images are evaluated by L1 loss between
the generated and real font images instead of using a discriminator.
Figure 13 shows the example font images generated by Deep-fonts. It
seems that the generated fonts are legible and have style consistency anddiversity to some extent. However, there are some collapsed fonts such as
those in the third and eighth rows.
We compared the qualities of the generated fonts by GlyphGAN and
Deep-fonts in terms of legibility, style consistency, and diversity. The metrics
for legibility and style consistency were recognition accuracy and Cs used in
Sections 5.4 and 5.6, respectively. Furthermore, we defined the metric ofdiversity as follows:
1(Q3 − Q1 ),
(6)
Cd =
2Q2
22

Table 3: Quantitative comparison with Deep-fonts.

Recognition accuracy [%]Method(Legibility)
Deep-fonts 72.51
GlyphGAN 83.90

Cs
Cd(Style consistency) (Diversity)
0.47
0.33
0.46
0.61

where Q1 and Q3 are the first and third quartiles of {d¯n }Nn=1 , and Q2 isthe median. This metric intuitively represents the coefficient of deviation in
Figure 10. Since there are some outliers due to collapsed generated images,
we used quartile deviation instead of standard deviation.
The results of the quantitative comparison are shown in Table 3. In
particular, GlyphGAN showed better legibility than Deep-fonts. This is because Deep-fonts occasionally generates collapsed and illegible font images.
Introducing the GAN framework allowed the generator to estimate a smoothfont manifold, thereby improving the style consistency and diversity of thegenerated fonts.
6. Discussion
Difference from existing GANs: As stated in Section 3, various GAN
derivations have been proposed. The most similar types are GANs that cancontrol the output such as the conditional GAN [22]. The main structuraldifferences from such GANs are that the character class information is provided only to the generator’s input, and that the sampling from the real data
distribution is associated with the character class. The procedure intrinsically makes GlyphGAN learn the conditional distribution of the target image
given the class information.
Legibility: In the legibility evaluation, we showed the learning method
employed in the GlyphGAN is effective in improving the legibility of thegenerated fonts. Compared with GlyphGAN, DCGAN-based and WGANClipping-based learning led to the collapse of the generated fonts. In the
results where the DCGAN was used as the learning framework shown in Fig.
8(a), almost the same patterns were generated even if different characterclass vectors were given. This is because of the phenomenon called mode
collapse in which the output is biased to a specific pattern. In Fig. 8(b), although WGAN-Clipping-based method generated fonts more efficiently thanthe DCGAN-based method, there were only a few patterns that could be23

recognized as letters. One possible explanation is that the WGAN-Clippingbased method could not represent the complexity of the data manifold owing
to its approximated learning.
Style consistency: Even though GlyphGAN employs unsupervised training in terms of style information, the generated fonts have a consistent style
for all of the characters. However, this property is guaranteed by having asufficient number of styles in the training data. If we use a training datasetthat includes a few styles, the generated font does not guarantee style consistency, as shown in Fig. 12 and Table 2. This is because a large number oftraining data are required to learn the manifold of font styles.
Limitations: This study includes some limitations. First, the dataset
used in the experiment contains only alphabet letters. Font sets from different languages such as Chinese and Japanese can contain a larger number ofcharacters than the alphabet; thus, expansion of the character class vector
is required. Second, the legibility is not perfect. Although GlyphGAN improved the legibility as shown in Table 1, there still exists a 10 % gap in therecognition accuracy between the generated fonts and existing fonts. The
increase in the number of training data is one of the solutions for filling thisgap. Finally, explicit style control is not performed. It is not obvious what
type of z s to use for the generation of a specific font style. In this study,
we obtained a latent space composed by z s . Clarification of the relationshipbetween the font design and the latent space using another framework willlead to explicit style control.
7. Conclusion
In this paper, we proposed GlyphGAN a style-consistent font generation
based on generative adversarial networks (GANs). In GlyphGAN, the input
vector for the generator network consists of a character class vector and stylevector, thereby allowing font generation with style consistency. In the fontgeneration experiment, we showed that the learning method employed in
the proposed method improved the legibility of the generated fonts. The
experimental results also showed that the generated font set had diversity
different from the training patterns.
In future work, we will review the GAN structure to improve the qualityof the generated font. Since many derivatives of GANs are still proposedeven today, a better structure that enables more realistic generation can befound. Analysis of internal representations including the latent space will24

be conducted to understand the generation process. Generation of multiple
characters will also be investigated. Finally, we plan to use vector images
that have contour control points instead of bitmap images. This trial willlead to more practical font design without the limitation of resolution.
Acknowledgment
This work was partially supported by JSPS KAKENHI Grant Number
JP17H06100.
References[1] Arjovsky, M., Chintala, S., Bottou, L., 2017. Wasserstein generative adversarial networks, in: Proceedings of the 34th International Conference
on Machine Learning (ICML), pp. 214–223.
[2] Atarsaikhan, G., Iwana, B.K., Narusawa, A., Yanai, K., Uchida, S.,
2017. Neural font style transfer, in: Proceedings of the 14th International Conference on Document Analysis and Recognition (ICDAR), pp.
51–56.
[3] Baluja, S., 2017. Learning typographic style: from discrimination tosynthesis. Machine Vision and Applications 28, 551–568.
[4] Bernhardsson, E., 2016a. Analyzing 50k fonts using deep neural networks. https://erikbern.com/2016/01/21/analyzing-50k-fontsusing-deep-neural-networks.html, (accessed 16 April 2019).
[5] Bernhardsson, E., 2016b. deep-fonts. https://github.com/erikbern/
deep-fonts, (accessed 16 April 2019).
[6] Campbell, N.D., Kautz, J., 2014. Learning a manifold of fonts. ACM
Transactions on Graphics 33, 91.
[7] Douzas, G., Bacao, F., 2018. Effective data generation for imbalanced
learning using conditional generative adversarial networks. Expert Systems with Applications 91, 464–471.
[8] Gao, X., Zhou, C., Chao, F., Yang, L., Lin, C.M., Xu, T., Shang, C.,
Shen, Q., 2019. A data-driven robotic chinese calligraphy system using25

convolutional auto-encoder and differential evolution. Knowledge-Based
Systems .
[9] Gatys, L.A., Ecker, A.S., Bethge, M., 2016. Image style transfer using
convolutional neural networks, in: Proceedings of the IEEE Conference
on Computer Vision and Pattern Recognition (CVPR), pp. 2414–2423.
[10] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley,
D., Ozair, S., Courville, A., Bengio, Y., 2014. Generative adversarial
nets, in: Proceedings of the Advances in Neural Information Processing
Systems (NIPS), pp. 2672–2680.
[11] Guérin, E., Digne, J., Galin, E., Peytavie, A., Wolf, C., Benes, B., Martinez, B., 2017. Interactive example-based terrain authoring with conditional generative adversarial networks. ACM Transactions on Graphics
36, 228:1–228:13.
[12] Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.C.,
2017. Improved training of Wasserstein GANs, in: Proceedings of theAdvances in Neural Information Processing Systems (NIPS), pp. 5769–
5779.
[13] Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A., 2017. Image-to-image translation with conditional adversarial networks, in: Proceedings of the IEEE
Conference on Computer Vision and Pattern Recognition (CVPR), pp.
1125–1134.
[14] Kingma, D.P., Ba, J., 2015. Adam: A method for stochastic optimization, in: The 3rd International Conference on Learning Representations(ICLR).
[15] Lake, B.M., Salakhutdinov, R., Tenenbaum, J.B., 2015. Human-level
concept learning through probabilistic program induction. Science 350,
1332–1338.
[16] Li, W., Song, Y., Zhou, C., 2014. Computationally evaluating andsynthesizing chinese calligraphy. Neurocomputing 135, 299–305.
[17] Lian, Z., Zhao, B., Chen, X., Xiao, J., 2018. Easyfont: A style learningbased system to easily build your large-scale handwriting fonts. ACM
Transactions on Graphics 38, 6:1–6:18.
26

[18] Lin, X., Li, J., Zeng, H., Ji, R., 2018. Font generation based on leastsquares conditional generative adversarial nets. Multimedia Tools andApplications , 1–15.
[19] Liu, J., Gu, C., Wang, J., Youn, G., Kim, J.U., 2019. Multi-scale
multi-class conditional generative adversarial network for handwritten
character generation. The Journal of Supercomputing 75, 1922–1940.
[20] Lv, F., Zhu, J., Yang, G., Duan, L., 2019. Targan: Generating target
data with class labels for unsupervised domain adaptation. KnowledgeBased Systems 172, 123–129.
[21] Lyu, P., Bai, X., Yao, C., Zhu, Z., Huang, T., Liu, W., 2017. Autoencoder guided GAN for Chinese calligraphy synthesis, in: Proceedingsof the 14th International Conference on Document Analysis and Recognition (ICDAR), pp. 1095–1100.
[22] Mirza, M., Osindero, S., 2014. Conditional generative adversarial nets.
arXiv preprint arXiv:1411.1784 [cs.LG] .
[23] Miyazaki, T., Tsuchiya, T., Sugaya, Y., Omachi, S., Iwamura, M.,
Uchida, S., Kise, K., 2019. Automatic generation of typographic fontfrom a small font subset. IEEE Computer Graphics and Applications(Early Access) .
[24] Odena, A., Olah, C., Shlens, J., 2017. Conditional image synthesis
with auxiliary classifier GANs, in: Proceedings of the 34th International
Conference on Machine Learning (ICML), pp. 2642–2651.
[25] Phan, H.Q., Fu, H., Chan, A.B., 2015. Flexyfont: Learning transferring
rules for flexible typeface synthesis. Computer Graphics Forum 34, 245–
256.
[26] Radford, A., Metz, L., Chintala, S., 2016. Unsupervised representation
learning with deep convolutional generative adversarial networks, in:
The 6th International Conference on Learning Representations (ICLR).
[27] Tenenbaum, J.B., Freeman, W.T., 2000. Separating style and content
with bilinear models. Neural Computation 12, 1247–1283.

27

[28] Tian, Y., 2017. zi2zi: Master Chinese calligraphy with conditional
adversarial networks. https://kaonashi-tyc.github.io/2017/04/06/
zi2zi.html, (accessed 16 April 2019).
[29] Uchida, S., Egashira, Y., Sato, K., 2015. Exploring the world of fonts fordiscovering the most standard fonts and the missing fonts, in: Proceedings of the 13th International Conference on Document Analysis andRecognition (ICDAR), pp. 441–445.
[30] Wang, Y., Wang, H., Pan, C., Fang, L., 2008. Style preserving Chinesecharacter synthesis based on hierarchical representation of character, in:Proceedings of the IEEE International Conference on Acoustics, Speechand Signal Processing (ICASSP), pp. 1097–1100.
[31] Yang, S., Liu, J., Lian, Z., Guo, Z., 2018. Text effects transfer via
distribution-aware texture synthesis. Computer Vision and Image Understanding 174, 43–56.
[32] Yi, X., Babyn, P., 2018. Sharpness-aware low-dose ct denoising using
conditional generative adversarial network. Journal of Digital Imaging
31, 655–669.
[33] Zhang, H., Sindagi, V., Patel, V.M., 2019a. Image de-raining using
a conditional generative adversarial network. IEEE Transactions on
Circuits and Systems for Video Technology (Early Access) .
[34] Zhang, H., Xu, T., Li, H., Zhang, S., Wang, X., Huang, X., Metaxas,
D.N., 2019b. Stackgan++: Realistic image synthesis with stacked generative adversarial networks. IEEE Transactions on Pattern Analysis
and Machine Intelligence 41, 1947–1962.

28