Font Style Transfer Using Neural
Style Transfer and Unsupervised
Cross-domain Transfer
Atsushi Narusawa, Wataru Shimoda, and Keiji
Yanai
Department of Informatics, The University ofElectro-Communications

Back groundPrevious works1

The skelton is generated by adjusting
sampling points
T. Miyazaki et al., arXiv:1701.05703

Previous works : without CNN

Good results
Needs making skelton datasets for extracting stroke

Deep learning
Extracting stloke automaticallyPrevious works 2-1（Style Transfer）＋Content=Style

Generated results
Gatys et al., CVPR2016

Font generation using
Neural Style TransferNot requires additional training and dataset.
High quality.
Requires large computational cost for generating an image

Previous works 2-2 : Fast Style Transfer
Speeding up Neural Style Transfer by training a model
Pre trained VGG 16

𝐿𝑐𝑜𝑛𝑡𝑒𝑛𝑡
Content images

𝐿𝑠𝑡𝑦𝑙𝑒Pre trained VGG 16

Style image

J. Johnson et al., ECCV2016

Previous works 2-2（Fast Style Transfer）

Inputs, Outputs

Inputs, Outputs

Inputs, Outputs

Fast Style Transfer

High readability
Has difficulty for natural texture of generated images

Previous works 3（ Cycle GAN ）Horsedataset

ZebradatasetJ.Zhu et al., ICCV2017

Previous works 3 (Cycle GAN)
Simsum to Gothic

𝐿𝑐𝑦𝑐𝑙𝑒REAL GOTHIC?

𝐿𝑎𝑑𝑣𝑒𝑟𝑠𝑎𝑟𝑖𝑎𝑙

REAL SIMSUM?
Gothic to Simsum

𝐿𝑐𝑦𝑐𝑙𝑒Previous works3 (成功例)Previous works 3 (失敗例)Unsupervised cross domain learning using Cycle GAN

Unsupervised learningsuperiority on transferring complexed style

The readability of generated image is lowProposed method :Cycle GAN With Style Loss + Content Loss
Simsum to Gothic

𝐿𝑐𝑜𝑛𝑡𝑒𝑛𝑡

𝐿𝑐𝑦𝑐𝑙𝑒REAL GOTHIC?

𝐿𝑎𝑑𝑣𝑒𝑒𝑠𝑎𝑟𝑖𝑎𝑙

𝐿𝑠𝑡𝑦𝑙𝑒REAL SIMSUM?
Gothic to Simsum

𝐿𝑐𝑜𝑛𝑡𝑒𝑛𝑡

𝐿𝑐𝑦𝑐𝑙𝑒𝐿𝑡𝑜𝑡𝑎𝑙 = 𝛼 𝐿𝑎𝑑𝑣𝑒𝑟𝑠𝑎𝑟𝑖𝑎𝑙 + 𝛽 𝐿𝑐𝑦𝑐𝑙𝑒 + 𝛾 𝐿𝑠𝑡𝑦𝑙𝑒 + 𝛿 𝐿𝑐𝑜𝑛𝑡𝑒𝑛𝑡

Encoder-Decoder Network

Encoder-Decoder Net with Res blocks

Discriminator Network

Datasets

SourceTargetDtasets

Number of image

SimSun font893

ketchup character

445

Sand character483

rope pattern796

Examples of datasets

Character

ArtRandompattern

A format of input image

Input image size
(256x256)

Target images

Setting 16 characters in a training image
Training: 450 , Validation : 50

Comparison on ketchup character dataset

Style Transfer

Cycle GANOurs

Comparison on sand character dataset

Style TransferCycle GANOursComparison on rope character dataset

Style Transfer

Cycle GAN

Ours

Difference between combination of losses

𝑨𝒅𝒗𝒆𝒓𝒔𝒂𝒓𝒊𝒂𝒍+𝑪𝒚𝒄𝒍𝒆(Cycle GAN)

Style+ 𝑪𝒐𝒏𝒕𝒆𝒏𝒕+𝑪𝒚𝒄𝒍𝒆

Adversarial+𝑺𝒕𝒚𝒍𝒆+𝑪𝒚𝒄𝒍𝒆

𝑨𝒅𝒗𝒆𝒓𝒔𝒂𝒓𝒊𝒂𝒍+𝑺𝒕𝒚𝒍𝒆+ 𝑪𝒚𝒄𝒍𝒆+𝑪𝒐𝒏𝒕𝒆𝒏𝒕

Difference between combination of losses

𝑨𝒅𝒗𝒆𝒓𝒊𝒂𝒍+𝑪𝒚𝒄𝒍𝒆(Cycle GAN)

Style

Adversarial+ 𝑪𝒐𝒏𝒕𝒆𝒏𝒕+𝑪𝒚𝒄𝒍𝒆+𝑺𝒕𝒚𝒍𝒆+𝑪𝒚𝒄𝒍𝒆

𝑨𝒅𝒗𝒆𝒓𝒔𝒂𝒓𝒊𝒂𝒍+𝑺𝒕𝒚𝒍𝒆+ 𝑪𝒚𝒄𝒍𝒆+𝑪𝒐𝒏𝒕𝒆𝒏𝒕Difference between combination of losses

𝑨𝒅𝒗𝒆𝒓𝒔𝒂𝒓𝒊𝒂𝒍+𝑪𝒚𝒄𝒍𝒆(Cycle GAN)

Style

Adversarial+ 𝑪𝒐𝒏𝒕𝒆𝒏𝒕+𝑪𝒚𝒄𝒍𝒆+𝑺𝒕𝒚𝒍𝒆+𝑪𝒚𝒄𝒍𝒆

𝑨𝒅𝒗𝒆𝒓𝒔𝒂𝒓𝒊𝒂𝒍+𝑺𝒕𝒚𝒍𝒆+ 𝑪𝒚𝒄𝒍𝒆+𝑪𝒐𝒏𝒕𝒆𝒏𝒕Mean of style loss on ketchup character dataset

Mean of style loss on sand character dataset

Mean of style loss on rope pattern dataset

Conclusion• we proposed a method to combine neural style network withCycleGAN• We optimized four types of loss adversarial loss, cycle loss, styleloss and content loss
• the effective combinations differed in each dataset• content loss keeps original image character structure• Future works• perturb the shape of input image to make it easy to findcorrespondence between sources and targets• introduce a patch-based approach