Multi-Style Transfer
Generative Adversarial Networkfor Text Images
Honghui Yuan and Keiji Yanai
Department of Informatics, The University
of Electro-Communications, Tokyo, Japan{yuan-h, yanai}@mm.inf.uec.ac.jp

Introduction• Neural style transfer have shown impressive
results in deep learning.
• Recent researches have successfully completed
the transition from the text font domain.
to the text style domain.

Image Style Transfer Using Convolutional Neural
Networks [Gatys, CVPR 2016]

Controllable Artistic Text Style Transfer via
Shape-Matching GAN [Yang, CVPR2019]ⓒ 2021 UEC Tokyo.

2Introduction• However, for text style transfer, multiple styletransfer often requires learning many models.
• Generating multiple styles images of texts in asingle model remains an unsolved problem.

ⓒ 2021 UEC Tokyo.

3

Introduction• We propose a multiple style transformation
network, which can generate multiple styles oftext images in a single model and control thestyle of texts in a simple way.

?

＋..
.
ⓒ 2021 UEC Tokyo.

4

Related work
1. Style Transfer• The existing researches (Neural image styletransfer,AdaIN) related to style transformationof images have made very significant progress.

cited from adain[CVPR,2017]ⓒ 2021 UEC Tokyo.

5

Related work
2.Image-to-image translation• SPADE [7] allows users to create an actual
composite image from a simple image drawn
by the user.
• Proposes a new normalization layer SpatiallyAdaptive Normalization.

cited from SPADE[Park et al.,CVPR2019]ⓒ 2021 UEC Tokyo.

6

Related work
• SEAN [18] made improvements for SPADE [7].
Individual control of each region of a semantic
segmentation image was achieved.

cited from SEAN[Zhu, CVPR 2020]ⓒ 2021 UEC Tokyo.

7

Related work
3. Text font style transfer• Can transform text styles by learning one
style image and can control different degreesof style.

＋cited from Shape-Matching GAN [Yang，CVPR2019]ⓒ 2021 UEC Tokyo.

8

Shape-Matching GAN• Base method Shape-Matching GAN.
• Stage 1:sketch module is used to change thestyle images into different degrees ofdeformation through the parameter l.

from Controllable Artistic Text Style Transfer via Shape-Matching GAN [Yang, CVPR2019]ⓒ 2021 UEC Tokyo.

9

Shape-Matching GAN• Base method Shape-Matching GAN.
• Stage 2: there are two main parts,structure
module (GS,DS) and texture module (GT,DT).

cited from Controllable Artistic Text Style Transfer via Shape-Matching GAN [Yang,CVPR2019]ⓒ 2021 UEC Tokyo.

10

Shape-Matching GAN• Network requires only one style image for text
style transformation.

• Shape-Matching GAN works well whenlearning just one style, but it does not workwhen learning multiple styles.
• multiple styles of text can not be generated
with only one model.
ⓒ 2021 UEC Tokyo.

11

Proposed method• we propose a multiple style transformation
network for text style transfer based on ShapematchingGAN.
• our main idea:
1.add conditions.
2.optimize the network.

ⓒ 2021 UEC Tokyo.

12

Proposed method• The red line shows the network structure that we havechanged for Shape-MatchingGAN.

ⓒ 2021 UEC Tokyo.

13

Conditional input• Conditional
1.putinput.
the maskimages as conditionsⓒ 2021 UEC Tokyo.

14

Conditional input• Input into the network in pairs with the style
images.

＋fire：greenfeather:purple

smoke:redwater:yellow

SketchmoduleGBⓒ 2021 UEC Tokyo.

15

Multi-style training• SPADE layer can effectively prevent theinformation about mask images from being
washed out in the network.

SPADE[Park et al.,CVPR2019]ⓒ 2021 UEC Tokyo.

16

Multi-style training• Implemented the SPADE layer in Shape-Matching
2.implemented theGAN.
SPADE layerⓒ 2021 UEC Tokyo.

17

Multi-style training• The mask of the four kinds of the style images
is used as input for SPADE ResBlk.

SPADE

SPADE

ResBlkResBlkⓒ 2021 UEC Tokyo.

18

Improving the quality of thegenerated images• Add a discriminator to make the quality of thegenerated images better.

add patchdiscriminatorⓒ 2021 UEC Tokyo.

19

Improving the quality of thegenerated images• Add a PatchGAN discriminator to our texture
network.

Generatedresults＋ⓒ 2021 UEC Tokyo.

Dpatch20

Dataset• Dataset: 129 text images, 4 style images andcorresponding mask images.

Example of text image

Style image and mask imageⓒ 2021 UEC Tokyo.

21

Network training• Training process: input the style images andthe corresponding mask images into thenetwork in pairs.

• Testing stage: input the selected text image
and style mask image to generate thecorresponding style text image.

ⓒ 2021 UEC Tokyo.

22

Results of the experimentsⓒ 2021 UEC Tokyo.

23

Results of the experimentsⓒ 2021 UEC Tokyo.

24

Results of the ablation study
Remove a part of the proposed method.
input style

w/o SPADE

w/o Dpatchⓒ 2021 UEC Tokyo.

full model

25

User study
Baseline vs Multi-style SMGAN
Number of votesⓒ 2021 UEC Tokyo.

26

Conclusions• In this study, we proposed a multi-style transfer
network for text.
• We can also control the generation of various stylesof text images in the generation stage.
• The results show that we have achieved a good effecton the generated style images based on the effective
transformation of multiple text styles.

ⓒ 2021 UEC Tokyo.

27