Word-Conditioned Image
Style Transfer
Yu Sugiyama and Keiji Yanai
The University of Electro-Communications,Tokyo

2018/12/03

UEC Yanai Lab. Tokyo.

1Introduction• Neural Style Transfer, Image style transfer

2018/11/27

UEC Yanai Lab. Tokyo.

2

Introduction• Style Transfer requires CONTENT and STYLE image→ User need to find good image

2018/12/03

UEC Yanai Lab. Tokyo.

3

Objective• Words condition makes easy to find good style"Leaves”
2018/12/03

UEC Yanai Lab. Tokyo.

4

Related Work
1. A Neural Algorithm of Artistic Style
Leon A. Gatys, CVPR, 2016
Image style synthesis for artistic style

2. Perceptual Losses for Real-Time Style Transfer
and Super-Resolution
Jastin Johnson, ECCV, 2016
Pre-training stylization network to fast image transfer

2018/12/03

UEC Yanai Lab. Tokyo.

5

Related Work
3. Unseen Style Transfer
Keiji Yanai, ICLR WS, 2017
improvement of fast style transfer
Stylize image with un-trained images

4. Arbitrary Style Transfer
Golnaz Ghiasi, Honglak Lee, et al. BMVC, 2017
Different method of Unseen Style Transfer
Synthesis images with Conditional Instance
Normalization

2018/12/03

UEC Yanai Lab. Tokyo.

6Method• Arbitrary Style Transfer Network

2018/12/03

UEC Yanai Lab. Tokyo.

7

Method• SSN(Style SelectorNetwork) predicts
Style Vector for transfernetwork

2018/12/03

UEC Yanai Lab. Tokyo.

8Train Word2Vec• Generate 200-dim vector from words• English Wikipedia Corpus• Reduce row corpus• Remove all Stopping words, low frequency word(under 5times)• Random remove High frequency word(over 1000 times)

2018/12/03

UEC Yanai Lab. Tokyo.

9

Method• SSN makes Style vector same as Inception-v3 feature
extractor
Inception v3(Style PrecisionNetwork)

Style VectorApproximate``Leather’’

2018/12/03

Word2VecEmbeddingvector

StyleSelector
NetworkUEC Yanai Lab. Tokyo.

Style Vector

Train Dataset• Stands by Yahoo100M wild images data
• Random select 1000 images for 500 categories• Adverbs tag annotation• Contains 500,000 image-word pairs

2018/12/03

UEC Yanai Lab. Tokyo.

11

Experiments - Detail• Optimize with Adam
• Style transfer network and style prediction network
are pre-trained• Train Only 3 FC layers• Training takes 10 min

2018/12/03

UEC Yanai Lab. Tokyo.

12

Experiments• L2 minimalize instead of adversarial loss
input

2018/12/03

Adversarial Lossnewurban indoor

UEC Yanai Lab. Tokyo.

newL2 Lossurban

13

indoorExperiments• There are some mismatches between content images
and style words
Mismatch
input
newold2018/12/03

match
input

UEC Yanai Lab. Tokyo.

newold14

Experiments - problem• Word2Vec cannot distinct “summer leaves” and“fall leaves” from only “leaves”• One word is not enough explain visual featureleaves

2018/12/03

UEC Yanai Lab. Tokyo.

15

Experiments• Leather images dataset• Crawled by Google image search• Search 84 keywords• About 500 images each keywords

Leather

2018/12/3*AdvancedAncientRecent
elderlyetc.

Leather advanced bag

Leather ancient bag*Bags
Shoeswallet

UEC Yanai Lab. Tokyo.

16

Experiments• Trained SSN only leather images model
INPUT

2018/12/03

newcurrent advanced

UEC Yanai Lab. Tokyo.

elderly

fossilold17

Results and Discussions• Adversarial loss generate superior feature mapthan L2 loss
• L2 loss model generates mean feature of trained images• No confidence to match the word meaning andvisual feature• leather experiments solved several problems• Cannot preserve background and object domain• Style transfer architectures are not match perfect.
2018/12/03

UEC Yanai Lab. Tokyo.

18

Conclusion and Future WorkConclusion• Image Stylization with words without conditional
approach• There are question the transformation is it right for
our feelings.

Future Work
• LSTM units for use sentences not only words• Other domain transfer techniques for image
synthesis network
2018/12/03

UEC Yanai Lab. Tokyo.

19

2018/12/1

UEC Yanai Lab. Tokyo.

20Typical Bad results• Input “red” --> image styled darker withoutcolor change• Input “black” --> sometime image styled blight• I think dataset not only that words area
• “leather red wallet” image --> black wallet with red emblem• Attention model can solve this problem• Segment “red” area of images
2018/12/1

UEC Yanai Lab. Tokyo.

21

Experiments
input

2018/11/27

oldnewcountry urban

UEC Yanai Lab. Tokyo.

22

2018/12/1

UEC Yanai Lab. Tokyo.

23

2018/12/1

UEC Yanai Lab. Tokyo.

24