Generative models forassisting graphic design
Naoto Inoue (CyberAgent Inc.)
June 28, Invited talk III at SNL2023

Self Introduction
Career:●●~ Mar. 2021: Ph. D at The University of Tokyo
Apr. 2021 ~ : Research Scientist at CyberAgent AI Lab○

Research related to creating advertisements

Research interest:●

generative models for graphic design2Graphic Design●●

Visual + textual content
Important to convey ideas

banner ad

presentation

meme (credit)3

Raster v.s. Vector Format
Raster●●for displaye.g., .jpg, .png, …

4

Raster v.s. Vector Format
Raster●●for displaye.g., .jpg, .png, …

Vector●●for edite.g., .pptx, .psd, .svg, …

<svg><image xlink: href=”...”><rect x=55 y =10 … ></rect><text x=10 … >X,XXX円</text>…
</svg>
5

Graphic Design in Vector Format
Features●●Multi-modal attributes
Large number of elements

Research question●

How to generate vector graphic document?

6

WED-AM-185

LayoutDM: Discrete Diffusion Model
for Controllable Layout Generation
Naoto Inoue Kotaro Kikuchi Mayu Otani
Edgar Simo-Serra Kota Yamaguchi

Layout= Simple yet essential interface to understand & control visual design: image: text: embellishment
8

Controllable Layout Generation
Our work: solve a broad range of tasks in a single model

9

LayoutDM●A discrete diffusion model tamed for layout generation10

LayoutDM●●A discrete diffusion model tamed for layout generation
Training-free algorithm to inject various conditions during inference11LayoutDM Results
Input:
category

OutputInput:
noisy layout

Output12

What is Layout?
●●A set of category (1-dim.) + positional info. (4-dim. e.g., xywh)
Recent trend: layout as a sequence of discrete variables (c.f., text)

image

[]

text

[“image”, 0.3, 0.5, 0.25, 0.9],[“text”, 0.7, 0.35, 0.25, 0.15],[“text”, 0.7, 0.6, 0.25, 0.4],[“logo”, 0.85, 0.95, 0.2, 0.04],

Data

logo

Visualization
13

Discrete Diffusion Models [Austin+, NeurIPS’21]●●= diffusion models for modeling categorical variables (e.g., text)
Corruption: a token is stochastically replaced with another in vocabulary

14

Adapting Discrete Diffusion Models for Layout15

Adapting Discrete Diffusion Models for Layout●[PAD] token to enable variable length generation

16

Adapting Discrete Diffusion Models for Layout●●[PAD] token to enable variable length generation
Modality-wise corruption process

17How to Feed Conditions during Inference?

18

How to Feed Conditions during Inference?
●

Hard condition: masking○e.g., “i-th element’s category is C”

19

How to Feed Conditions during Inference?
●

Hard condition: masking○●e.g., “i-th element’s category is C”

Soft condition: logit adjustment○e.g., “an element at the top”, “an element bigger than another”

20

Logit Adjustment
Inject soft condition as a prior term21

Logit Adjustment
Inject soft condition as a prior term

How to implement a prior?
●●Hard coding (e.g., refinement task)
Gradients from loss functions w.r.t. the prediction (e.g., relationship task)
22

Advantages over Existing Methods●

No fixed generation order unlike auto-regressive models○●

Flexibly changing the number of elements to be generated○●c.f., LayoutTransformer [Gupta+, ICCV’21]
c.f., BLT [Kong+, ECCV’22]

Incorporating both hard and soft conditions○c.f., NDN [Lee+, ECCV’20]

23

Results in Rico [Deka+, UIST’17]

24

Results in PubLayNet [Zhong+, ICDAR’19]

25

Quantitative Evaluation (in category + size → position)
LayoutDM achieves the best speed-quality tradeoff

26

Quantitative Evaluation (in category + size → position)
LayoutDM achieves the best speed-quality tradeoff
Worse

Better27

Quantitative Evaluation (in category + size → position)
LayoutDM achieves the best speed-quality tradeoff
WorseBetter

28

Summary●●●A discrete diffusion model tamed for layout generation
Training-free algorithm to inject various conditions during inference
Favorable performance against task-specific/agnostic baselines

Check codes and more results at
https://cyberagentailab.github.io/layout-dm/
29

Generative Colorization of
Structured Mobile Web Pages
Kotaro Kikuchi Naoto Inoue Mayu Otani
Edgar Simo-Serra Kota YamaguchiWeb page colorization

31

Data structure of web page
= Tree structure where each element has style and content information

32

Structured color prediction
Generate color styles given content and hierarchical structure of elements

33

New dataset for web page colorization●●E-commerce mobile web pages adapted from [Hotti+, arXiv’21]
Convert to a tractable data format…
e.g., keep only elements that contribute to the first view (Avg. elements: 1656 → 61)
34

Proposed Hierarchy-aware CVAE model
Generate color styles given content and hierarchical structure of elements

35

Proposed Hierarchy-aware CVAE model
Embed content with message passing to capture hierarchical relationships

36

Proposed Hierarchy-aware CVAE model
Compute style features with the content embeddings and latent vectors37

Proposed Hierarchy-aware CVAE model
Predict color style for each element based on the style feature

38

Experimental Results

Ours (CVAE)
39

Summary●●●New dataset for web page colorization
Generate color styles given content and hierarchical structure of elements
Our hierarchy-aware CVAE model performs better than baselines

Dataset, code, and pre-trained models are available!
https://github.com/CyberAgentAILab/webcolor40

WED-PM-185

Towards Flexible
Multi-modal Document Models(Highlight)

Naoto Inoue Kotaro Kikuchi Mayu Otani
Edgar Simo-Serra Kota Yamaguchi

Flexible Document Model (FlexDM)
Our work: solve many design tasks in a single model42

Key Idea of FlexDM
Multi-modal masked field prediction as a unified interface

43

FlexDM ResultsInput

Output44

Vector Graphic Document●●●A data format for making visual design (e.g., banner by Photoshop)
Consists of a set of visual elements (+ global info) [Yamaguchi+, ICCV’21]
Scalable, editable, human-interpretable

45

Design Tasks in Iterative Design Process46

Design Tasks in Iterative Design Process●●High variety of possible actions
Complex interaction between multi-modal elements→ We handle design tasks in a principled manner47

Masked Field prediction (MFP)●●Predicting arbitrary number of fields hidden by [MASK]
Challenges○○

How to encode/decode various type of fields?
How to handle larger number of fields?

48

Network for Masked Field Prediction (MFP)

E: encoder49

Network for Masked Field Prediction (MFP)

E: encoder, T: Transformer encoder50

Network for Masked Field Prediction (MFP)

E: encoder, T: Transformer encoder, D: decoder
51

Challenges and solutions in MFP●●

Various type of fields → attribute-specific enc. and dec.
Large number of fields → consider interaction only in element-level52

Training FlexDM
Training
1.
2.

In-domain pre-training (15% random masking)Explicit multi-task learning for target design tasks

Loss: reconstruction error
Preprocess●●

Quantization for numerical attributes
Feature extraction using pre-trained models for image and text53

Attributes Prediction (ATTR)
Input

Output54

Texts Prediction (TXT)
Input

Output55

Element Filling (ELEM)
Input

Output56

Quantitative Evaluation in Crello

1.
2.
3.

Much better than baselines
Almost close to task-specific expert
Both components are important

57

Quantitative Evaluation in Crello

1.
2.
3.

Much better than baselines
Almost close to task-specific expert
Both components are important

58Quantitative Evaluation in Crello

1.
2.
3.

Much better than baselines
Almost close to task-specific expert
Both components are important59Summary●●●

Masked field prediction (MFP) as a unified interface
A model handling larger number of fields and tasks efficiently
Promising performance in various documents (e.g., banner, web, ...)

Check codes and more results at
https://cyberagentailab.github.io/flex-dm/
60

Conclusion
Summary●●●

Graphic design = multi-modal data
Formulate many generation tasks using sequence-like data structure
Many challenges remaining○○

End-to-end generation including texts and images, or some alternative?
ChatGPT-like one-model-fits-all moment for design generation?

61

Acknowledgement

Kotaro Kikuchi(CyberAgent)

Edgar Simo-serra(Waseda University)

Mayu Otani(CyberAgent)

Kota Yamaguchi(CyberAgent)62