Retrieval-Augmented Layout Transformer
for Content-Aware Layout Generation

DaichiHoritaNaotoInoue

KotaroKikuchi

Kota
Yamaguchi

Kiyoharu
AizawaConcept
Content-aware layout generation

2

RALF
Retrieval-Augmented Layout Transformer

1) Retrieve nearest neighbor layouts based on the input image
2) use them as a reference to augment the generation process.

3Challenges• Data Scarcity & Training Efficiency• Content-Layout Harmonization• Controllability to User-Specified Constraints

4Challenges
Data Scarcity & Training Efficiency

WebAuthoring tool5

Challenges
Content-Layout HarmonizationMisalignment
Underlay
Text

Logo

Overlap
Text
Text

Logo
6

Challenges
Controllability to User-Specified Constraints
Category →Size +Position

Relationship“Logo, Text x2, Underlay”“Logo top on Text”

7

Contributions• Data Scarcity & Training Efficiency• Retrieval augmentation effectively

addresses the data scarcity problem• Content-Layout Harmonization• Propose RALF
• Controllability to User-Specified Constraints• Show RALF outperforms the baselines
on unconditional & conditional tasks

8

Why Retrieve & Generate?
Just retrieve

GTInput

Top1Output

9

Why Retrieve & Generate?
Retrieve & GeneratèGTInput

RetrieveOutput

10

Preliminaries
Representation of layout

Z = (bos,① c ,x ,y ,w ,h ,1 1 1 1 1② c2, x2, y2, w2, h2,. . . , eos)
Sorted by raster scan order

①②③
④

11

Preliminaries

B binsTokenization of layout• Quantize bbox bi:• Autoregressive modeling:

I: image, S: saliency map
In exp, B = 128 12

Overview of RALF
Propose a Retrieval-Augmented Layout Transformer (RALF)

13

Overview of RALF
Encodes an input canvas image and a saliency map14

Overview of RALF
Retrieves nearest neighbor layout examples based on thesimilarity of an input image.

15

Overview of RALF
Fuses the features of retrieved layouts with the image featureusing cross-attention.

16

Overview of RALF
Incorporates user-specified constraints following
LayoutFormer++ [Jiang+ CVPR23], which tokenizes constraints.

17

Overview of RALF
Autoregressively generates a layout.

18

Layout Retrieval• A challenge lies in the absence of joint embedding forimage–layout retrieval, unlike CLIP for image—text retrieval.

DogNo joint embedding!

joint embedding(CLIP)

image—text retrieval

image—layout retrieval 19

Layout Retrieval• We hypothesize that given an image–layout pair (Ĩ, L̃),L̃ is more likely to be useful when Ĩ is similar to I.
Retrieve images Ĩusing similarity,then use paired
layout L̃
GT layout

Image I(query)(Ĩ1, L̃1)

(Ĩ2, L̃2)

20

Layout Retrieval

21

Layout Retrieval

22

Layout Retrieval
RandomLayoutfidelity

betterDreamSimContent-layout harmonization
23

Analysis
How effective RALF?

24

Analysis
How effective RALF?

Effective regardless of the
training dataset size.

25

Analysis
How effective RALF?

Not sensitive to the numberof retrieved layouts.

26Analysis

How different K affects the output? Compare K=1 with K=16
Similarresults
example(s)Output1 1 Output
2 2
geRetrievedRetrieved
example(s)OutputOutput! 1= 1!=K=1

Reference

Output27

! =! 1= 1

Analysis
How different K affects the output? Compare K=1 with K=16
Diverse and plausible results.

! 16= 16!
=K=16Reference

Output28

Analysis
Limitation of FID.

K=1

K=16
Same FID 🤔
29Qualitative Results
CGL-GANDS-GAN

LayoutDM

PKU Dataset

CGL Dataset!

Embellishment

Autoreg
Baseline

RALF(Ours)

Logo

Text

Underlay

Logo

Text

Underlay30

Quantitative Results
Unconstrained generation results on PKU dataset [Hsu+ CVPR23]
Train:Test:Val = 7,734:1,000:1,000
Content: an overlap of saliency object and layout

31Quantitative Results
Real data is supposed to be upper bound

Validation data32

Quantitative Results
Just top-1 retrieval is the worst in content metrics“Retrieval-augmented” generation is important
Top-1 retrieved layout

33

Quantitative Results
RALF significantly outperforms the baselines

Baseline methods

34Quantitative Results
Baseline methods + Retrieval augmentation

35

Quantitative Results
Out-of-domain generalization
e.g. Train / DB: CGL dataset, Test: PKU dataset

36

Quantitative Results
Constrained
generation
Category →Size + Position

Relationship37Conclusion

Thank you!

• Retrieval augmentation effectively addresses the data
scarcity problem.

• Propose RALF: Retrieval-augmented Layout Transformer• Retrieval augmentation + Autoregressive Transformer.
• Show that RALF successfully generates high-quality
layouts, significantly outperforming baselines.
Contact:
horita@hal.t.u-tokyo.ac.jp@udoooom38