TextCenGen: Attention-Guided Text-Centric Background Adaptation for Text-to-Image Generation
Authors: Tianyi Liang
Jiangqi Liu
Yifei Huang
Shiqi Jiang
Jianshen Shi
Changbo Wang
Chenhui Li
Venue: ICML 2025 poster (2025)
Keywords: Image Synthesis
Primary area: applications->computer_vision
TL;DR: TextCenGen introduces a dynamic adaptation of the blank region for text-friendly image generation, and enhances T2I model outcomes on specially collected prompt datasets, catering to varied text positions.

Abstract: Text-to-image (T2I) generation has made remarkable progress in producing high-quality images, but a fundamental challenge remains: creating backgrounds that naturally accommodate text placement without compromising image quality. 
This capability is non-trivial for real-world applications like graphic design, where clear visual hierarchy between content and text is essential.
Prior work has primarily focused on arranging layouts within existing static images, leaving unexplored the potential of T2I models for generating text-friendly backgrounds.
We present TextCenGen, a training-free approach that actively relocates objects before optimizing text regions, rather than directly reducing cross-attention which degrades image quality. Our method introduces: (1) a force-directed graph approach that detects conflicting objects and guides them relocation using cross-attention maps, and (2) a spatial attention constraint that ensures smooth background generation in text regions. Our method is plug-and-play, requiring no additional training while well balancing both semantic fidelity and visual quality.
Evaluated on our proposed text-friendly T2I benchmark of 27,000 images across three seed datasets, TextCenGen outperforms existing methods by achieving 23\% lower saliency overlap in text regions while maintaining 98\% of the original semantic fidelity measured by CLIP score and our proposed Visual-Textual Concordance Metric (VTCM).

TextCenGen: Attention-Guided Text-Centric Background Adaptation for Text- Generation
Tianyi Liang" 123
Jiangqi Liu
Yifei Huang
Shiqi Jiang' Jianshen Shi
Changho Wang' Chenhui Li*
Abstract
1. Introduction
Text-to-image (T2I) generation has made remark- able progress in producing high-quality images, but fundamental challenge remains: creating backgrounds that naturally accommodate text placement without compromising image quality. This capability is non-trivial for real-world ap- plications like graphic design, where clear vi- sual hierarchy between content and text is es- sential. Prior work has primarily focused on arranging layouts within existing static images, leaving unexplored the potential of T21 models for generating text-friendly backgrounds. We present TextCenGen, a training-free dynamic background adaptation the blank region for text-friendly image generation. Instead of di- rectly reducing attention in text areas, which degrades image quality, relocate conflicting objects before background optimization. Our method analyzes cross-attention maps to iden- tify conflicting objects overlapping with text re- gions and uses a force-directed graph approach to guide their relocation, followed by attention excluding constraints grounds. Our method is plug-and-play, requir- ing no additional training while well balancing both semantic fidelity and visual quality. Eval- uated on our proposed text-friendly T2I bench- mark of 27,000 images across four seed datasets. TextCenGen outperforms existing methods by achieving 23% lower saliency overlap in text re- gions while maintaining 98% of the semantic fi- delity measured by CLIP score and our proposed VextullTextual Concordance Metric (VTCM).
In graphic design, achieving a harmonious visual effect be- tween text and imagery is essential for clear expression. The choice of background can influence text visibility and comprehension. A common design objective is to place text within image in way that is both visually pleas- ing and clearly conveys the intended message. A preferred strategy is to position the text in the golden ratio, which is believed to be aesthetically optimal. However, design- ers often grapple with the issue of backgrounds that com- pete with or obscure the text, as illustrated in Figure 1@, where unsuitable backgrounds detract from the text's read- ability and aesthetic appeal, regardless of any adjustments to text color or size. We aim to facilitate the creation of text- friendly images (see Figure ). ideal for text placement and meeting the growing demand due to the increasing use of Text-to-Image (T2I) models for background graphics.
Traditional approaches to graphic design, especially in poster creation, have largely focused on arranging layouts in static natural background images, elements, and text (Guo et al., 2021; Cao et al., 2012; 2; O'Donovan et al., 2014; Li et al., 2022). However, producing text-friendly images remains a challenge due to the complexity of background elements. Our insight, derived from Figure 1, reveals that clear separation between the main objects and the text ar- eas is essential. Recent advancements in diffusion-related research have shown that it is possible manipulate the primary objects within cross-attention maps, making the adaptation of background images to accommodate text a feasible endeavor (Hertz et al., 2022; Epstein et al., 2023; Wang et al., 2024). However, we discovered that directly reducing attention in the target region leads to a semantic reduction in the generated image's 's match with the prompt. Could we move the conflicting objects out of the target area before reducing attention?
ensure smooth back-
'Equal contribution 'College of Computer Science and Technology. East China Normal University, Shanghai, China "Shanghai Institute of AI Education, Shanghai, China SShangaai Artificial Intelligence Laboratory, Shanghai, China. Correspon- dence to: Chenhui Li <chli@ es. www.ccu...cccoo wcw...uu.cc...
In response, we introduce TextCenGen as illustrated in Figure 1, new method that employs cross-attention maps and force-directed graphs for effective object place- ment and whitespace optimization. We also implement
Proceedings of the 4nnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Open source code https::/github.com Tlanyilt/TextCeree Background Adapt

---

Tex:CenGen: Attention-Guided Background Adaptation for T2I
Prompt: "A playful dolphin jumping out of the ocsar."
9:41
9:41
9:41
9M41
TextCenGen
Prompt: "A space scene with planets, stars, and a distant galaxy."
9:41
9:41
TextCenGen
Target Wallpaper Container
Planned Space Region
Text-Friendly Background Adaption
No Guidance
Figure I. TexiCenGen a training-free method designed to generate text-friendly images. By using simple text prompt and a planned blank region as inputs, TextCenGen creates images that satisfy the prompt and provide sufficient blank space in the target region. For example, the text-friendly T2I approach helps users customize their favored text- friendly wallpapers for mobile devices with any T2I model, avoiding visual confusion caused by the main objects overlapping with UI components.
spatial excluding cross-attention constraint ensure smooth attention in areas designated for text. To establish benchmark for this innovative task, constructed diverse dataset gathered from three unique sources, along with five evaluation metrics, to comprehensively assess the performance. The contributions of our paper are three-fold:
et al., 2020). The integration of deep learning in text layout of natural image generation has led to various models such GAN (Goodfellow al., 2014; Zheng et al., 2019; Li et al., 2019; Zhou et al., 2022), VAE (Jyothi et al., 2019), transformers (Vaswani et al., 2017; Inoue et al., 2023a;
Wang et al., 2023) and diffusion models (Ho et al., 2020;
Hui et al., 2023; Inoue et al., 2023b; Chai et al., 2023; Li et al., 2023a). These models have been instrumental in learning layout patterns from large datasets. Subsequent re- search explored methods to retrieve matching background images based on text and image description (Jin et al., 2022). Since the development of image editing method based on diffusion model, scene text generation methods such as TextDiffuser (Chen et al., 2023a;b) and DiffText (Zhang et al., 2024) have addressed the challenges of gen- erating clear text with diffusion models. However, these methods often rely the presence of "sign" similar element within the prompt (e.g., T-shirt) to place text. They do not explicitly tackle the problem of creating text- friendly images where the background itself crafted to adapt pre-defined text regions. Our approach extends these capabilities by allowing images to yield space text regions, resulting in more har- monious and aesthetically pleasing compositions.
We propose new task of text-friendly T2I generation, which creates images that satisfy both the prompt and reserve space for pre-defined text placements. The task consists of benchmark including a specialized dataset and tailored evaluation metrics.
We introduce TextCenGen, plug-and-play, training- free background adaptation framework for dynamic text placement in generated images.
We develop force-directed cross-attention guidance, adaptable to various attention mechanisms across dif- ferent T21 models, ensuring a harmonious layout of text and imagery.
primary objects generated
2. Related Work
Text Layout of Natural Images has evolved significantly, transitioning from traditional layout designs to more ad- vanced methods influenced by deep learning. Initially, poster design focused on creating layouts with given back- ground images, elements, and text (Guo et al., 2021; Cao al., 2012; O' O'Donovan al., 2014 4; Li et al., 2022; Zhang
Text-to- Immage Generation has advanced with diffusion models (Ho et al., 2020), producing realistic images and videos that align closely with text prompts (Rombach et al., 2022; Singer et al., 2023; Chen al., 2023c; Esser et al., 2024). Innovations in this field include GLIDE (Nichol
2

---

TextCenGen: Attention-Guided Background Adaptation for T21
al., 2022), which integrates text conditions into the dif- fusion process, Dall-E 2 (Ramesh et al., 2022) with its dif- fusion prior module for high-resolution images, and Ima- gen (Saharia et al., 2022), which uses a large T5 language model to enhance semantic representation. Stable diffusion (Rombach et al., 2022) projects images into latent space for diffusion processing. Beyond text conditions, the manipu- lation of diffusion models through image-level conditions has been explored. Methods such image inpainting (Bal- aji et al., 2022) aim to generate coherent parts of an image, while SDG (Liu et al., 2023) introduces semantic inputs to guide unconditional DDPM sampling. In addition, tech- niques such as (Meng et al., 2021) use images as editing conditions in denoising processes. Scene text generation methods such as Tex(Diffuser (Chen et al., 2023a;b) and GlyphDraw (Ma et al., 2023) have also emerged, utilizing textual layouts masks guide text generation in images. These advancements represent the growing versatility and potential of T2I models in diverse applications.
output image that has (1) reduced overlap between the pri- mary object and R, (2) sufficient smoothness and minimal color variation in R, and (3) the image still fits the prompt. Our framework is shown in Figure 2.
As cross-attention map A serves as a medium to locate and edit objects within generated images (Epstein et al., 2023), we focus on the 4thh token of T promppt. In response to our first concern, we analyze the denoising process of an un- guided image (Tori) from given prompt, and establish the subset of tokens (O) that refer to objects that need modi- fication. We design a conflict detector that determines ob- ject conflicts based on the average attention intensity in the overlapping regions between Aş and R. For every token k E O, we introduce Force-Directed Cross-Altention Guid- ance for moving objects. In this scheme, objects are treated as centroid vertex within a graph, with sequence of forces being applied (see Figure 3) to adjust object positions.
For the second consideration, inspired by recent technolo- gies that limit the range of the attention map (Zhang et al., 2024), we propose a spatial excluding cross- cross-attention con- straint to prevent an extensive attention density from en- croaching on R.
Attention Guided Image Editing has emerged as a fun- damental solution to the challenge of translating human preferences and intentions into visual content through text descriptions. These approaches, as seen in works like (Li et al., 2023b; Avrahami et al., 2023; Zhang et al., 2023; Zhao et al., 2023), involve learning auxiliary modules on paired data. However, a limitation of these training-based methods is the substantial cost and effort required for re- peated training for different control signals, model archi- tectures, and checkpoints. In response to these challenges, training-free techniques have emerged, using the inherent weights of attention and the pre-trained models to control the attributes of objects such as size, shape, appearance, and location (Hertz et al., 2022; Epstein et al., 2023; Patash- nik et al., 2023; Xie et al., 2023; Zhou et al., 2024a;b). These methods typically utilize basic conditions, such as bounding boxes, for precise control over object positioning and scene composition (Mo et al., 2023). Desigen (Weng et al., 2024) discovers relationship between attention and saliency and introduces attention reduction to weaken the attention within layout boxes. Our approach takes this fur- ther by applying force-directed graph techniques to cross- attention map edits, allowing for more automated and pre- cise object transformations in T21 editing.
Addressing the third concern, our denoising process in- corporates loss function with additional regularization terms to safeguard the shapes and positions of other objects. Sometimes excessive repulsive force can occasionally dis- place essential objects from the image. To prevent objects from being dislocated outside limits while retaining their reasonable shapes, we also introduce the notions of Margin Force Fm))) and Warping Force Fw.()
3.1. Force-Directed Cross-Attention Guidance
Cross-Attention and Centroid of Object. To seam- lessly integrate the concept of force-directed graphs into the loss guidance of the denoising process in latent dif- fusion models, we delve into the extraction and manipu- lation of attention maps and activations. For denoising image i, we use softmax normalized attention matrices Ar, E R ",W extracted from the standard denoising manipulate
forward step c0, (264, y). This enables the control over objects referred to in the text conditioning y at distinct indices k. by adjusting the related attention channel(s) A....... E R "R1,,,,xxie The centroid of the attention map is a two-dimensional vector, defined by the equation:
3. Method
Given an input text region R and a prompt T prompt. our framework aims to generate a text-friendly image Ires- This research is motivated by extensive interviews with wallpaper designers, who emphasized the need for precise control over text areas, object positioning, and background consistency in T21-model-generatee images. Specifically, our framework addresses these concerns by producing
Enn hAbu, En w A..h.n
centroid (1)) = Σn, w An.w.t
(I)
We assume that all objects are convex sets, adhering to the mathematical definition that for every pair of points within the object, the line segment connecting them lies entirely
3

---

TextCenGen: Attention-Guided Background Adaptation for T2I
Prompt: lighthouse shines through thick fog."
latents for T-
I ori
TEXT
Decoder
Region R
Text Enc
Down
Downn
Down
Down
Mid
Original Image
Original Image
Ires
TextCenGen
TEXT
Result Image
Arrange as you
A
lighthouse shines through
the
thick
fog
Apus(vv)
Aori
Farce Directed Guidance
Conffict Detector
Replace
Carce-Directed Guidancce
Ares
Spatial Exclucling Constraint
Input
Cross-Attention Module
Block
U-Net
Cross-Attention Map
Element-wise Multiplication
Region Based Info Flow
Data Flow
Equal
Replace
Figure 2. In our approach, the model receives a blank region (R) denoted as red-dotted area, and a text prompt as its inputs. The prompt is then used concurrently in Text-to-Image (T2I) model to generate both original image and result image. During each step of the diffusion model's denoising process, the cross-attention map from the U-Net associated with the original image is used to direct the denoising of the result image the form of loss function. Throughout this procedure, conflict detector identifies objects that could potentially conflict with R. To mitigate such conflicts, force-directed graph method is applied to spatially repel these objects, ensuring that the reserved for the prompt remains unoccupied. To further enhance the smoothness of the attention mechanism, spatial excluding cross-ate atteettonn constraint is integrated into the cross goss-attention map.
represents average of A across all layers / in the U-Net architecture. Practically, our method is applied individu- ally to each layer, ensuring a nuanced and layer-specific approach to force-directed attention guidance.
TEXT
TEXT
TEXT
TEXT
TEXT
(nT)) Intersottoon
(60) Onjocc Text
It Warping Force
Layer-wise Conflict Multi-Target Detector. To effec- tively manage conflicts between text and objects in im- ages, we have developed layer-wise conflict multi-target detector, denoted as D() This detector is crucial for iden- tifying tokens A: within each layer of the U-Net that cor- respond to objects that require modifications in relation to text regions. The detector function D(k, R, A operates as follows:
TEXT
TEXT
1tt1
(+9) Seporation
(44) Text Objjot
(0) Farcee
Figure 3. Illustration of four relationships and their associated forces. The Repulsive Force separates object and text centroids during intersections (al) and object text (a2). The Margin Force (b) and Warping Force (c) prevent boundary overstepping. Text within object regions (a4) requires cooperation between force and attention constraint. Separation (a3) isn't required to process.
1, if mean (Ah,,, k O (R) (2)
D R, R,,,
10, otherwise
within the object. This assumption allows us to treat the extracted centroid (k) as vertices in graph, which are then subjected to force-directed attention guidance. Indeed, is important to clarify that while each token & associated with A appears single entity in Figure 2, it actually
where Al A is the attention map for token k at layer 1, and R is the region designated for text. The function re- turns a value lap between Ah, and exceeds a predefined threshold 0, indicating conflict that requires our guidance function.
when the mean attention within the over-

---

TexiCenGen: Attention-Guided Background Adaptation for T2I
Identifying and adjusting the bounding boxes is visualized in Figure 2.
this canvas A, identify a key region, the object O, de- fined by the coordinates (x; y, a, b), where (x.yy marks the upper-left corner and (a, b) the lower-right corner. The movement is calculated based on the sum of repulsive force (v) and margin force Fm (v), yielding the total displace- ment Apos(v) = Feep( (v) + (m). Applying Apos(v) to both A and O, we obtain the transformed canvas A' and the object O'. Subsequently, we shift our coordinate sys- tem's origin to vertex within A' that remains within the canvas boundaries, establishing a new origin Onew. This repositioning is crucial when O' exceeds the visual bound- aries of A. In such cases, we scale the moved A to en- sure that the bounding box of O' fits precisely within the confines of AL. Scaling factors S₂ and Sy, are calculated as S₂ = min (I, - and S, = min (1, W where (a', 6") are the new coordinates of O'. Finally, the scaled A', and O' are reverted back to their original coordinate origin (see the warping force in Figure 3). A critical as- pect of our approach is the transformation of the reference frame. After displacing O due to Apos(v), (v), a new reference frame is established, centered at Onew- The coordinates of O in this new frame are calculated as (araww Vnew)
The fundamental repulsive force [ ensures that each ele- ment is placed separately. ç denotes the general strength of the force, while p(u;) and p(t/o) indicate the positions of the vertex and the target object, respectively. For scenarios that encompass multiple targets, our framework adopts a cumulative force approach to balance attention across these elements. This is quantified by the formula:
Repulsive Force.
(rop()s 11)
T1
Fmn (w;)
Σ W/
[P((;) - p(tar,)ll'
where wy are coefficients for balancing attention across tar- gets. To regulate the impact of these forces and avoid ex- cessive dominance by any single target, we introduce a a+FFepp ensures that the forces exerted do not exceed a practical threshold, thereby maintaining visual equilibrium in complex scenes.
force balance constant a in the form of Frep_.
(x + Ax Onew.: y/ + Ay- Onew,). The affine transforma- tion, considering this frame shift, is represented as:
Margin Force. The Margin Force is a critical component of our force-directed graph algorithm, designed to prevent significant vertices from being expelled from visual bound- ariis. This force, (v) borbbrrdrr is activated as a vertex v approaches the edge of the display area, typically a delineated rectangular space. The force is directed inward to ensure that crucial vertices remain within the designated visual region. The constant m modulates the force's inten- sity, and d(v,, border) represents the distance of the vertex from the nearest boundary (see Figure 3).
Sz
0
Ar Onew.
T =
0
Sy
Ay
Onew,
(4)
0
0
1
This matrix transforms the coordinates of O, ensuring that it remains visible within A after transformation. This care- fully planned process secures the region O within A, even after dynamic changes. It upholds the structure of the cross- attention map, balancing key vertices visibility and graph fluidity.
To compute the to- tal displacement Apos(e), sum the repulsive and margin forces together: Apos(v)
Displacement and Position Update.
(r) + (v) Subse- quently, update the vertex's position as follows: pos(v) = pos(e) + (p).(v). For the attention map AL. this update is applied as a whole, with excess regions outside the bound- aries being discarded and the remaining areas filled with zeros. But this method introduces the risk of the object be- ing moved out of the boundaries and then being discarded, so it is necessary to introduce the following 'warping force" to prevent this from happening.
3.2. Spatial Excluding Cross-Attention Constraint
Our goal is to maintain a smooth background in the text re- gion denoted as R. As illustrated in Figure 2, during each time-step of the forward pass in the diffusion model, we modify the cross-attention maps at every layer. The cross- attention map is represented as .a.. E R R'xW*KK where H x W are the dimensions of A' at different scales, and K signifies the maximum token length at layer I. The set I consists of indices of tokens corresponding to areas outside the text in the prompt. We resize R to align with the H W dimensions. Subsequently, a new cross-attention map for each layer / is computed as Al = (A) (1- Vk O). where O represents the tokens needing editing. This procedure effectively redirects the model's s attention away from R. ensuring that the background in this region re- mains undisturbed and visually smooth. This spatially ex- clusive approach enhances the clarity and coherence of the
Warping Force. In addressing the dynamics of our force- directed graph algorithm, particularly for the movement of cross-attention maps Al. in each layer, we employ affine transformations as a pivotal mechanism. This approach fa- cilitates the comprehensive translation and scaling of the entire map, preserving the relative positions within the im- age domain. Initially, we delineate our visual area or im- age space as H W two-dimensional array A. Within
5

---

TextCenGen: Culention-Guided Background Adaptation for T2I
Dataset
Metrics
Dall-E 3
AnyText
Desigen
SD 1.5
Ours (SD 1.5)
Improve (%)
P2P Template
Saliency IOU +
52.64
54.34
30.62
29.89
22.86
23.52%
TV Loss +
18.02
22.35
13.7
14.11
8.81
37.56%
VTCM f
1.92
1.75
2.95
4.4
49.15%
DiffuisonDB
Saliency IOU 4
56.82
55.88
30.3
30.11
23.59
21.65%
TV Loss
21.16
20.39
11.99
12.3
8.41
31.63%
VTCM f
1.67
1.79
3,20
3.19
4.39
37.62%
Syn Prompt
Saliency IOU
51.52
53.24
31.57
31.42
27.7
11.84%
TV Loss 4
17.85
21.46
15.63
15.61
11.37
27.16%
VTCM f
1.98
1.79
2.67
2.73
3.49
27.84%
Table I. Quantitative comparison of metries across different methods and datasets. Bold indicate the best scores.
Dataset
Metries
SD 1.5
Ours 1.5
Imp (%)
SD 2.0
Ours 2.0
Imp (%)
SDXL
Ours XL
Imp (%)
P2P Template
Saliency IOU
29.89
22.86
23.52
37.97
33.33
12.22
29.83
26.64
10.69
TV Loss 4.
14.11
8.81
37.56
17.73
12.06
31.98
12.09
9.10
24.73
VTCM f
2.95
4.40
49.15
2.52
3.38
34.13
3.41
4.24
24.34
DiffusionDB
Saliency IOU +
30.11
23.59
21.65
33.40
29.52
11.62
34.22
32,31
3.16
TV Loss 4
12.30
8.41
31.63
16.17
12.78
20.96
13.22
10.66
15.86
VTCM f
3.19
4.39
37.62
2.63
4.06
54.37
3.27
3,43
4.89
Syn Prompt
Saliency IOU +
31.42
27.70
11.84
38.59
36.22
6.14
28.84
24.77
14.11
TV Loss I
15.61
11.37
27.16
18.92
15.25
19.40
12.31
8.48
31.11
VTCM
2.73
3.49
27.84
2,42
2.83
16.94
3.59
4.78
33.15
Table 2. Performance comparison across different Stable Diffusion versions. Bold indicates the best scores for SD 1.5, which serves as primary baseline. "Imp" represents the percentage improvement the corresponding base model.
Dataset
Dall-E 3
AnyText
Desigen
Ours
in ten different random region R. The dataset com- bined synthesized prompts generated by ChatGPT, and 700 prompts from the Prompt2Prompt template (Hertz al., 2022) designed for attention guidance, focusing spe- cific objects and their spatial relationships. Additionally, we included 1,000 DiffusionDB prompts (Wang et al., 2022), chosen for their real-world complexity. This di- verse and comprehensive dataset, spanning synthetic to user-generated prompts, provided broad test ground evaluate the efficacy of our TextCenGen method in vari- ous T2I scenarios. Additionally, Desigen benchmark using 771 images from the Desigen dataset validation set (Weng et al., 2024), along with their corresponding static text masks, to evaluate performance on layout design spegnn-peccficc content. These 771 images were drawn from the original Desigen dataset of 53,577 usable images, where 52,806 were used for training the special- ized Desigen model in Table 4.
P2P Template
2.53
0.32
0.55
0.30
DiffuisonDB
2.04
1.04
0.34
0.31
Syn Prompt 2.25
1.02
0.51
0.29
Table 3. Quantitative comparison of CLIP Scores Loss for differ- methods. Bold indicates the best scores.
generated images, particularly in areas designated for text insertion.
constructed targeted
4. Experiments
The evaluation is structured into quantitative and qualita- tive analyses, alongside an ablation study to understand the contribution individual components of our model.
4.1. Implementation Details.
Experimental Settings. Our model built with Dif- fusers. The pre-trained models are stable-difuusion-v-5 and stable-diffusion--220 While generating, the size of the output images is 512 512. We A40 GPUs for evaluation. Detailed parameter settings provided in the appendix.
4.2. Comparison with Existing Methods
We compared TextCenGen with several potential models to evaluate its efficiency. The baseline models included: Native Stable Diffusion (Rombach et al., 2022), Dall-E 3 (Ramesh et al., 2022), AnyText (Tuo et al., 2023) and Desigen (Weng et al., 2024). Dall-E used the prompt "text- friendly in the (position]" to specify the region R. Similar to AnyText, we chose randomly generate several masks
A6000 and ten
Dataset for Evaluation. Our evaluation contains 27,000 images generated from 2,700 unique prompts, each tested
6

---

TextCenGen: Attention-Cuided Background Adaptation for T21
Dataset
Metrics
Desigen- +AR
Desigen- +AR
Ours (SD 1.5)
P2P Template
Salieney IOU +
30.62
34.99
22.8
TV Loss 4
13.70
14.77
8.81
VTCM f
2.92
2.97
4.40
DiffusionDB
Saliency IOU +
30.30
31.11
23.59
TV Loss 4
11.99
14.58
8.41
VTCM f
3.20
2.82
4.39
Syn Prompt
Saliency IOU +
31.57
32.60
27.70
TV Loss 4
15.63
14.51
11.37
VTCM f
2.67
2.93
3.49
Table 4. Comparison with trained versions of Desigen general datasets. Desigen-TF: Training Traeeing-free, Desigen-T. T: Trained, AR: Atten- tion Reduction. Even with specialized training graphic design data, Desigen-T does not match our training free method's perfor- mance.
blucktt fullon supples
Photo of tree made with
slean
Whimmiaal fuines and glowing
SD-1.5
Dall-E
Anytext
Desigen
Ours
Figure 4. The results of comparison. Each column showcases six prompts across three datasets, the final column depicting the saliency map result image generated from mushroom prompt. red-dotted area denotes planned blank region, Note some methods fail to follow the orange- highttiigt words in the prompt, leading to semantic loss.
Method D-TF D-T D-T+Ours
Saliency IOU 1
TV Loss 4
VTCM 1
the generated images. We proposed the CLIP Score Loss to evaluate the reduction in prompt semantic alignment for the training- method compared to the vanilla diffusion model. The CLIP Score (Hessel et al., 2021; Huang et al., 2021; Radford et al., 2021) measured semantic fidelity, en- suring images align with textual descriptions. The total variation (TV) loss (Rudin Osher, 1994; Jiang al., 2021) assessed the visual coherence and smoothness the background in relation text region R, crucial for harmo- nious compositions. Saliency Map Intersection Over Union (IOU) (Qin et al., 2019) quantified the focus and clarity around text areas. We proposed the Visual-Textual Con- cordance Metrie (VTCM), which combined a global met- rie increasing with value (CLIP Score) and local metries that benefit from lower values (Saliency IOU and TV Loss)
40.79
19.44
2.36
41.66
18.06
2.30
38.48
12.19
2.85
Ours (SD 1.5)
31.99
9.74
3.69
Table 5. Performance the specialized Desigen benchmark dataset. D-TF: BeiigenTraaiiin-ffee D-T: Desigen-Traind Trained.
Underline indicates second-best performance.
in fixed pattern across the map to simulate regions need to be edited. More detail can be found in appendix.
Metrics and Quantitative Analysis. To evaluate model performance, we used metrics assessing various aspects of

---

TextCenGen: Autention-Guided Background Adaptation for T21
Dataset
Metrics
Text2Poster Best@5
Tex(2Poster Avg@5
Ours (SD 1.5)
P2P Template
Saliency IOU +
31.25
36.22
22.86
TV Loss 4
11.49
12.45
8.81
VTCM f
2.48
2.28
Clip Score f
20.80
20.80
27.96
DiffusionDB
Saliency IOU
24.07
34.38
23.59
TV Loss +
7.99
10.65
8.41
VTCM f
2.93
2.22
4.39
Clip Score f
17.57
17.57
27.20
Syn Prompt
Saliency IOU +
31.25
36.91
27.70
TV Loss &
11.49
13.28
11.37
VTCM f
2.48
2.16
3.49
Clip Score f
20.80
20.90
28.10
Table Comparison with retrieval- based methods. Text2Poster Best@5 shows the best results selected from five preset positions, while Avg@5 shows the average. Our method outperforms Text2Poster most metrics, particularly CLIP Score, demonstrating better semantic fidelity while maintaining exttfriendllness.
within R. The VTCM formula is:
(a) TV Loss vs Clip Score
(b) Saliency JOU Clip Score
VTCM CLIP Score CLIP Score
(5)
Saliency IOU
TV Loss
Oln Seage
Our quantitative analysis in Table and Table 3 shows that TextCenGen minimizes semantic loss while maintaining smoothness and saliency harmony within region R, even surpassing the latest Dalle-3. Scene text rendering and at- tention reduction Desigen often focus on local attention, neglecting global semantics. Especially in Text in Object situation (see Figure 3), this weakens the ability to create whitespace. Our approach moves main objects away, then reduces attention create space, resulting in more natural and harmonious text layouts with the highest VTCM. The trade-off between background smoothness and semantic fi- delity shown in Figure 5.
TV Loss
Salumncy 1OU
Figure 5. Performance trade-offs between different metrics. The dashed lines represent iso-utility curves, where points the curve indicate equivalent trade-off levels. Our method achieves a better balance between background smoothness and semantic fi- delity. Higher utility curves (green) represent better overall per- formance.
judge methods (Chen et al., 2024; Wu et al., 2024), we present the MLLM-as-Judge ELO ranking for design ap- peal across different datasets in Table 7. The results demon- strate that the structured output from the GPT-4o provides consistent ratings, indicating that our method, along with Anytext and Dall-E, shows advantages in terms of design appeal. Interestingly, Dall-E excels in the Synthesized Prompts dataset, which contains higher proportion of nat- ural landscapes. For detailed information on the prompts and evaluation, please refer to the appendix.
We compared our method with the trained version Desi- gen and retrieval-based methods like Text2Poster (Jin al., 2022). Table 4 shows that the trained version of Desi- gen performs less effectively than our method despite us- ing specialized graphic design data. As shown in Table 5, our plug-and-play method can be directly applied to pre- trained model weights, yielding superior results compared to the same model using only attention reduction. When integrated with Desigen-Trained, our approach improves performance significantly, demonstrating that training- free method can enhance specialized models without requir- ing additional training. As shown in Table 6, Poxt2Poster achieves competitive TV Loss on DiffusionDB. but our method provides better semantic fidelity (CLIP Score) and visual-textual coherence (VTCM). These results demon- strate the effectiveness of our training-free free approach com- pared both trained models and retrieval-basee methods.
Rank
Method
DDB
P2P
SP
TextCenGen
702.21
752.85
122.96
2
Anytext
279.56
329.32
-89.68
3
Dall-E
-17.32
-39.05
78.87
Desigen
-322.63
-291.80
-7.24
5
SD1.5
-629.33
-738.83
-92.40
Table 7. Method ELO design appealing rankings across three datasets.
MLLM-as-Judge ELO Ranking. Following the rising trend of multimodal large language model (MI.LM) as
8

---

TextCenGen: Attention-Guided Background Adaptation for T2I
Qualitative Analysis. Our qualitative analysis, shown in Figure 4, involved comparison across different models us- ing prompts and positions within the same quadrant. Dall-E 3, which relied solely on text inputs, exhibited significant variability and could not consistently clear the necessary ar- cas for text placement. Desigen reduced attention in region R, but this method was not always effective without pre- trained graphic design-specific model weights, especially when region R was within main object, such in the bi- cycle example. Introducing text to images was tricky, as TV Loss showed, but TextCenGen maintained object detail and background quality well. It showed that our force-directed method effectively balances text and visuals in images.
(al) Friendly:
(b1) Text-Friendly Selection
Frequency of Issues Encountered
Sdicoal Tines
X7
Averge Teees
Vey Cousstent (44.2
Cousistou (39-47%)
Incomiston
Incominentt (2.65%)
(22) Importance of the Question
(b2) Questions of Text-Friendly, Prompt-Image Image Alignment, Aesthetics
MonRonkk Ronk, one hest
Importast (44, 9996
Nay Impootaant (44 74h
Nestul 19%5
User Study. To understand the importance of the text- friendliness issue and explore users' subjective perceptions of the T2I Model's results, conducted a user study with 114 participants. We used the qualitative results (see Fig- ure 4) to develop questionnaire. Figure 6 illustrates that the prompt-image alignment and aesthetics of our results were well-received in human perception.
Very Unmmareeritts)) Uningettare (2.65%)
Figure 6. User study of task importance and result evaluation. The left shows user perceptions of task encounter frequency (al) and importance (a2), rating 5 as the highest. The right side details user rankings (b2) and the average chosen times in multiple-selectinn scenario (bl). The y-axis in b2 represents rankings from I to 5, demonstrating significant mean differences (a=0.05) across three standards for all methods.
4.3. Ablation Study
Table 8 presents the results of our ablation study. As part of our comprehensive analysis, we evaluated the two main contributors: (I) the impact of the force-directed compo- nent and (2) the effectiveness of the Spatial Excluding Cross-Attention Constraint.
w/o All
w/o FDG
w/o SEC
Ours
CLIPS Loss J TV Loss Saliency IOU 4
2.2
1.71
0.32
14.39
12.44
12.76
8.81
30.32
28.56
28.61
22.86
VTCM
2.93
3.03
3.05
4.4
Impact of Force-Directed Cross-Attention Guidance. The force-directed module is key to gently shifting where objects are placed. Without this, we might just bluntly edit the cross-attention map, which could mess up important parts of the picture. This part of our model helps make sure we don't 't ruin the image structure by harshly removing attention map from areas.
Table 8. Ablation study results. We examine TextCenGen with or without the implementation of Force-Directed Cross-Attention Guidance (FDG) and Spatial Excluding Cross-Attention Con- straint (SEC). The CLIP Score Loss w/o both indicates the use of the vanilla SD-1.5 without our training-free frre method, resulting in no loss (-).
attention guidance to arrange whitespace. Furthermore, we have integrated a system to identify and relocate conflicting objects and spatial exclusion cross-attention constraint for low saliency in whitespace areas.
Effects of Spatial Excluding Cross-Attention Con- straint. Despite successful relocation of conflict object- related tokens, spaces left behind may not inherently lead to well-blended transition. Our experimental results un- derscore that the integration of the Spatial Excluding Cross- Attention Constraint improve the smoothness of the remain- ing image sections.
Our approach has certain limitations. The force-directed cross-attention guidance, which assumes convexity and centering on object centroids, may not be suitable for non- convex shapes. This may lead to reduced size or fragmenta- tion of objects. Future work will address these challenges to improve the quality of the output images.
5. Conclusion
We present TextCenGen, plug-and-play method for text- friendly image generation and requires no additional train- ing while well balancing both semantic fidelity and visual quality. This method abandons the traditional method of adapting text to pre-defined images. TextCenGen modi- fies images to adapt text, employing force-directed cross-
A Acknovledgements
This work was supported by the NSSFC under Grant 22ZD05. We thank the anonymous reviewers and Dr. Sicheng Song for their valuable feedback and suggestions.

---

TextCenGen: Attention-Guided Background Adaptation for T21
Impact Statement
generation. Advances in Neural Information Processing Systems, 2023.
This paper presents work whose goal to advance the field of Machine Learning. There are many potential societal consequences our work, none of which feel must be specifically highlighted here.
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H.,, Levi, Y., Lorenz, D., Sauer, A., Boesel, F. et al. Scaling rectified flow transformers for high- resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024.
References
Clipscore.
https://githab..com cl clipscore,202
Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley. D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. Advances in neural infor- mation processing systems, 27, 2014.
Avrahami, O., Hayes, T., Gafni, O., Gupta, S., Taigman, Y., Parikh, D., Lischinski, D., Fried, O., and Yin, X. Spatext: Spatio-textual representation for controllable image gen- eration. In CVPR, pp. 18370-18380 2023.
Guo, S., Jin, Z., Sun, F., Li, J., Li, Z., Shi, Y., and Cao, N. Vinci: an intelligent graphic design system for generat- ing advertising posters. In CHI, pp. 1-17, 2021.
Balaji, Y., Nah, S., Huang, X., Vahdat, A., Song, J., Kreis, K., Aittala, M., Aila, T., Laine, S., Catanzaro, B., et al. ediffi: Text-to-image diffusion models with ensemble of expert denoisers. arXiv preprint arXiv:2211.01324, 2022.
Hertz, A., Mokady, R., Tenenbaum, J.. Aberman, K., Pritch, Y., and Cohen-Or, D. toppptt---romptt im- age editing with cross attention control. arXiv preprint arXiv:2208.01626 2022.
Hessel, J., Holtzman, A., Forbes, M., Le Bras, R., and Choi, Y. Clipscore: A ceference-ffee evaluation metric for im- age captioning. In EMNLP, pp. 7514-7528, 2021.
Cao, Y., Chan, A. B., and Lau, R. W. Automatic stylistic manga layout. ACM Transactions on Graphics, 31)6): 1-10, 2012.
Ho, J., Jain, A., and Abbeel, Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-688 2020.
Chai, S., Zhuang, L., and Yan, F. Layoutdm: Transformer- based diffusion model for layout generation. In CVPR, pp. 18349-18358, 2023.
Huang. Y., Xue, H., Liu, B., and Lu, Y. Unifying multi- modal transformer for bi-directional image and text gen- eration. In ICML, pp. 1138 1147, 2021.
Chen, D., Chen, R., Zhang, S., Wang. Y., Liu, Y., Zhou, H., Zhang, Q. Wan, Y., Zhou, P., and Sun, I.. Mllm-as-a- judge: Assessing multimodal lim-as-a-judge with vision- language benchmark. In JCML, 2024.
Hui, M., Zhang, Z., Zhang, X., Xie, W., Wang, Y., and Lu, Y. Unifying layout generation with decoupled diffusson model. In CVPR, pp. 1942-1951, 2023.
Chen, J., Huang, Y., Lv, T., Cui, L., Chen, Q., and Wei, F. Textdiffuser: Diffusion models as text painters, arXiv preprint arviv:2305. /0850 2023a.
Inoue, N., Kikuchi, K., Simr-Serra, E., Otani, M., and Yam- aguchi, K. Towards flexible multi-modal document mod- els. In CVPR, pp. 14287-14296, 2023a.
Chen, J., Huang, Y., Lv, T., Cui, L.., Chen, Q., and Wei, Textdiffuser-2: Unleashing the power of language mod- els for text rendering, arXiv preprint arXiv:2311.16465 2023b.
Inoue, N., Kikuchi, K., Simo-Serra, E., Otani, M., and Yam- aguchi, K. Layoutdm: Discrete diffusion model for con- trollable layout generation. CVPR, pp. 10167-10176. 2023b.
Chen, X., Wang, Y., Zhang, L., Zhuang, S., Ma, X., Yu, J., Wang, Y., Lin, D., Qiao, Y., and Liu, Z. Seine: Short-to- long video diffusion model for generative transition and prediction. arXiv preprint arXiv:2310.200
Jiang, S., Li, C., and Wang, C. Copaint: Guiding sketch painting with consistent color and coherent generative adversarial networks. In CGI, 2021, 2021.
2023c.
Duan, H., Yang, J., Qiao, Y., Fang. X., Chen, L., Liu, Y., Dong, X., Zang, Y., Zhang, P., Wang, J., Lin, D., and Chen, K. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models, 2024. URL https:/ /awivv.vvvv. org/ abs/ 11691.
Jin, C., Xu, H., Song, R., and Lu, Z. Text2poster: Laying out stylized texts on retrieved images. In ICASSP, pp. 4823 4827. IEEE, 2022.
Jyothi, A. A., Durand, T., He, J., Sigal, L., and Mori, G. Layoutvae: Stochastic scene layout generation from a label set. In ICCV, pp. 9895-9900 2019.
Epstein, D., Jabri, A., Poole, B., Efros, A. A., and Holyn- ski, A. Diffusion self- geff-guidance for controllable image
10

---

TextCenGen: Attention-Cuided Background Adaptation for T21
Li, C., Zhang, P., and Wang, C. Harmonious textual layout generation over natural images via deep aesthetics learn- ing. IEEE Transactions Multimedia, 24:3416-3428, 2022.
detection. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In ICML, pp. 7748-8763, PMLR, 2021.
Li, F., Liu, A., Feng. W., Zhu, H., Li, Zhang, Z., Lv, J., Zhu, X., Shen, J., Lin, Z.., and Shao, J. Relation-aware diffuuson model for controllable poster layout generation. In Proceedings of the 32nd ACM International Confer- ence on Information and Knowledge Management, pp. 12491258, 2023a.
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204066125 2022.
Li, J., Yang. J.. Hertzmann, A., Zhang, J., and Xu, T. Lay- outgan: Generating graphic layouts with wireframe dis- criminators. In International Conference on Learning Representations, 2019.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Om- mer, B. High-resolution image synthesis with latent dif- fusion models. CVPR, pp. 10684-10695, 2022.
Li, Y., Liu, H., Wu, Q., Mu, F., Yang. J., Gao, J., Li, C., and Lee, J. Gligen: Open-set grounded text-to-image generation. In CVPR, pp. 22511-22521 2023b.
Rudin, L. I. and Osher, S. Total variation based image restoration with free local constraints. In Proceedings of 1st international conference image processing, vol- ume 1, pp. 31-35. IEEE, 1994.
Liu, X., Park, D. H., Azadi, S., Zhang, G., Chopikyan, A., Hu, Y., Shi, H., Rohrbach, A., and Darrell, T. More control for free! image synthesis with semantic diffusion guidance. In WACV, pp. 289-299, 2023.
Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E. L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al. Photorealistic text-to-image diffusion models with deep language un- derstanding. Advances in Neural Information Processing Systems, 35:36479 11444499-36444, 2022.
Ma, J., Zhao, M.. Chen, C., Wang, Niu, D., Lu, H., and Lin, X. Glyphdraw: Learning draw chinese characters in image synthesis models coherently. arXiv preprint arXiv:2303. 13300.117777 17870, 2023.
Singer, U., Polyak, A., Hayes, T., Yin, X., An, J., Zhang, S., Hu, Q., Yang, H., Ashual, O., Gafni, O., Parikh, D., Gupta, S., and Taigman, Y. Make-a-video: Text-to-video generation without text-video data, In ICLR, 2023.
Meng. C., He, Y., Song. Y.. Song, J., Wu, J., Zhu, J.-Y., and Ermon, S. Sdedit: Guided image synthesis and edit- ing with stochastic differential equations. arXiv preprint arXiv:2 (0ti:2/08.0007.... 2021.
Tuo, Y., Xiang, W., He, J.-Y., Geng, Y., and Xie, X. Any- text: Multilingual visual text generation and editing. arXiv preprint arXiv:2311.0305. 2023.
Mo, S., Mu, F. Lin, K. H., Liu, Y., Guan, B., Li, Y., and Zhou, B. Freecontrol: Training-free spatial control of any text-to-image diffusion model with any condition. arXiv preprint arXiv:2312.07536. 2023.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. At- tention is all you need. Advances in neural information processing systems, 30, 2017.
Nichol, Q. Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., Megrew, B., Sutskever, I., and Chen, M. Glide: Towards photorealistic image generation and edit- ing with text-guided diffusion models. In ICML, pp. 16784-16804, PMLR, 2022.
Wang, R., Chen, Z., Chen, C., Ma. J., Lu, H., and Lin, X. Compositional text-to-image synthesis with atten- tion map control of diffusion models. arXiv preprint arXiv:2305. 1392/, 2023.
O' Donovan, Agarwala, A., and Hertzmann. A. Learning layouts for kinle-pagegraphte designs. IEEE Transac- tions Visualization and Computer Graphics, 20:1200- 1213, 2014.
Wang. X., Darrell, T., Rambhatla, S. S., Girdhar, R., and Misra, Instancediffusion: Instance-level control for image generation, 2024.
Patashnik, O., Garibi, D., Azuri, I., Averbuch-Elor, H., and Cohen-Or, D. Localizing object-level shape variations with toxt-to-image diffusion models. In ICCV, 2023.
Wang. Z. J., Montoya, E., Munechika, D., Yang, H., Hoover, B., and Chau, D. H. DiffuusonDD: A large- scale prompt gallery dataset for text-to-image generative models. arXiv:2210. 14896 [cs], 2022. URL https:
Qin, X., Zhang, Z., Huang, C., Gao, C., Dehghan, M., and Jagersand, M. Basnet: Boundary-awan salient object
//arxiv. //arxiv..rg/abs//220.00

---

Tex:CenGen: Attention-Cuidel Background Adaptation for T21
Weng, H., Huang, D., Qiao, Y., Hu, Z., Lin, C.-Y,, Zhang, T., and Chen, C. L. P. Desigen: A pipeline for control- lable design template generation, 2024.
Wu, T., Yang. G., Li, Z.,, Zhang, K., Liu, Z., Guibas, L.., Lin, D., and Wetzstein, G. Gpt-4v (ision) is human- aligned evaluator for text-to-3d generation. In CVPR, pp. 22227-22238, 2024.
Xie, J., Li, Y., Huang, Y., Liu, H., Zhang, W. Zheng, Y., and Shou, M. Z. Boxdiff: Text-to-image synthesis with training-free box-constrained diffusion. In ICCV,
pp. 7452-7461, 2023.
Zhang, L.., Rao, A., and Agrawala, M. Adding conditional control to text-to-image diffusion models. In ICCV, pp. 3836-3847, 2023.
Zhang. L.,, Chen, X., Wang, Y., Lu, Y., and Qiao, Y. Brush your text: Synthesize any scene text on images via diffu- sion model. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 7215-7223, 2024.
Zhang, P., Li, C., and Wang, C. Smarttext: Learning to generate harmonious textual layout over natural image. In 2020 IEEE International Conference on Multimedia and Expo (ICME), pp. 1-6. IEEE, 2020.
Zhao, S., Chen, D., Chen, Y-C.,, Bao, J., Hao, S., Yuan, L.., and Wong, K.-Y. K. Uni-controlnet: All-in-one control to text-to-image diffusion models. Advances in Neural Information Processing Systems, 2023.
Zheng, X., Qiao, X., Cao, Y., and Lau, R. W. Content- aware generative modeling of graphic design layouts. ACM Transactions on Graphics, 38(4):1-15, 2019.
Zhou, D., Li, Y., Ma, F., Zhang, X., and Yang, Y. Mige: Multi-instance generation controller for text-to-image synthesis, 2024a.
Zhou, D., Xie, J., Yang, Z., and Yang. Y. 3dis: Depth- driven decoupled instance synthesis for text-to-image generation. arXiv preprint arXiv:2410. 12669, 2024b.
Zhou, M., Xu, C., Ma. Y., Ge, T., Jiang, Y., and Xu, W. Composition-aware graphic layout gan for visual- textual presentation designs. In Raedt, L. D. (ed.), Proceedings of the Thirty-Firs International Joint Con- ference on Artificial Intelligence, IJCAI-22, pp. 4995- 5001. International Joint Conferences on Artificial Intel- ligence Organization, 7 2022. doi: 10.249633ijcai 692. URL https: oroioorg/10.. 24 63 /jjcai. 2022/6 692. 692. Al and Arts.
12

---

Tex:CenGen: Attention-Cuided Background Adaptation for T2I
We provide details of the proposed method and additional experimental results to help better understand our paper. Contents
Task Introduction
13
Experiment Setting
14
B.1 Region Random Sampling Method
14
B.2 Analysis of Text Box Shape Orientations
14
B.3 Evaluation Metrics
15
B.4 Details of Compared Methods
16
B.5 MLLM-as-Judge ELO Ranking...
17
Influence of the force balance constant
17
More Results of Proposed Method
18
D.1 More Result of Ablation Study
18
D.2 Compatible with Lora Checkpoint
18
Limitations of Our Model
18
Prompt: "Realistic Jeju Island view with Hallasan Mountain." Icon.0c
Stable Diffusion
Origin
J0g
Result
top
buttom
middle
left
right
Figure 7. Can you spot the TCG logo first glance? TextCenGen training-free method designed generate text-friendly images. It simultaneously generates the original and result images. During the denoising process of the original image, the generation of the result image guided based on encroachment into the planned blank regions in the original. This approach ensures sufficient blank space at specific positions, typically where text or icons are centered, in the result image.
A. Task Introduction
Text-friendly images refer to images designed or selected with an emphasis on enhancing the readability and clarity of overlaid text. These images typically have simple, non-distracting backgrounds, balanced color palette, and areas of
13

---

TextCenGen: Attention-Cuided Background Adaptation for T21
negative space that can accommodate text without compromising visibility design aesthetics. Common applications include marketing materials, presentations, and social media graphics where the text plays a crucial role in conveying the message. Key considerations for creating or selecting text-friendly images include contrast, alignment, and ensuring the image content does not compete with the overlaid text.
Don't Requires Training
Don't Requires Annotation
Type of Layout Specification
Required Number of Anchors
Task Name
Layout-to-lmage (Rombach et al., 2022)
X
Object Space Region
> 5
Text-Friendly Image Generation Visual Text Generation
V
1-2
Text
1-2
Table Comparison our task with existing tasks. Unlike layout-to-imne tom-ia tasks requiring training and intensive annotation, our method only needs space region annotation as input for downstream modifications. This makes it particularly suitable for applications such as dynamic wallpapers for mobile devices and e-commerce posters.
B. Experiment Setting
Our proposed model is designed using the Diffusers library, specifically leveraging the stable-diffusion--l-5 pre-trained models with DDPM Scheduler. The model generates images of dimensions 512 x 512. In our method, we have set the force balance constant a to 0.5. The coefficients for regularization term ạ is fixed at 0.01. Within the detector, we upscale all cross-attention maps to a 64 x 64 resolution. Additionally, we expand the height and width of the region by a margin of 0.06. During the first 20 steps, we identify conflicting objects when the Intersection over Union (IOU) exceeds 0.14. For the subsequent 30 steps, we initiate push operation only if the average density inside region R surpasses 0.8. Neg- ative prompts are "monocolor, monotony, cartoon style, many texts, pure cloud, pure sea, extra texts, texts, monochrome, flattened, lowres, longbody, bud anatomy, bad hands, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality".
Our mros-aateention replacement method requires less than 15GB of VRAM, making it feasible to run inference on con- sumer GPUs like the RTX 3090. For evaluation purposes, we utilized one NVIDIA A6000 and 8 H800 GPUs. The entire experimental assessment took approximately
hours to complete. Particularly on the A40 GPU, the image generation process, which includes both the original and the resultant images, took around 50 seconds for 50 steps of inference. In contrast, utilizing attention guidance with difftext results in faster average inference time of approximately 30 seconds, as it only requires the inference of a single image.
B.1. Region Random Sampling Method
Our region random sampling method is variation inspired by DiffText. It involves two predefined regions, measuring 160 x 64 and 64 x 160. During the evaluation, excluding Dall-E, we randomly select areas of these dimensions from the entire image. The output image of Dall-E is not 512 x 512, so we proportionally scale down the corresponding regions to 116 x 46 and 46 116 while calculating metrics. Given that Dall-E operates exclusively on textual inputs as provided by the prompt, we intend to specify regions within Dall-E's output at five distinct locations: left, right, bottom, top, and center.
B.2. Analysis of Text Box Shape Orientations
The shape orientation of text boxes was randomly generated following our region random sampling method, ensuring an unbiased experimental setup. To investigate potential biases related to text box orientations, we conducted additional analysis by separating results based on horizontal (160 x 64) and vertical (64 160) orientations. Table 10 shows the differences between horizontal and vertical orientations across different metries and datasets, calculated as horizontal minus vertical values.
TextCenGen shows relatively consistent performance across orientations, with slightly better metrics for horizontal boxes in the P2P Template dataset. Different methods exhibit varying preferences for orientation, suggesting the underlying complexity of text-friendly image generation across different box shapes. The variance in performance across orientations generally smaller for TextCenGen compared to baseline methods, indicating more robust handling of different text box
14

---

Tex:CenGen: Attention-Cuidel Background Adaptation for T21
Dataset
Metrics
SD
Dall-E 3
AnyText
Desigen
Ours
Saliency IOU
-1.54
0.17
-1.17
-0.57
0.31
P2P Template
TV Loss
-4.83
5.89
-9.77
-1.26
3.77
VTCM
-0.42
0.00
0.00
0.09
0.55
Saliency IOU
0.83
-0.94
0.00
0.32
0.00
DiffusionDB
TV Loss
2.57
2.00
-3.48
2.49
-0.66
VTCM
-0.32
1.76
-0.12
0.09
-1.14
Table 10. Differences between horizontal and vertical text box orientations across metrics and datasets. Positive values indicate better performance with horizontal boxes.
shapes.
B.3. Evaluation Metries
To evaluate model performance, we used 4 metrics assessing various aspects the generated images. The CLIP Score, total variation (TV) loss, Saliency Map Intersection Over Union (IOU) and Visual-Textual Concordance Metric (VTCM).
CLIP Score described as metric used to measure the similarity between generated images and input prompts, employing off-the-shelf code referenced as (cli, 2022). Typically, the original image achieves the highest CLIP score when it's associated with stable diffusion techniques. However, when blank areas defined within an image, there might be slight reduction in the CLIP Score. This score is crucial in determining how closely Al Al-generated image aligns with the given textual prompt, playing significant role in the overall evaluation of the image's fidelity to the intended text description.
Total Variation Loss is regularisation term commonly used in image processing tasks, particularly those involving image reconstruction or denoising. is designed to encourage spatial smoothness in the output image while preserving important structural details. The total variation loss for a region R can be computed as follows:
TV(R) = E (.A. (A,, Ray))
SJER
Here, A, Rag and Ry represent the discrete differences in the horizontal (x-axis) and vertical (y-axis) directions, respectively, at pixel location (i.j) within the region R. The term Rs, (Ay calculates the squared gradient magnitude at each pixel, and the summation is taken over all pixels within the region R.
This formulation of the total variation loss ensures that the reconstructed or processed region R does not have abrupt changes in pixel values, leading to a smoother and more visually appealing result.
Saliency Map Intersection Over Union is specifically tailored for assessing the overlap between a saliency map and designated region within an image. The formula for this metric is given as:
S R S U R
IoU R)
In this context, S represents the saliency map, and R denotes a specific region in the image. The term n R] is the count of pixels that are common both the saliency map and the region R, signifying their intersection. On the other hand, JS U R| refers the count of pixels present in either the saliency map S or the region indicating their union.In our case, lower loU value is desirable as it indicates less overlap, aligning with our goal ensure that the ROI is non-salient and distinct from the saliency map.
Concordance Metric is formulated to assess the coherence between text prompts and generated images in AI-driven text-to-image synthesis. Defined VTCM CLIP Score x Saliency IOU TVioss) it combines three elements. The CLIP Score reflects the degree of match between the generated image and the text prompt, where higher scores indicate better alignment. The Saliency IOU (Intersection Over Union) in Region R measures how well the most salient parts of the image align with the specified region, with lower scores being better. The TV Loss in
15

---

TextCenGen: Attention-Cuidel Background Adaptation for T21
Region R assesses the smoothness consistency in the image's region, where lower TV Loss scores indicate uniform and less noisy region. The VTCM thus encourages the generation of images that not only coherent with the text prompt but also exhibit focused quality in specified areas, aligning with the goal of creating text-friendly images.
Figure The Limitation Model.
A playful dolphin jumping out of the ocean, text-friendly in the {position)
JOPEM HERE
Lonem Ipsum
image:
Home
position:
buttom
top
middle
left
right
Figure 9. The data generation method in Dall-E.
B.4. Details of Compared Methods
We compare the proposed model with Stable Diffusion, Dall-E, AnyText and DiffText. The details of compared methods are follows:
Stable Diffusion, detailed in Rombach al. (2022) (Rombach et al., 2022), is innovative open-source model. For our experiments, we set the sampling steps to 50 and the classifier- free guidance scale at 7.5. The model serves both as the source for generating attention guidance from the origin image in our methodology and as one of the compared methods.
We utilize the widely accessible pre-trained model labeled as trunwayml/saaleddfftiiioo.
Dall-E (Ramesh et al., 2022) a groundbreaking Al model developed by OpenAI, known for its capability to generate complex images from textual deseriptions. Utilizing a variant of the GPT-3 architecture, Dall-E transforms inputs
16

---

Tex:CenGen: Attention-Guidel Background Adaptation for T21
into detailed and creative visual outputs, showcasing deep understanding of both language and visual concepts. In experiments, the output image of Dall-E is not 512 512, so we proportionally scale down the corresponding regions to 116 X 46 and 46 x 116 while calculating metrics. To enhance the focus text compatibility these positions, we add the phrase 'text-friendly in the corresponding position' directly into the original prompts, as depicted in Figure 9. Restricted to the purely textual input of Dall-E, the performance falters when the position falls between left and right and when the text lacks precision.
AnyText Text (Tuo al., 2023) is diffusion-based multilingual visual text generation and editing model that focuses on rendering accurate and coherent text in the image that outperformed all other approaches. We generate a fully black mask for that region and fully white for the rest of the image. Then we use this mask to replace the "draw_pos" in the input parameters, while keeping the other parameters unchanged.
Desigen (Weng et al., 2024) is an automatic template creation pipeline which generates background images as well as harmonious layout elements over the background as well as an iterative inference strategy to adjust the synthesized adopted the following approaches: (1) (2) Our comparison targets training- free
background and layout in multiple rounds is presented. For fair comparison, We used random regions as layout elements, similar to other baselines. methods, so we used vanilla SD1.5 without any LoRA as the base weights. (3) Using Desigen's attention reduction method on this base weight, we set the attention ratio within the region to 0.
B.5. MLLM-as-JJdge ELO Ranking
To comprehensively assess the performance of TextCenGen against other baseline methods, we employed the Elo rating system, method originally developed for ranking chess players but now widely used in various competitive contexts (Duan al., 2024; Chen et al., 2024; Wu al., 2024). In our evaluation framework, we apply a multi-moda large language model (MLLM) as a judge to compare TexiCenGen with other baseline methods. The evaluation is based on the Elo rating system, commonly used in competitive ranking, which allows for continuous adjustments of scores as pairwise comparisons are made. Specifically, for each comparison between two methods, the MLLM assesses the "Design Appeal" of the generated outputs and determines a winner. The Elo method then updates the ratings of the two competing methods accordingly.
The Elo rating process involves calculating the expected scores EΛ and En for methods and B, given their current ratings R and Rg. The expected scores are computed using the formula:
E =
EB =
(6)
RB RA
RA RB
10
400
1 10
400
Based on the comparison outcome SA (where SA 1 if method A wins, or SA 0 if it loses), the ratings are updated using:
RA = RA + K (SA EA)
R' = RB + K x ((1 (SA) - EB)
Here, K represents the adjustment factor, set to 32 in our experiments, which determines the sensitivity of rating changes.
In practice, our evaluation system iterates through each dataset, initializing the Elo scores for all methods and adjusting them dynamically as new comparison results are processed. After all comparisons, the methods are ranked based on their aggregated Elo scores, providing insights into the relative strengths of TextCenGen and other baselines in terms of "Design Appeal." " This approach ensures consistent and scalable evaluation across diverse datasets while reflecting the preferences derived from the MLLM's judgments.
C. Influence of the force balance constant
a ensures that the forces exerted do not exceed a practical threshold, thereby maintaining visual equilibrium in complex scenes (shown in Figure 12).
17

---

Tex:CenGen: Inlention-Guidee Background Adaptation for T2I
Prompt
Donara
Ours
RenoOTe
Oheeg palo Beveni uodee Srossnal
teegg can
Alandosad space stution otbting
mnne enciyin en und surign on
A canene Vasaraile undar
cherr soheee compuetion
ditart auplanet
mrrcc lmag. whiie with wetn lucee
MILMM
Avant
"TANO NET chie Porr
carsain samall nazast
A Aandacapa Image mountaina right.
Gen peropoon comperto moaet leagges
Judgo
MLLM
intureenil on mor (no men der
feattng slow Carth with boveal's
Landicaps Imagedt formst surriss
mushig
MIL www----kedgge
Figure 10. More results of elo milm judge.
D. More Results of Proposed Method
We present additional outcomes of the proposed methodology. Referring to Figure 11, model produces 12 instances drawn from three distinct datasets. The figure also illustrates the strong performance of approach across diverse scenarios. These encompass urban landscapes, natural scenes, still life, Valentine's ' Day greeting cards, and environments from both 3D and 2D video games.
D.1. More Result of Ablation Study
Figure 13 presents the results of our ablation study. This study was designed to elucidate the individual contributions of the various components embedded within our proposed model. As part of our comprehensive analysis, we evaluated the three main contributors: (1) the impact of the force-directed component, (2) the effectiveness of the Spatial Excluding Cross-Attenton Constraint.
D.2. Compatible with Lora Checkpoint
Figure demonstrates the effect of using LoRA by utilizing the LoRA rev-animated model from Civitai for an animated style, while the qualitative evaluation uses original SD weights for comparison.
E. Limitations of Our Model
Our approach could result in unexpected changes, exemplified by the left of Figure 8, where might initially think the figure shifts downward, rather than the forehead area. We noted the emergence of unintended objects within the empty spaces on the right of Figure 8, spaces which the original prompts did not specify.
18

---

Tex:CenGen: Iolention-Guidedd Background Adaptation for T2I
Figure II. More results of the proposed method. The first and third lines display the original images, while the second and fourth lines exhibit the result images.
19

---

TextCenGen: Attention- Background Adaptation for T21
1.0
0.8
Calculated Force
0.6
0.4
0.2
force_ ciree_alance_ecee _ooosccccctt_ 0.1
force balance_coaaeee const 0.5
force, latee_allnne_eeeeee const 1
0.0
force balance const 2
0
4
10
f (Force)
Figure 12. Influence of the force balance constant. The graph illustrates the effects four different values for on Frep versus the final force. The curves reveal that increases, the rate of convergence to the limit progresses more rapidly.
w/o FDG, w/o SEC
w/o FDG
w/o SEC
Ours
Figure 13. The results of ablation. We examine TextCenGen with without the implementation of Force- DiiretDde Cross-Attentie Guidance (FDG) and Spatial Excluding Cross-Auention Constraint (SEC). The red-dotted area denotes the target area preserved for either text icon images. Images produced with both FDG and SEC yield outcomes identical to those created using TextCenGen alone. Conversely, images created without FDG and SEC are comparable to those derived from the original stable diffusion model.
20


Reviews:
[{"review_id": "0QcJ4CBcJd", "reviewer": "Reviewer_kAwC", "summary": "This paper aims to solve the problem of generating a background image conditioning on intended size and position of overlaying texts. The proposed approach, TextCenGen, is to generate a regular image and a background image at the same time, and use the text intended region and attention map from the regular image to guide the generation of background image such that it leaves space for texts. \n\n2700 text prompts were used as background captions to generate images. The source of text prompts include ChatGPT generated, Prompt2Prompt, and DiffusionDB. Baselines are other off-the-shelf models in their training-free versions. Qualitatively, the TextCenGen leaves a cleaner space in the background image compared to selected baselines; quantitatively, TextCenGen scores better numbers for lower saliency overlap and higher semantic fidelity.\n\n## update after rebuttal\nThe provided additional results comparing with baselines in their trained versions and retrieval based method are highly appreciated. This answers my question, and it shows that the proposed method indeed provide additional improvements over existing work. As such, I update my recommendation to 3 weak accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable review.\n\n### Evaluation on Poster/Ads Datasets\n\nWe selected the Desigen dataset as a layout design dataset for evaluation. We successfully downloaded 53,577 usable images, with 52,806 used for training the Desigen version. The remaining 771 images from the validation set and their corresponding static text masks were used to construct our new Desigen benchmark.\n\nOur core contribution is text-friendly background generation for arbitrary user-specified regions without training constraints. **This approach extends beyond posters and ads to applications like mobile wallpapers where users can specify any position for text, rather than being limited by the distributions in datasets [1,2].**\n\n### Comparison with Trained Versions of Baselines\n\nOur method focuses on a training-free approach due to the diverse range of user prompts and space region positions. Training often compromises generalization across subtasks.\n\nTable 1 shows Desigen-Trained + Attention Reduction improves over its training-free version on certain metrics. When combining our method with the trained Desigen model, we observe substantial improvements:\n\n| Dataset | Metrics | Desigen-Training-free + Attention Reduction | Desigen-Trained + Atention Reduction | Ours-Replace (SD 1.5) |\n|-|-|--|-|-|\n| P2P Template | Saliency IOU ↓ | 30.6 | 35.0 | **22.9** |\n| | TV Loss ↓ | 13.7 | 14.8 | **8.8** |\n| | VTCM ↑ | 2.9 | 3.0 | **4.4** |\n| DiffusionDB | Saliency IOU ↓ | 30.3 | 31.1 | **23.6** |\n| | TV Loss ↓ | 12.0 | 14.6 | **8.4** |\n| | VTCM ↑ | 3.2 | 2.8 | **4.4** |\n| Synthetics Prompt | Saliency IOU ↓ | 31.6 | 32.6 | **27.7** |\n| | TV Loss ↓ | 15.6 | 14.5 | **11.4** |\n| | VTCM ↑ | 2.7 | 2.9 | **3.5** |\n| Desigen benchmark | Saliency IOU ↓ | 40.8 | 41.7 | **32.0** |\n| | TV Loss ↓ | 19.4 | 18.1 | **9.7** |\n| | VTCM ↑ | 2.4 | 2.3 | **3.7** |\n\nTable above shows Desigen-Trained improves over its training-free version. \n\nWhen combining our method with the trained Desigen model, we observe substantial improvements:\n\n| Dataset | Metrics | Desigen-Trained+ Attention reduction | Desigen-Trained+Ours |\n|-|-|-|-|\n| Desigen benchmark | Saliency IOU ↓ | 41.66 | **38.48** |\n| | TV Loss ↓ | 18.06 | **12.19** |\n| | VTCM ↑ | 2.3 | **2.85** |\n\n\n### Comparison with Retrieval-Based Methods\nWe evaluated against Text2Poster [3] across all datasets. Our approach differs from retrieval methods: we support arbitrary user regions and create novel content beyond image databases. Since Text2Poster doesn't support user-specified space regions, we tested on preset positions (top, bottom, left, right, center) and selected the best result based on VTCM for fair comparison.\n\nOur related work section (line 083) explicitly mentions Text2Poster, which inspired us to consider background adaptation in the T2I era. We appreciate the reviewer highlighting this discussion direction, as it has provided us with valuable insights.\n\n| Dataset | Metrics | Text2Poster Best@5 | Text2Poster Avg@5 | Ours-Replace (SD 1.5) |\n|-|-|-|-|-|\n| P2P Template | Saliency IOU ↓ | 31.25 | 36.22 | **22.86** |\n| | TV Loss ↓ | 11.49 | 12.45 | **8.81** |\n| | VTCM ↑ | 2.48 | 2.28 | **4.4** |\n| | Clip Score↑ | 20.8 | 20.8 | **27.96** |\n| DiffusionDB | Saliency IOU ↓ | 24.07 | 34.38 | **23.59** |\n| | TV Loss ↓ | **7.99** | 10.65 | 8.41 |\n| | VTCM ↑ | 2.93 | 2.22 | **4.39** |\n| | Clip Score↑ | 17.57 | 17.57 | **27.2** |\n| Synthetics Prompt | Saliency IOU ↓ | 31.25 | 36.91 | **27.7** |\n| | TV Loss ↓ | 11.49 | 13.28 | **11.37** |\n| | VTCM ↑ | 2.48 | 2.16 | **3.49** |\n| | Clip Score↑ | 20.8 | 20.9 | **28.1** |\n\nOur method outperforms retrieval approaches in most text-friendliness metrics while preserving prompt semantics. \nThe emergence of autoregressive models like GPT-4o creates potential for combining Text2Poster's retrieval methodology with attention-guided RAG to achieve both text-friendliness and semantic fidelity.\n\n\n### Broader Impact\n\nOur work extends beyond text placement to other UI elements including icons (Appendix Figure 1). While approaches like TextDiffuser-2 handle text with specific styles through dedicated training, our method achieves user instruction following and arbitrary space region support through attention control. This approach avoids reliance on \"sign\" or similar elements in prompts (line 088-089), opening a pathway to more flexible controllable generation that preserves both user intent and element compatibility.\n\n\nWe appreciate your reconsideration of our paper.\n\n[1] Zheng G, Zhou X, Li X, et al. LayoutDiffusion: Controllable diffusion model for layout-to-image generation. CVPR 2023\n\n[2] Zhou, Min, et al. \"Composition-aware graphic layout GAN for visual-textual presentation designs.\" IJCAI, 2022\n\n[3] Jin, C., et al. Text2Poster: Laying out stylized texts on retrieved images. ICASSP 2022 [PS: Pioneering work]", "claims_and_evidence": "One major claim is that TextCenGen outperforms existing methods measured by automatic metrics (saliency IoU, CLIP score, Total Variation Loss, etc.) and user studies. The supporting evidence was 27000 images generated from 2700 selected text prompts, comparing with generic off-the-shelf models, Dalle-3, SD1.5, AnyText, or training-free version of another background generation work, Desigen. \n\nHowever, given that background generation for texts has its target use cases, like posters and ads, it would be more representative to evaluate on those datasets [1] [2].\n\nAlso, even if the proposed method is training free, it would be informative to compare with existing method as is; in particular, to compare with Desigen's trained version. \n\n[1] Weng, Haohan, et al. \"Desigen: A pipeline for controllable design template generation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n[2] Zhou, Min, et al. \"Composition-aware graphic layout GAN for visual-textual presentation designs.\" IJCAI, 2022", "methods_and_evaluation_criteria": "Instead of evaluating on generic text prompts, I think it would be more relevant to evaluate on captions from images in layout design datasets [1] [2].", "theoretical_claims": "There are no theoretical claims in this paper.", "experimental_designs_or_analyses": "The dataset to run experiments on could be more relevant to designs. The baseline to compare with could be more informative if they were not limited to their training-free version.", "relation_to_broader_scientific_literature": "If the work conditions not just the text position and size to generate a background, but further conditions on the text font style, color, and its semantic meaning or context, the impact would be broader.", "essential_references_not_discussed": "I appreciate the authors including discussions to retrieval based layout planning methods [3], and not just generation based only. It would be interesting to compare with retrieval based methods.\n\n[3] Jin, C., Xu, H., Song, R., and Lu, Z. Text2poster: Laying out stylized texts on retrieved images. In ICASSP, pp. 4823–4827. IEEE, 2022.", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jbxArLaEg0", "reviewer": "Reviewer_tH44", "summary": "This paper introduces TextCenGen, a training-free method for generating text-friendly images. While traditional text-to-image (T2I) models can create high-quality images, they typically don't account for the need to reserve space for text placement. TextCenGen addresses this challenge through several innovations:\n\n1) A dynamic background adaptation method that creates smooth backgrounds for predefined text regions\n2) A force-directed graph algorithm that guides the relocation of objects in attention maps to avoid overlap with text regions\n3) A spatial excluding cross-attention constraint that ensures background smoothness in text regions\n\nThe authors constructed a benchmark with 27,000 images, demonstrating that TextCenGen reduces saliency overlap in text regions by 23% compared to existing methods while maintaining 98% of the original semantic fidelity. The method requires no additional training and can be plugged into various T2I models.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable review. We address your concerns with additional experiments and analysis below.\n\n### [Q1,Q3]Multi-Object and Complex Scene Handling\n\nOur approach shows strong performance in complex scenes with multiple objects. We additionally evaluated on the Desigen dataset, which includes multiple text boxes representing a challenging multi-region scenario:\n\nIn multi-object cases, our method creates complementary forces guiding content away from specified regions. This approach extends to complex layouts as shown in our Desigen benchmark results:\n\n| Dataset | Metrics | Desigen training-free | Desigen-Trained | Ours (SD 1.5) |\n|---------|---------|----------------------|-----------------|---------------|\n| Desigen | Saliency IOU ↓ | 40.8 | 41.7 | **32.0** |\n| | TV Loss ↓ | 19.4 | 18.1 | **9.7** |\n| | VTCM ↑ | 2.4 | 2.3 | **3.7** |\n\nIn multi-object cases, our method creates complementary forces guiding content away from specified regions.\n\n[W1,Q2] The reviewer may misunderstood our approach. Our warping force and other forces specifically address object size reduction and fragmentation caused by attention reduction, and **this fragmentation equally occurs in both convex and non-convex shapes**. We mentioned non-convex shapes in limitations only to consider more fine-grained region requirements from users, such as spiral-shaped regions.\nFor non-convex shapes, our implementation approximates them using their convex hull. While a limitation, user studies show most text and icon placement scenarios involve convex regions, making this practical for common use cases.\n\n[Q4] We're exploring image reward metrics for optimal parameters across different scene complexities, including MLLMs as judges to automatically adjust parameters.\n\n### [W3] Applicability to Other T2I Models\n\nOur method is extensible to other text-to-image models. We have successfully applied it to multiple diffusion models (see line330 table 2 in the paper).\n\nThe implementation is straightforward. After defining the attention processor class, integration requires only a few lines:\n```python\nfor name in pipe.unet.attn_processors.keys():\n    if \"attn2\" in name:\n        attn_procs[name] = my_attn_processor(guidance_func, region, len(tokens), place_in_unet=name)\n    else:\n        attn_procs[name] = pipe.unet.attn_processors[name]\npipe.unet.set_attn_processor(attn_procs)\n```\n\nWhen combined with the finetuned diffuison model, we see improvements:\n\n| Dataset | Metrics | Desigen-Trained+Attention reduction | Desigen-Trained+Ours | Improvement |\n|---------|---------|-------------------------------------|----------------------|-------------|\n| Desigen benchmark | Saliency IOU ↓ | 41.66 | 38.48 | 7.60% |\n| | TV Loss ↓ | 18.06 | 12.19 | 32.50% |\n| | VTCM ↑ | 2.30 | 2.85 | 23.90% |\n| | CLIP Score ↑ | 28.99 | 26.41 | -8.90% |\n\n### [Com 3] Computational Complexity and Runtime Analysis\n\nWe will add runtime analysis to the appendix. Our method maintains computational complexity comparable to standard diffusion models with minimal overhead from force computations. Our implementation is efficient, requiring less than 15GB VRAM (compatible with consumer GPUs like RTX 3090) and averaging approximately 30 seconds per image generation, as it requires only a single inference pass rather than multiple candidates.\n\n### [Com 1] Algorithm Pseudocode\n\nWe will add simplified pseudocode to method section in the revised paper to clarify the implementation process. Here is the algorithm that summarizes our approach:\n\n```\nAlgorithm: Text-Friendly Background Adaptation\nInput: Text prompt P, Target region R for text placement\nOutput: Text-friendly image I with clear space in region R\n\n1. Initialize diffusion model with prompt P\n2. For each timestep t:\n   a. Extract cross-attention maps A_k^l for tokens k, layers l\n   b. For each token k, layer l:\n      i. Detect conflicts D(k, R, A_k^l)\n      ii. For conflicting tokens:\n         - Compute attention centroid\n         - Calculate repulsive force F_rep and margin force F_m\n         - Compute total displacement Δpos = F_rep + F_m\n         - Apply warping through affine transformation\n      iii. Apply constraint: A_k,new^l = A_k^l ⊙ (1-R)\n   c. Apply modified maps for guidance\n3. Return text-friendly image I\n```\n\n### Application Scenarios\nWe have already supplemented our evaluation with the Desigen dataset(see table 2), which represents poster and advertisement design scenarios. Mobile interface design represents an interesting future direction that we plan to explore.\n\nWe appreciate your consideration and helpful suggestions.", "claims_and_evidence": "The paper's claims are well-supported by experimental evidence:\n\n1. The authors propose new evaluation metrics (Saliency IOU, TV Loss, CLIP Score Loss, and VTCM) that comprehensively assess the quality and text-friendliness of generated images.\n2. Quantitative comparisons with existing methods (Dall-E 3, AnyText, Desigen, and Stable Diffusion 1.5) show that TextCenGen outperforms these methods across all datasets.\n3. A user study with 114 participants validates the effectiveness and user perception of the method.\n4. Detailed ablation studies demonstrate the importance of force-directed cross-attention guidance and spatial excluding cross-attention constraint.\n5. Rich visual results intuitively showcase the method's effectiveness.", "methods_and_evaluation_criteria": "The proposed methods are well-suited for addressing the text-friendly image generation problem:\n\n1. Force-Directed Cross-Attention Guidance: Cleverly applies physical mechanics concepts to attention map editing, enabling smooth relocation of conflicting objects.\n2. Spatial Excluding Cross-Attention Constraint: Ensures background smoothness in text regions, suitable for text overlay.\n3. The training-free approach makes the method more accessible and applicable to various pre-trained models.\n\nThe evaluation criteria are comprehensive and reasonable:\n1. Testing on three datasets (P2P Template, DiffusionDB, and Synthesized Prompts) covers diverse scenarios.\n2. Evaluation metrics balance considerations of image quality, semantic fidelity, and text region adaptability.\n3. MLLM-judged ELO ranking provides a more comprehensive assessment of design appeal.", "theoretical_claims": "The theoretical contributions focus on adapting force-directed graph algorithms, borrowing concepts from physics to apply to attention map editing:\n\n1. Repulsive Force: Ensures each element is placed separately\n2. Margin Force: Prevents vertices from being expelled from visual boundaries\n3. Warping Force: Maintains object visibility within the canvas through affine transformations\n\nThese theoretical claims are clearly understood, logically sound, and well-formulated in the paper.", "experimental_designs_or_analyses": "The experimental design is comprehensive and reasonable:\n\n1. Benchmark comparison: Thorough comparison with existing methods across multiple datasets\n2. Ablation studies: Evaluation of the contribution of each component\n3. User study: Validation of the method's practicality and user perception\n\nThe experimental analysis is also thorough:\n1. Combines quantitative metrics and qualitative results\n2. Compares performance across different Stable Diffusion versions\n3. Analyzes the influence of the force balance constant\n4. Provides generalization analysis across different types of prompts and region shapes", "relation_to_broader_scientific_literature": "This paper relates to several research directions:\n\n1. Text layout generation: Extends traditional layout design methods, shifting from static image layout to dynamic generation of text-suitable backgrounds.\n2. Text-to-image generation: Leverages recent advances in diffusion models but adds consideration for text placement.\n3. Attention-guided image editing: Adopts training-free methods to manipulate attention maps, avoiding expensive retraining.\n\nThe paper skillfully combines these areas to propose a novel text-friendly image generation method.", "essential_references_not_discussed": "The paper already includes the major literature in related fields", "other_strengths_and_weaknesses": "Strengths:\n1. Addresses an important and practical problem: generating images with space reserved for text\n2. The training-free method design is clever and easy to integrate into existing models\n3. Comprehensive evaluation including multiple datasets, metrics, and user studies\n4. The proposed force-directed cross-attention guidance concept is innovative, providing a new perspective for attention map editing\n\nWeaknesses:\n1. Limited handling of non-convex shapes, which may lead to object size reduction or fragmentation\n2. May produce unexpected changes or unspecified objects in certain cases\n3. Evaluation primarily focuses on SD 1.5, requiring more validation for broader applicability to other T2I models", "other_comments_or_suggestions": "1. Consider providing a simplified algorithm pseudocode to make the entire process easier for readers to understand\n2. Further testing in practical application scenarios (such as poster design, mobile interface design) would strengthen the work\n3. Provide more analysis on computational complexity and runtime", "questions_for_authors": "1. How robust is the method when handling complex scenes (such as multiple objects overlapping)? Are there coping strategies?\n2. What improvements are being considered for handling non-convex shape objects?\n3. How could the method be extended to support simultaneous optimization of multiple text regions?\n4. Is adaptive setting of force balance parameters feasible? How could they be automatically adjusted based on scene complexity?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4qlt8ggVL5", "reviewer": "Reviewer_XmgK", "summary": "This paper mainly targets text-to-image generation. The authors focus on an interesting problem: after generating images, one would potentially want to insert specific visual texts into the images, and it would be better if the area to be inserted has clean background rather than being occupied with other objects. The authors show that current T2I models can hardly fulfill this goal without specific design. To this end, they propose an attention-guided text-centric background adaptation to modify the attention maps according to several criteria. The authors conduct multiple experiments to show the effectiveness.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable review. We address each point below.\n\n### [W1]Task Importance Placement in Introduction\n\nWe'll add to the introduction: \"As shown in Fig. 6, creating text-friendly images is essential for graphic design applications (validated by our 114-participant user study).\"\n\n### [W2]Correlation Between Attention Maps and Target Objects\n\nCorrelation is an important assumaption for our method, which relies on attention manipulation. Strengthening the correlation between semantic content and attention maps is a critical research direction in this field [1,2]. **Any training-free methods that improve this correlation could be readily integrated as upstream components to our approach.** This represents a promising direction for future research that would complement rather than replace our contribution. Notably, Our results still demonstrate that attention manipulation remains an effective mechanism for our task even with current models with semantic loss (with clip score). \n\n### [W3]Alternative Approaches Comparison\n\nTo clarify differences between our task and existing approaches, we present the following comparison:\n\n| Task | Training-Free | Annotation-Free | Layout Specification | Required Anchors |\n|------|---------------|-----------------|----------------------|------------------|\n| Layout-to-Image [3] | ❌ | ❌ | Object | >5 |\n| Text-Friendly Image Gen (Ours) | ✓ | ✓ | Space Region | 1-2 |\n| Visual Text Generation | ✓ | ✓ | Text | 1-2 |\n\nUnlike layout-to-image tasks requiring training and intensive annotation, our method only needs space region specification, which is crucial for dynamic applications like mobile wallpapers (Figure 1) or e-commerce posters.\n\nLayout-based methods primarily focus on object arrangement rather than creating text-friendly backgrounds. Their objective is fundamentally different:\n- Layout-grounded T2I: Positions objects according to a layout\n- Our approach: Creates backgrounds that harmonize with planned text regions\n\nDirect TV restriction in target regions would yield effects **similar to attention reduction**, which we have already validated in our ablation studies. As shown in Table 1, simply applying spatial constraints without force-directed guidance (w/o FDG) leads to higher CLIPS Loss (2.2 vs 0.32), indicating reduced semantic fidelity to the original prompt. This demonstrates that naive approaches like direct TV restriction fail to balance text-friendliness with instruction following.\n\n| Method | CLIPS Loss ↓ | TV Loss ↓ | Saliency IOU ↓ | VTCM ↑ |\n|--------|-------------|-----------|----------------|--------|\n| w/o FDG | 2.2 | 12.44 | 28.56 | 3.03 |\n| Ours | **0.32** | **8.81** | **22.86** | **4.4** |\n\nOur force-directed approach enables precise object placement control while maintaining semantic coherence, as evidenced by lower CLIPS Loss and superior text-compatibility metrics.\n\n\n**Both alternatives would ultimately need similar attention control mechanisms to achieve our goal of text-friendly background generation that respects user-specified regions**.\n\n### [W4]Testing with More Advanced Models\n\nWe have deployed our method on SD1.5, SD2.0, and SDXL to demonstrate the broad applicability of our method across different model architectures. Our approach can be adapted to MMDiT-based models like FLUX through methods similar to those demonstrated in recent work [4] similar to this [code](https://github1s.com/krafton-ai/Rare-to-Frequent/blob/main/R2Fplus_Diffusion_sd3.py#L986). The key difference in MMDiT models is the attention structure, which can be described as:\n```\nMMDiT Attention =  | Text-Text (TT) | Text-Image (TI) |\n                   | Image-Text (IT) | Image-Image (II)| ,\n```\n\nIn UNet models, we modify cross-attention (equivalent to TI block). For MMDiT models, one possible solution is to apply our force-directed approach to the Image-Text (IT) block (such as transpose the IT block), and maintain consistency in the Text-Image (TI) block.\n\nThis adaptation is technically straightforward based on existing implementations like Rare-to-Frequent. While implementation details differ, the fundamental principle of manipulating attention to create text-friendly backgrounds remains valid across architectures.\n\nWe appreciate your thoughtful suggestions and will address them in our revised manuscript.\n\n[1] Yang et al. \"Dynamic prompt learning: Addressing cross-attention leakage for text-based image editing,\" Neurips 2023\n\n[2] Liu B, Wang C, Cao T, et al. \"Towards understanding cross and self-attention in stable diffusion for text-guided image editing,\" CVPR 2024\n\n[3] Zheng et al., \"LayoutDiffusion: Controllable diffusion model for layout-to-image generation,\" CVPR 2023\n\n[4] Park D, Kim S, Moon T, et al. \"Rare-to-Frequent: Unlocking Compositional Generation Power of Diffusion Models on Rare Concepts with LLM Guidance,\" ICLR 2025 Spotlight", "claims_and_evidence": "Yes, the authors have pointed out an interesting and important problem in T2I generation. The experiments can validate this.", "methods_and_evaluation_criteria": "Yes, the relationship between attention map and generated objects has been studied in previous works. Besides, the experiments designed by the authors are reasonable enough to show the effectiveness of the proposed method.", "theoretical_claims": "No theoretical claims are presented in this paper.", "experimental_designs_or_analyses": "Yes. The main experiment includes comparison with several T2I models. Both qualitative and quantitative results illustrate the superiority of the proposed method. The ablation study can show the role of each term in the proposed guidance.", "relation_to_broader_scientific_literature": "The proposed method can better enhance the application of T2I models in design of wall papers, posters, etc. Besides, the method can be potentially extended to T2V/T-2-3D models.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths:\n1. The whole paper is well written and easy to understand.\n\nWeaknesses:\n1. I wonder if it would be better to place the task importance in Fig.6 in Sec.1?\n2. It has been shown that actually models such as SD1.5 and SDXL do not enjoy strong correlation between the attention map and the target objects. I wonder if this could affect the proposed method.\n3. While the proposed method is solid, it is to some extent complicated. Some simpler methods, for example, generating staggered layouts and using layout-grounded T2I to generate images, or directly restricting the TV in the target region, may be a better choice.\n4. It would be better to present results with more advanced diffusion models such as SD3 and FLUX.", "other_comments_or_suggestions": "Please refer to the weaknesses.", "questions_for_authors": "Please refer to the weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers acknowledge the thorough evaluation presented in the paper and the promising performance of the proposed method, supported by subjective, objective, and qualitative results. While there were concerns about the lack of comparisons with highly relevant baselines, evaluation settings that may not fully align with practical applications, and limited evaluation on more recent models, these issues were partially addressed in the rebuttal. Despite these concerns, the reviewers overall gave positive final ratings. The Area Chair agrees with the reviewers and finds the paper to be practical and empirically promising. The paper is considered to be above the acceptance threshold, provided the authors appropriately incorporate the content and promised changes outlined in the rebuttal into the final version.
Decision: Accept (poster)