T$^3$-S2S: Training-free Triplet Tuning for Sketch to Scene Synthesis in Controllable Concept Art Generation
Authors: Zhenhong Sun
Yifu Wang
Yonhon Ng
Yongzhi Xu
Daoyi Dong
Hongdong Li
Pan Ji
Venue: Accepted by TMLR ()

Abstract: 2D concept art generation for 3D scenes is a crucial yet challenging task in computer graphics, as creating natural intuitive environments still demands extensive manual effort in concept design. While generative AI has simplified 2D concept design via text-to-image synthesis, it struggles with complex multi-instance scenes and offers limited support for structured terrain layout. In this paper, we propose a Training-free Triplet Tuning for Sketch-to-Scene (T3-S2S) generation after reviewing the entire cross-attention mechanism. This scheme revitalizes the ControlNet model for detailed multi-instance generation via three key modules: Prompt Balance ensures keyword representation and minimizes the risk of missing critical instances; Characteristic Priority emphasizes sketch-based features by highlighting TopK indices in feature channels; and Dense Tuning refines contour details within instance-related regions of the attention map. Leveraging the controllability of T3-S2S, we also introduce a feature-sharing strategy with dual prompt sets to generate layer-aware isometric and terrain-view representations for the terrain layout. Experiments show that our sketch-to-scene workflow consistently produces multi-instance 2D scenes with details aligned with input prompts.

Published in Transactions on Machine Learning Research (02/2026)
T Training-free Triplet Tuning for Sketch to Scene Synthesis in Controllable Concept Art Generation
Zhenhong Sun
zhenhong. wawpaahongsuuueeeeeen sun@anu.. edu, 1 Ifwann927 @gmaail com yonhon. ng @hotmaill com 25 1 05844 3,388 32mall unsw. edu daoyi. dong edu on hongdong. Ji@ana edu peterji530 @gmail. com
Yifu Wang2t
Yonhon Ng
Yongzhi Xu
Daoyi Dong1
Hongdong Li!
Pan JI
Australian National University
Vertea Lab
Independent Researcher
University of Technology Sydney
Reviewed on OpenReview: https: // optrr/veewweeee..om net/ forum? id= Lym2BgKQ8F
Abstract
2D concept art generation for 3D scenes is a crucial yet challenging task in computer graph- ies, as creating natural intuitive environments still demands extensive manual effort in concept. design. While generative AI has simplified 2D concept design via texl-to-image synthesis, it struggles with complex multi-instance scenes and offers limited support for structured terrain layout. In this paper, we propose a Training-free Triplct Tuning for Skeuch-to-Scene (TS-S2S) generation after reviewing the entire cross-allention mechanism. This scheme revitalizes the ControlNet model for detailed multi innttiiinttacee generation via three key modules: Prompt Balance ensures keyword representation and minimizes the risk of missing critical instances; Characteristic Priority emphasizes
fea- tures by highlighting TopK indices in feature channels; and Dense Tuning refines contour details within instance-related regions of the attention map. Leveraging the controllability of S22, we also introduce a feature-sharing strategy with dual prompt sets to gen- erate layer-aware isometric and terrain-view representations for the terrain layout. Ex- periments show that our sketch-to-scene workflow consistently produces multi-instance 2D scenes with details aligned with input prompts. Code is available at the official repository https: /gtgguthhubboommm. Tencent/ /Triplet Tuning, with a maintained mirror for future updates https: http://ggtthbbooo////neennnnggg AL-LAB/ //triplet. tuning
1
Introduction
Scene generation plays a significant role in visual content creation across various domains, including video gaming, animation, filmmaking, and virtual/augmentt reality. Traditional methods heavily rely on manual efforts, which require designers transform initial sketches into detailed multi-instance scene concept art through numerous iterations. Recently, technological innovations such Stable Diffusion (Rombach et all,
2022
Podell et al. 2023) equipped with ControlNet (Zhang et al. 2023b) and integrated with advanced texi- to- to-image technologies (Kim et al. 2023), have streamlined this process. These advancements have notably decreased the workload for designers by automating the conversion of simple sketches into complex scenes. While these technologies perform well with common scenes involving typical instances, they struggle with
"Work conducted with Yifu Wang, Yonhon Ng. Yongzhi Xu, and Pan Ji during the author's internship at Tencent 2024. i "Coresppnndecee Yifu Wang <I

---

Published in Transactions on Machine Learning Research (02/2026)
Prompts
Sketch Images
Synthesized Images by SDXL ControlNet
(0)
A painting couple
holding yellow
umbrella in a on rainy night. The with white dress and the man with blue suit.
Isometric view of game seene. forest with few trees, a river, bridge, bridge, post station,
Figure The SDXL-base model (Podell et al. 2023) and ControlNet model Xinsir 2023 perform well with common instances like humans, but they struggle with complex multi-instance scenes involving small instances and fail to accurately follow users' prompt.
generating complex multi-instance scenes and often miss fine details. For the terrain layout, designers still manually interpret concept art and build terrain by dragging grids with preset tools.
Alternatively, sketch-based synthesis can adopt multi-instance strategies by incorporating instance layouts via bounding boxes to guide the generation of multiple clements. While effective, most existing meth-
ods (Yang et al. 2023 Li et al. 2023: Liu al. 2023 Sun et al. 2024 Wang et al. 2024b Zhou et al. 2024
are training-based and require adaptation for sketches, relying on large datasets often restricted by copyright
in gaming, animation, and film. In contrast, training-free approaches (Xie et al.
2023 Chen et al.) 2024 Feng
et al. 2022 Kim et al. 2023) leverage attention maps to exploit inherent model capabilities, offering flexible adaptability and low cost. new lasks. Building on this, we retain rolt's sketch-following ability with a training- free tuning mechanism requiring no additional data. Through a theoretical cross-attention analysis, find that imbalanced prompt energy and value non-prominence undermine the competitivenoss of instances and increase coupling among similar ones, leading to deviation from intended prompts.
In this paper, we introduce Training-free Triplet Tuning for Sketch-to-Scene (T*-S28) generation via three modules. Prompt balance adjusts instanco-specific keyword energy to ensure all instances remain com- petitive. Characteristics priority amplifies instance-specific traits using a TopK selection from value matrices enhance channel prominence in the feature map. Dense tuning adapted from (Kim et al. 2023) strength- ens instance-related contour details in the attention map of the ControlNet branch. Together, these three form unified triplet strategy that generates detailed, multi-instance 2D scenes that closely align with input prompts and sketches. Based on the controllability, we propose twin structural concept framework that uses a dual-branch prompt synthesis with a mask-guided feature-sharing mechanism to generate layer-aware isometric and lerrain-view representations for reconstructing the terrain layout. Experimental evaluations indicate that our T3-$2S approach boosts the performance existing text-to-image models, consistently producing detailed, multi-instance scenes that closely align with the input sketches and input prompts.
The key contributions of our work summarized as follows:
We theoretically investigate the underlying cross-attention mechanisms and identify the imbalance of prompt energy and value non-prominence, leading to deviation from intended prompts.
The triplct tuning advances controllable concept art generation by balancing token competition, enriching attention expression, and accentuating each instance's characteristics.
Our S22 workflow consistently generates detailed multi-instance 2D images and terrain layout aligned with input prompts.
2

---

Published in Transactions on Machine Learning Research (02/2026)
2
Related Works
Diffusion Techniques. Recently, diffusion models (Ho et al. 2020 Rombach et al., 2022 Podell et al. 2023 Saharia et al. 2022) have marked a major breakthrough, improving the fidelity and realism in text- to-image generation, which shows infinite potential for concept art generation. The emergence of diffusion models has also advanced the development of 3D content generation tools. However, creating high-fidelity 3D scenes from images remains complex task due the diversity and intricacy of object shapes and appcarances.
Sketch-to-image Synthesis. While text-to-image models can generate high-fidelity, realistic images, they struggle to accurately convey complex layouts with lext prompts alone. In the field of diflusion-based generation, notable works include ControlNet (Zhang et al. 2023b). Make-a-scene (Gafni et al. 2022). and T2I Adapter (Mou et al. 2023) handle various additional visual conditions, including sketches, while mothods like Dense Dillusion (Kim et al. 2023), SpaText (Avrahami et al., 2023) and MultiDiffusion (Bar-Tal et al. inputs. In particular, Dense Diffusion is training-free free approach that adjusts the attention map by amplifying sketch-relevant tokens and downplaying less important ones, allowing the model to better distinguish between instances. ControlNet is powerful solution for sketch- to-scene generation, recognized for its exceptional ability to accurately follow conditions. However, these models often struggle with complex multi-instance scene generations, particularly when handling unusual or unique instances, and frequently overlook smaller instances. Recently, Xu et al. (2024) proposed an efficient pipeline for automatically generating interactive 3D game scenes from users' natural input sketches using the SDXL. and ControlNet models. However, the approach is also limited by the diversity and multi-instance representation in the intermediate 2D isometric image generation.
2023) focus specifically
Multi-instance Synthesis. Multi-instance synthesis is closely related to sketch-to-scene generation due to its controllable layout. Training-free modulations (Xie et al.
ControlNet+SDXL
Dense Tuning
Prompt:
2023
Chen al. 2024 Lian et al.
2023
Feng et al. 2022
Isometric view game scene, a plain, walk path, a river, high mountain, houses
Seed I
and training- baseddg- based fine-tuning
methods
YYagg el al. Sun et al. 2024
2023
Li et al.] 2023 Liu et al.
2023
2024 tackle the chal- lenge of diflusion models accurately representing mul- tiple instances with bounding boxes. For example, Li let al. (2023) (GLIGEN) used bounding box coordinates as grounding tokens, integrating them into a gated self- attention mechanism to improve positioning accuracy, while Liu et al. (2023 employed latent object detection model to separate objects, masking conflicting prompts and enhancing relevant ones. Despite existing methods of generating images with correct positions, these box-based approaches struggle with simple sketch inputs and fail strictly follow the designer's sketch. Our work lever- ages ControlNet's sketch-following capabilities and inves- tigates the challenges of synthesizing multiple instances. We aim to design training-fres free tuning mechanism to hance modeling within cross-attention, addressing these challenges effectively.
Wang et al. 2024b Zhou et al.
Scod
Scod
Sketch Image
Figure 2: The SDXL-base (Podell et al. 2023 and ControlNet models (Xinsir 2023) struggle with complex multi-instance scenes based on sketch images and text prompt, even with im- proved dense tuning (Kim et al. 2023).
3 Controllability Analysis of Cross-attention Mechanism
Problem Statement. ControlNet [Zang et al. 2023a is adopted alongside SDXL (Podell et al. 2023 as our bascline for concept art generation due to its controllability in sketch-to-image generation. Given a textual prompt C. = {c') (I is words number) and sketch image C, E RRx the system generates images via cross-attention mechanism in the UNet. (Rombach et al. 2022), aligning spatial features with text
3

---

Published in Transactions on Machine Learning Research (02/2020)
End
End
agam
pidwwllk
Energy (L2 Norm)
40-
Energy (L2 Norm)
Cosine Similarity
0.3
mountain
pet
35-
galne
houses
0.6
30-
plan
35
eceine river
vyww
30-
0.4
houses
25
Token Index
Taken Index
Token Index
(a) windle-word prompts
(b) Multi-word prompts
(c) Cosine similarity.
Figure 3: Embedding energy comparison between a global prompt ("Isometric view of game scene, a plain, walk path, a river, a high mountain, houses.)) and single-word prompts (each keyword separated and embed- ded individually, higher than that in the group). The energy imbalance can lead to attention competition, and low-energy small instances ("path" and "houses") easily forgotten. (c) Cosine similarity bet wetw embeddings of (a) and (b)
embeddings S Rexd (encoded from Cg):
F = Am Vm = softmax Qm K Te/ dm
Vm
where m is layer number, the number of text tokens, and d the token embedding dimension. Qm E R comes from fore-features, and Km, Vm E Radd are derived from text embeddings S by linear projections, where bm is the flattened spatial dimension, and dm is the embedding dimension. As shown in Figure 2 the system struggles with complex multi-instance natural scenes guided by sketch images and text prompts, often failing to render all intended objects.
Dense Diffusion (Kim al. 2023) can effectively highlight different instances' attention values based on the sketches in the attention map Am Rhm.x x but directly applying its strategy to ControlNet leads to per- formance degradation. To adapt it, we make improved Dense Tuning (DT) (Details in Appendix B): 1) Only add the dense tuning at the "down_bloo" 2" and expand function, not use the suppress function in the self-attention. Despite this, the image quality improves, but instances with small areas, such as "path' and "houses', are easily neglected in the final image, despite having strong responses in feature maps. This highlights that beyond tuning attention maps alone, there remains room to explore controllability within the broader cross-attention process. In particular, the roles of K,, in shaping attention distributions and Vn in contributing the final feature outputs have received limited attention.
O' of ControlNet; 2) Only use the
Imbalance of Prompt Energy. In practice, increasing prompt weights (e.g., "(houses: in WebUT) can enhance the visibility of specific instances multi-instance gencration. To understand the underlying mechanism, we refer to Henry el al. (2020) to decompose each loken embedding from S [st s. E Baxd as:
E 8 where E ||s' [|']|=11
where, Es represents the prompt energy (L2 norm), and s denotes the normalized embedding. Since both keys and values in cross-attention are linear projections of s", their magnitudes are bounded:
| = ||s W K < |W
||v*|| = | & F W
Thus, tokens with lower energy yield weaker keys kf and values vi, diminishing their impact on attention and feature aggregation.
Considering the prompts in Fig. 2 we analyze the embedding energy within a global prompt and each keyword embedding separately, as shown in Fig. 3 Global encoding obviously lowers the energy of the individual tokens, especially the instances like "path" and "houses", aligns with the observed instance omission in The final attention value of the valid token be represented
Fig. 2
Eã, where denotes the
4

---

Published in Transactions on Machine Learning Research (02/2020)
TopK-1
TopK-2
TopK-3
60
Attentio
X
(E
Rive-15 m)
Mountah-PP/C m)
Howes m)
Q
River C
Fature
(a, x xd.
Ch.0 d.)
Cb-l de)
Ch.2 (,,
Ch-3 (8,4)
(h
Figure 4: Interaction ween attention maps and value matrices with prompts from Fig. 2 using dense tuning. (a) Attention maps highlight strong sketch relevance. (b/c) Five-channel value-feature pairs reveal the importance of extrema. Despile feature enhancement improving instance chances, forgetting still occurs as extrema are not prominent. Statistics are shown in Fig. 10 and Appendix C.
Figure Generations by ampli- fying the TopK extrema twice in the value matrices based on the pipeline in Fig. 4 where most in- stances appear, but uniform ampli- fication also introduces noise.
mcan attention weight, empirically observed as approximately 0.43 across 100 prompt scts. After applying exponentiation, the ratio between two attention values becomes:
Ratio e F² R2 R'a =e
Due to exponential weighting, energy differences of (1,3,5) result in attention Ratio disparity of {1.54, 3.63, 8.58) before the softmax average, and higher-energy tokens easily win the attention competi- tion. Even if a low-energy token receives high attention via tuning, its contribution remains limited due to the low magnilude of its value vector V; which weakens the aggregated feature in the product A, Vm- This energy imbalance, which allows dominant tokens to overshadow weaker ones and increases the risk of instance omission in multi-instance synthesis, underscores the importance of balancing and scaling prompt energy an interesting perspective improve multi-instance scene generation.
Non Non-prominence of Value Matrices. As core part of cross-attention, the interaction between attention maps and value matrices shapes each channel's characterist to instance-level patterns such as geometry and attributes. To maximize expressiveness, each channel should focus on distinct semantics, similar to SENet's calibration (uu et al. 2018). The aggregated fcature at channel j is:
pj Am v = {aaz a a.
[u o
As shown Fig. aa), dense tuning sharpens attention, allowing each spatial position & to attend a single instance via high az.. However, The non-prominence between value values can cause responses to multiple instances, leading entangled features. Specifically, the extreme values determine which instances are activated spatially, consistent with the observations in Fig. bb) and (c).
To probe this, we double the TopK values in each channel of the value matrices, as shown in Figure 5 As increases, the model initially synthetizes instances (e.g., at K = 2), but it also introduces excessive noise (c.g., over-detailed houses). This demonstrates that stronger value prominence improves token com- petitiveness, but requires trade-off between instance completeness and visual clarity via controlled value amplification.
5

---

Published in Transactions on Machine Learning Research (02/2026)
(b)
Prompt Balance
(a) Structure
ControlNet
Characteristics Prominence
Global Prompt: Isometric view of
Self
Cross Attention
game scene, plain, walk path,
Attention
fiver high mountain, houses
Sketch
Unet
Self
Cross Attention
Words Prompt: plain path river mountain houses
Decoder
Latent
Down
Mid
Encoding
Energy
(c)
Characteristics
Attention Maps
Start
End
Prominence
G
BRin
houses
Replacing
Feature
K
Latent
Balancing
Ch3
Feature
Enhanced
Enhanced
Feature
Balanced Text Embeddings
Balanced Embeddings
Select Top indices with related sketches
Figure Overview of the proposed training-free triplet tuning strategy in the frozen pre-trained latent diffusion model. (a) The orange parts indicate the proposed module plugged into the ControlNet and U- Net framework. (b) The left part shows the energy tuning of the prompt balance. (c) The bottom part
indicates the training- free tuning of the characteristics priority. T ensures the final generation responds effectively to both text and sketch inputs.
4 Diffusion for Controllable Concept Art Generation
8 we propose training-free triplet tuning To address the cross-attention challenges discussed in Section strategy for controllable concept art generation, as illustrated in Fig. 6 This strategy consists of three modules: Prompt Balance (PB), Characteristics Priority (CP), and Dense Tuning (DT))
(1) Prompt Balance: This module identifies instance keywords within global text prompts, replaces their embeddings with corresponding single-word embeddings, and adjusts the energy of these keyword embod- dings to maintain balance. By balancing the energy of the keyword embeddings, the method enhances the representation of instances within key and value matrices. This process improves the competitiveness of instance tokens among all tokens, ensures consistency across instance tokens, and reduces the likelihood of overlooking rare or unusual instances.
This module selects instance-related lokens and their sketches by identifying the TopK values for each channel in the value matrices, creating an instance-specifit mask. The mask is then used scale up the feature map for the corresponding channel. This approach enhances the distinction of instances within the multi-channel feature space without additional parameters, ensuring that instances' characteristics are more prominently emphasized.
(2) Characteristics Priority:
(3) Dense Tuning: While prompt balance increases the strength of the embedding matrices related to instances, enhancing their competitiveness in the attention map, the overall strength of the attention map remains suboptimal. Meanwhile, given that more contour information resides in the ControlNet branch, we employ dense modulation directly within this branch to augment the attention map for better modulation. Since attention manipulation has been extensively explored in prior works (Hertz al. 2022; Xie el al., 2023
Kim et al., 2023 Chen al. 2024), we only adopt a modified dense tuning scheme from Kim et al.
2023
detailed in Section
In the subsequent section, we will provide detailed explanation of two newly designed modules and their underlying rationales.
4.1 Prompt Balance
As discussed previously, the imbalance of prompt energy influences competition among key and value matrices related to instances, increasing the risk of missing instances. To mitigate this, we propose a plug-in prompt balance strategy for the text embeddings that enhances energy uniformity across keywords and scales up

---

Published in Transactions on Machine Learning Research (02/2026)
values, displayed in Figure (b). Specifically, use NLP network (e.g., the SpaCy library) to identify instance keywords from the global text prompts cg = {c') resulting in reorganized instance keyword prompts {c'}9, where q is the indices vector of keywords in cy. Then, we encode both the global text prompts and each instance keyword prompts separately into text embeddings S = {s)} E Raxd and {st IRIxdja by a text encoding network. Next, we replace the embedding of keywords in Sg with the single-word embedding of S to form a new combined embedding S, E R
S, (s, = if E q otherwise (s)))
Generally, the special "end of text" token (located fend) always has the maximum energy as shown in Fig. 3 which could be the upper bound for us to scale up the embeddings of the keywords S, that all keywords have balanced energy relative to the "end of text" token embedding, mathematically represented as:
{s {EE /E,) $9 where Ex l|s || and E = ||st
Finally, the balanced text embeddings, denoted as So, enhance the instance- maseddce-based token values in the key, value matrices, and attention map, improving their competitiveness and consistency for more concise and effective instance representation.
4.2 Characteristics Priority
While balanced text embeddings help equalize instance competition, value matrices still face low prominence. As discussed in Section 3 enhancing TopK values per channel improves prominence. To balance instance completeness and noise clarity, we propose a CP technique that applies localized TopK enhancement to specific sketch regions for features. Specifically, instead of directly enhancing the TopK values along the n dimension in the value matrix Vm Raxdm, we apply enhancement based on the indices of the TopK values on the feature map Fm E Xdm (before the residual adding). For each channel in the value matrix Vms find the indices of the TopK values across all valid tokens (between "start" and "end" tokens):
YK lyk TopK(abs(V,,,) (V. fend)); K) R Kxdm
where the number of top values considered. For jth channel f/, E R Em. we check whether each index yk belongs to the instance keyword vector q. it does, the index i corresponds to specific instance token E q. Then the sketch u _ of the instance at the current scale will be summed together to generate enhancement mask h for the jth channel:
h = Σ' u i E (yk and q}-
The whole mask matrices Hm (h Jmm E Bbbxdd Xdm are used to proportionally scale up the corresponding values the feature map F... by a factor B, obtaining the enhanced feature map F
F = Fm + + H F
where denotes element-wise multiplication. This enhancement emphasizes instance tokens in the multi- channel feature space, improving instance distinction. The CP technique strengthens the attention mecha- nism by amplifying instance-relevan regions in the feature map, even with small sketches.
4.3 Structural Concept Representations
Building the previous modules, the S2-22 can produce detailed, multi-instance scenes that align closely with input prompts and sketches. To further enhance scene structure, es- pecially in maintaining terrain consistency while separating foreground objects, we propose a twin structural concept representations framework. As depicted in Figure 7 it adopts a dual- branch prompt synthcsis with mask-guided featuro-sharing mechanism: one for the complete iso- metric image with full skelches and another for the emply terrain with non-foreground sketches.

---

Published in Transactions on Machine Learning Research (02/2026)
The feature maps for the complete isometric image and the empty terrain are denoted as (l) and Fiem re- spectively, at inference step t. The terrain branch updates with selective and progressive blending, defined as:
Isometric VIew game scenie
plain walk path,
F Fior = Fter + M M (() - 7() - 70 ( (7)
t
mountain, houses,
riveer, high
TP-S2S
Isometric Image
Feature Sharing
Terrain Image
where @ denotes element- wise multiplication, To the ini- tial scaling factor. M is binary mask derived from the sketch sub-prompts, with 1 indicating background re- gions and 0 indicating foreground regions (o.g., houses, bridges). The dynamii scaling 7() ensures that the ter- rain branch inherits global features from the isometric branch at the early stages, while allowing independent refinement later, with the mask M preserving foreground integrity. Together, mask-guided sharing and dynamic scaling enable harmonious terrain generation and effec-
plain. walk path. anvver mountain.
Isometric view of empty scene
Dual Concept Art Generation
Figure 7: Twin-structure framework with dual- branch prompt synthesis, where a mask-guided feature-sharing mechanism handles full-sketch isometric and non-foreground terrain images.
tive foroground- backgroouddd scparation.
5 Experiments
5.1 Implementation Details
Basclincs. Our baseline leverages the sketch-proccssing capabilities of the ControlNet model (Zhang et al.] 2023b Xinsir 2023) with the SDXL-buse model (Podell et al.] 2023), compared with two SDXL- base sketch-oriented approaches: the training-basee T2I Adapter (Mou et al. 2023) and the training- free Dense Diffusion (Kim et al.) 2023). We also further validate PB with Attend-and-Exee (Chefer et al. (Appendix G), and T TSS2 with T2I adapter (Appendix H).
2023
Setup. In the triplet tuning scheme, the prompt balance module is integrated into the text encoding process, while the characteristics priority module are incorporated across all cross-attention layers. The dense tuning module is specifically added to the "down blocks 2' layers and the "mid blocks 0' layers within the ControlNet branch. We set K 2, a 5, To = 1 and 8 1. During inference, we use the default Fuler Discrete Scheduler (Karras et al. 2022) with 32 steps and a guidance scale of 9 at resolution of 1024 x 1024. All experiments are conducted
single Nvidia Tesla V100 GPU.
Metrics. Given that our current approach involves sketch-based multi-instance scene generation, existing benchmarks should be adjusted for our evaluation, such as adding sketch inputs for T21-CompBench (Huang et al. 2023). Therefore, we design 50 complex sketch scenes, each with more than four sub- sub-prompts, encom- passing various terrains (plains, mountains, deserts, tundra, cities) and diverse instances (rivers, bridges, stones, castles). We utilize CLIP-Score (Hessel al.) 2021) for the global prompt and image, and evaluate the CLIP-Score for each background prompt and instance prompts by cropping the corresponding regions. Additionally, we conduct user study to assess different variants of our approach, using a 1-5 rating scale to evaluate image quality, placement, and prompt-image consistency. Details can be found in Appendix D&F.
5.2 Main Results
Qualitative Evaluation. Building upon the scene design, we show two representative and complex multi- instance scene scenarios, cach incorporating diverse array of elements to foster varied interactions. We evaluate several approaches, with visual comparisons displayed in Figure g Due to the specialized nature of this task, most existing solutions will overlook objects aligned with the prompts. When combined with the triplet tuning strategy, our T3-$2S method improves the generation performance of existing SDXL, models. For example, Figure g (a) showcases the enhanced detail in smaller instances such as "houses" and "path", and even less common elements like "mountains". Similarly, Figure S (b) illustrates the effective
8

---

Published in Transactions on Machine Learning Research (02/2020)
5DXL
DenseDiff 121 Adaptor
ControlNet
ControlNett Neeti sketches
(a) Prompt: Isometric view of game scene, a plain, walk path, a river, a high mountain, houses,
PB
SDXL
DenseDiff
T21 Adaptor
ControlNeet
T?-$25
Terrdin
PB-CP
T+-225
Prompt: Isonectric VieW game field with snow, ioed hills, winding road, red house.
Figure 8: Qualitative comparison with baseline methods. (a) T--S2S per- forms well for smaller instances like "houses" and "path" and unusual "mountain'. (b) T performs well with large number of small in- stances "trees". Note that the original Dense Diffusion (Kim et al.) 2023) based on SD V1.5 (Rombach et al. 2022),, has limited prompt response ca- pabilitics. For fair comparison, we apply it to the SDXL model.
Figure 9: Qualitative compar- ison of different inserted mod- ules based on the ControlNet and SDXL model. The quan- titative results are present in Table 2
Table Evaluation results on T2I- CompBench. # indicates the method evaluated the new benchmark, others from the T2i- Cenchench paper.
Table 2: Comparison of CLIP-Score across several variants based on SDXL model, evaluated on whole images, masked instance regions, and masked background regions. Includes user study ratings on scale of 5.
Model Globalt
Instances
Background
Usert
Color B B-VQAt
2D Spatial UniDett
Numeracy UniDet
Model
DenseDiff
0.3438
0.2459
0.2543
2.25
SDXL
0.5879
0.2133
0.4991
T2I-Adaptor
0.3422
0.2405
0.2523
2.17
ControlNetgt
0.5925
0.2686
0.4639
NontrolNet
0.3435
0.2427
0.2531
2.38
Df
0.6154
0.3419
0.5143
PB 0.3439
0.2472
0.2559
2.62
T3.S28 #
0.6872
0.4256
0.6257
DT
0.3430
0.2469
0.2547
Stable v3
0.8132
0.3200
0.6174
CP
0.3457
0.2503
0.2565
3.40
FLUUX.1
0.7407
0.2863
0.6185
-PR+CP
0.3466
0.2552
0.2571
3.55
T'S22
0.3491
0.2566
0.2581
3.79
generation of numerous small instances, like "trees'. By leveraging the triplet tuning strategy within the cross-altention mechanism, approach consistently generates detailed, multi-instance scenes that closely adhere to the original sketches and texts. Additional game and common scenes are provided in Fig. 15 and Appendix D&E.
Quantitative T2I-CompBench. We evaluate common scene generation on the large-scale T2I- CompBench (Huang et al. 2023) using Color (B-VQA), 2D Spatial (UniDet), and Numeracy (UniDet) metrics (Standard: 300 prompts/metric, 10 inages/prommp). As our method requires sketch inputs,
9

---

Published in Transactions on Machine Learning Research (02/2020)
0.15
scene
Max Indices
Max indlices
view
gomee
Indices
Max Indices
0.06
plain lk
mpuntainn
Percentage
0.10
Percentage
eath
river
houses
game
0.04
walk
river
mount
plain
path
0.02
0.00
0.00
21
12
Token Indices
Taken Indices
(a) The first and second extreme points.
(b) The third and fourth extreme points.
Figure 10: Histogram of the distribution of extrema in Value matrices. At the first and second extrema, the probability of later instance tokens is lower than that of earlier tokens, so increasing their values helps enhance their representations. However, at the third and fourth extremes, the probabilities tend to converge, implying that only a few tokens are key to defining features.
GPT-4V to generate square sketches (similar to bounding boxes) for all instances, with manually adjusted sizes, presented in Table [ Our method improves about 9.5% in the Color metric and 15.7% in the 2D Spatial metric over the ControlNet baselines. Especially for the Numeracy metric, we ensure that the 100 examples contain 1-3 small instance sketches with a minimal resolution of (ensuring pixel at the minimal feature level). Compared to the SDXL baseline, ControlNet performs 3.5% drop on the Numeracy metric leaturing small instances, while DenseDiffusion improves 1.4%. Our approach shows noticeable 16.2% gains over ControlNet for the generation of small instances.
Quantitative Customized Evaluation. We compare CLIP-Scores for global image, instances, and back- ground across different variants and the base ControlNet. A user study is also conducted with a 1-5 rating scale. As shown in Table 2 our approach demonstrates better performance on the 50 complex multi-instance scenes, with improved fidelity and precision in aligning with text prompts and sketch layouts. The PB module shows modest improvement, while the CP and DT modules provide comparable enhancements. Combining these components allows our TS-S2S approach to achieve a well-balanced outcome.
5.3 Ablation Study
Module Comparison. We perform an ablation study (Fig. 9) to evaluate the individual and combined effects of three modules, as well as the quantitative results in Table 2 (1) DT constrains instance regions within sketches; (2) PB improves small object visibility (c.g., "houses") but may introduce noise; (3) CP sharpens features and suppresses noise. PB+CP addresses most issues, and combining all three (T*-S2S) yields the best Aketchhprompp alignment between generated scene images and their corresponding sketches and texts, aligning with the quantitative results present in Table 2
5.4 Top K Anaylsis
In the module of characteristic prominence, two hyperparameters, K and 8, require meticulous tuning. K determines the indices of extreme values within the value matrices. For our analysis, we save these indices and construct histogram, depicted in Fig. [0] We observe that the probability of later instance tokens achieving the maximum and second maximum values is comparatively lower than that of earlier tokens. Thus, increasing the value of later instance tokens will be beneficial for their representations. However, at the third and fourth extremes, the probabilities lend to converge, indicating that not every token is essential for defining key characteristics. Increasing values for later instances at this point would introduce additional noisc. Therefore, sctting = advisable based on the observed trends. For B, which enhances the characteristics of instances within the feature matrix, an initial increase is beneficial. Nonetheless, there is critical threshold beyond which increases in 3 begin to disrupt the distribution within the value matrices.
10

---

Published in Transactions on Machine Learning Research (02/2026)
(a)
(b)
k=0
K=2
B-1
8=
k=1
k=2
B=1
B-1
k=2
K=2
B=1
B=1.5
K=2
B=
B=2
K+2
B=1
8=3
Figure 11: Visual comparison of two typer-paramcters K and B suggests that setting K 2 and is favorable choice. With 1, increasing K initially improves results but eventually adds noise. Fixing == 2, higher values yield stable and favorable outcomes. Another comparison presented in Figure 18
Hyper-parameter Comparison. To validate our hypothesis, study TopK distribution in Fig. 10 In Fig. [0] later tokens have lower chances of reaching top values at the first and second extrema, so increasing their values helps enhance their representations. But at the third and fourth extrema, probabilities even out, implying only a few tokens are key to defining features. Based on this observation, we further examine the effects varying K and A on generation quality Fig. [T When fixing 3 = 1, increasing K initially improves visual quality by strengthening
represetedd tokens, but further increasing K introduces noticeable noise, less relevant tokens also amplified. Conversely, when fixing K 2, larger & values lead to more stable and consistently favorable results, indicating that moderate amplification of a small number of key tokens is sufficient to achieve effective feature modulation without disrupting overall coherence. These results suggest a trade-off between expressiveness and stability, where overly large K increases noise, while appropriately scaled 3 preserves structural consistency. Consequently, we identify = 2 and = 1 as a robust and effective configuration across our experiments.
6 Conclusion
In conclusion, our study on the training-free triplet tuning for sketch-to-scene generation has enhanced the ability of text-to-image models to process complex, multi-instance scenes. By incorporating prompt balance, characteristics prominence, and dense tuning, have effectively addressed issues such imbalanced prompt energy and value homogencity, which previously resulted in the inadequate representation of unusual and small instances. Our experimental results confirmed that our approach not only preserves the fidelity of input sketches but also elevates the detail of the generated scenes. This advancement vital in fields like video reality, where precise and dynamic visual content creation is generation processes, our model offers promising
gaming, filmmaking, and crucial. Facilitating more efficient and less avenue for future developments in automated sketch-to-scene transformations.
11

---

Published in Transactions on Machine Learning Research (02/2026)
References
Omri Avrahami, Thomas Hayes, Oran Gafni, Sonal Gupta, Yaniv Taigman, Devi Parikh, Dani Lischinski, Ohad Friod, and Xi Yin. Spatext: Spatio-textual representation for controllable image generation. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recognilion, pp. 10370-138880) 18380, 2023.
Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel. Multidiifuusion Fusing diffusion paths for con- trolled image generation, 2023. URL htttss://axxiiii htttp:://aaxx...or///b///////// wttp.//wrrrrabb/22220111183
Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models, 2023.
Minghao Chen, Iro Laina, and Andrea Vedaldi. Training free layout control with cross-attention guidance. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 5333, 2024.
Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Reddy Akula, Pradyumna Narayana, Sugalo Basu, Xin Eric Wang, and William Yang Wang. Training-free structured diffusion guidance for compo- sitional text-to-image synthesis. In The Eleventh International Conference on Learning Representations, 2022.
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scene: Scene-based text-to-image generation with human priors. In European Conference on Computer Vision, pp. 89 106. Springer, 2022.
Richard Hartley and Andrew Zisserman. Multiple view geometry in computer vision. Cambridge university pross, 2003.
Alex Henry, Prudhvi Raj Dachapally, Shubham Pawar, and Yuxuan Chen. Query-key normalization for transformers. arXiv preprint arXiv:2010.00; 04245, 2020.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yacl Pritch, and Daniel Cohen-or. Prompt-to- prompt image editing with cross-altention control. In The Eleventh International Conference on Learning Representations; 2022.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.0871 2021.
Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840 6851, 2020.
Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. arXio preprint NNi:22311.04400 2023.
Jie Hu, Li Shen, and Gang Sun. Squeeve-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2018.
Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: A comprehensive bench- mark for open-world compositional text-to-image generation. arXiv preprint arXiv: 2307.06350, 2023.
hyperhuman. http:///lyoermmma.deemmoss
2024.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based 00364
generative models, 2022. URL
Yunji Kim, Jiyoung Loe, Jin-Hwa Kim, Jung- Woo Ha, and Jun-Yan Zhu. Dense text-to-image generation with attention modulation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7701-7711 2023.
12

---

Published in Transactions on Machine Learning Research (02/2026)
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv:2304.0260 2023.
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jac Lee. Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22511 222211, 2023.
Long Lian, Boyi Li, Adam Yala, and Trevor Darrell. ulm-grounded diffusion: Enhancing prompt under- standing tox--to- diffusion models with large language models. arXiv preprint arXiv:2305.188 13655,, 2023.
Tuping Liu, Zijian Zhang, Yi Ren, Rongjie Huang, Xiang Yin, and Thou Zhao. Detector guidance for multi-object text-to-image generation. arXiv preprint or arxiv:206.. 02236, 2023.
Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. T2i- adapter: Learning adapters to dig out more controllable ability for text-lo-image diffusion models. arxiv preprint XrX:::2302.0844 2023.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pallern recogriilion, pp. 10684-161000 10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorcalistic text- to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 00366499-6499- 2022.
Zhenhong Sun, Junyan Wang, Zhiyu Tan, Daoyi Dong, Hailan Ma, Hao Li, and Dong Gong, Eggen: Image generation with multi-entity prior learning through entity guidance. In ACM Multimedia, 2024.
Andrey Voynov, Qinghao Chu, Daniel Cohen-Or, and Kfir Aberman. p+: Extended textual conditioning in text-to-image generation. arXiv preprint arXiv:2303.0952. 2023.
Junyan Wang, Zhenhong Sun, Zhiyu Tan, Xuanbai Chen, Weihua Chen, Hao Li, Cheng Zhang, and Yang Song. Towards effoctive usage of human-centric priors in diffusion models for text-based human image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognétion (CVPR), pp. 8446-8455, 8455, June 2024a.
Xudong Wang, Trevor Darrell, Sai Saketh Rambhatla, Rohit Girdhar, and Ishan Misra. Instancediffusion: Instance-level control for image generation. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pp. 6232-6243 6242, 2024b.
Xinyue Wei, Kai Zhang, Sai Bi, Hao Tan, Fujun Luan, Valentin Deschaintre, Kalyan Sunkavalli, Hao Su, and Zexiang Xu. Meshlrm: Large reconstruction model for high-quality mesh. arXiv preprint 2XXi:2204.1238. 2024.
Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wentian Zhang, Yefeng Zheng, and Mike Zheng Shou. Boxdiff: Text-to-image synthesis with training-free free box-constrained diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 7452-7461, 2023.
Xinsir. Controlnet- union-sdxl-1.0. httpp:://huggggfacc../////////// Accessed: 2024-09-30.
wwweeeett--iin--nee sxxiionnsxx-11000 2023. 2023.
13

---

Published in Transactions on Machine Learning Research (02/2026)
Yongzhi Xu, Yonhon Ng, Yifu Wang, Inkyu Sa, Yunfei Duan, Yang Li, Pan Ji, and Hongdong Li. Sketch2scene: Automatic generation of interactive 3d game scenes from user's casual sketches, 2024.
URL http:///aaxiii.orraa//////////// 2000///00
Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In CVPR, 2024.
Zhengyuan Yang, Jianfeng Wang, Zhe Gan, Linjie Li, Kevin Lin, Chenfei Wu, Nan Duan, Zicheng Liu, Ce Liu, Michael Zeng, et al. Reco: Region-controlled text-to-image generation. In Proceedings of the IEEE/CVP Conference on Compuler Vision and Pallern Recognition, pp. 14246 14255, 2023.
Yugian Yuan, Wentong Li, Jian Liu, Dongqi Tang, Xinjie Luo, Chi Qin, Lei Zhang, and Jianke Zhu. Osprey: Pixel understanding with visual instruction tuning. arXiv preprint arXiv22312.10082 2023.
Lymin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models, 2023a.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the TEEE/CVF International Conference Computer Vision, pp. 3836-3847, 2023b.
Dewei Zhou, You Li, Fan Ma, Xiaoting Zhang, and Yang. Mige: Multi-instance generation controller for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6818 68288, 2024.
14

---

Published in Transactions on Machine Learning Research (02/2026)
A Discussion and Limitations
While our approach is innovative and enhances multi-instance scene generation, it also has some room to improve, primarily stemming from the inherent capabilities of the base model.
Texture Generation. Generating fine-grained textures and detailed instances remains challenging. This issue is partly due to the limited ability of CLIP-based representations to precisely encode complex descriptive attributes. Furthermore, the charactoristic prominence module prioritizes instance-level tokens, which can lead to reduced attention on adjective-level descriptors that are critical for texture and appearance modeling, For example, as illustrated in Fig. 17 in cases where multiple instances of the same object with different colors are specified, our method is generally able generate the desired targets. However, attribute confusion may still occur, leading to mismatches between instances and their corresponding color attributes.
Large Scene Challenge. Our method encounters difficulties in accurately generating expansive scenes (e-g: game maps exceeding 4096 4096 pixels). Processing such high resolutions directly incurs prohibitive computational costs, typically necessitating patch-based based processing or upsampling strategies. However, a more fundamental constraint lies in the model architecture itself. Taking SDXL as an example, the network downsamples 1024 x 1024 imput to minimum bottleneck feature size of 32 x 32. This imposes an intrinsic 32x scaling limit, meaning the minimum controllable granularity for object generation is restricted to this ratio. For example, in the first case of Figure 16 the control over minute objects degrades at the bottleneck layer. Consequently, even with increased output resolution, accurately resolving and controlling tiny instances or intricate interactions below this structural floor remains a significant challenge.
Concept Art Understanding. Additionally, the 3D scene shows that terrain and objects can be generated concept art, but improvements are needed in understanding concept art, particularly in segmentation and depth estimation, which may not be accurately represented in the synthesized data. One polential remedy is to concurrently generate multiple modalities at the same time, such as RGB, semantic, depth, material, and object footprint, and fuse these results.
Building on our current achievements, we plan further explore these areas future work to improve detailed multi-instance sketch-to-scene 3D generation.
B Details of Dense Tuning
2023) can effectively amplify instance-specific attention responses in slandard diffusion models. However, when adapting dense modulation strategies to ControlNet, it is crucial to account for the hicrarchical roles of different layors. We follow the layer-and-step-wise empirical observations (Voynov) et al. 2023 Wang et al. 2024a Sun et al. 2024] that peripheral (early) layers predominantly encode low- level geometric cues and boundary information, while more central layers are responsible for shaping object contours and structural layouts. This behavior is consistent with the hierarchical design of diffusion UNets and ControlNet, where early steps preserve fine spatial details and later steps increasingly capture semantic structure. As demonstrated in Table 2 DT alone yields marginal quantitative improvement, indicating that DT primarily acts as a modulation mechanism rather than the main source of performance gain, which is instead driven by the proposed PB and CT modules. Thus, to evaluate its application, we introduce adapted variants of DT across different layers within our proposed PB+CT framework.
Dense Diffusion (Kim et al.
presents qualitative comparison of Dense Tuming (DT) appliod at different depths of ControlNet. Applying DT at mid_ block_ already produces visually coherent and well- controlled results, while extending DT to down down_block 2 + mid_ block_ 0 further improves structural con- sistency. In contrast, including carlier layers such as down block_1 introduces overly strong contour and edge responses, resulting in visually intrusive ControlNet outlines that degrade image naturalness. This observation indicates that early ControlNet layers predominantly encode low-level geomet cues, where dense modulation tends to over-amplify sketch boundaries rather than semantic structure.
Qualitative analysis. Figure 12
Efficiency analysis. Table 3 reports GPU memory consumption and inference latency under different DT layer configurations. As DT is activated in additional layers, both memory usage and inference time increase
15

---

Published in Transactions on Machine Learning Research (02/2026)
Table 3: Cost benefit analysis of Dense Tuning (DT) applied to different layer combinations in ControlNet
and U-Net within the PB+CT framework.
CoutrolNet DT
U-Net DT
Memory (MB)
Inference Time (s)
Global
Instances! 1
Background"
MD
N/A N/A
18175
10.20
0.3:553
0.2532
0.2548
D2 +
18215
10.61
0.3499
0.2566
0.2581
D1 + D2
N/A
D2 Mo
MO
18061
10,96
0.3119
0.2571
0.2569
D2 + Mo
D2 MU + UU
189-9
11.14
0.3411
0.2492
0.2032
D2
MO +
19297
11.35
0.3401
0,2496
0.2547
Down, blocks. Crosf
blocks (Seif-Croos))
Mid blocks_ Cross]
Skeech
ControlNet
Down, blocks_ 2*2 Cross)
Down, blocks 2 2010 Crosf)
Mid blocks 2 2010 (SaffCCass
Up. blocks 0 3 (SelfxCross)
Up blocks 1 3*10 (Self-Cross)
Unet
ControlNet
N/A
ControlNet
Unet N/A
ControlNet
D1-D2-MO
N/A
ControlNet
D2+M0
Unet MO
ControlNet
D2+M0
D2+MD+U0
D2-M0
Unet
D1+D2HMOYUD +U1
Figure 12: Qualitative comparison of Dense Tuning (DT) applied different ControlNet layers. DT at mid block_0 0 yields coherent and well-controleed results, while extending DT to down_block: mid block_0 further improves structural consistency. In contrast, applying DT to earlier layers (e.g.. down_ block_ 1 over-amplifies sketch contours and edges, leading to visually intrusive ControlNet outlines and degraded image naturalness. Applying DT to mid-level U-Net layers leads to marginal visual differences, whereas tuning peripheral U-Net layers introduces noticeable degrading changes. Prompts and Sketches are from Figures
15 and
consistently. Given the limited performance gains observed in Table 2 and the growing computational overhead shown in Table Y, restricting DT dom_bbloc_2 and mid_block, yields the most favorable trade-off between controllability, efficiency, and overall visual quality.
16

---

Published in Transactions on Machine Learning Research (02/2026)
Based on these qualitative and quantitative observations, adopt down_block_ mid blockllokkkk. 0 as the default DT configuration in ControlNet.
C Visualization of Game Scenes
example, begin each prompt with 'Isometric view of game scene' to generate controlled synthetic images for game settings. This helps maintain consistent angle and style, igmoring any incoherent instance sketches that might appear in real-world scenes, thereby focusing on object placement and verifying text-image consistency. We generate all 50 complex scenes using hyperparameters identical to those used in the main results (Fig. 7 in the main paper), shown in Fig. 12 in the main paper and Fig. 16 The colored sketches are used solely to distinguish between different. instances, and the colors used are arbilrary without class or semantic information. To validate this, we also use grayscale sketches as input, and the resulting images are nearly identical under the same random seed (two columns pointed by the red arrows in Fig. 16p. Meanwhile, our approach is not limited to game scenes. We also test prompts without the fixed game scene phrase, resulting in more diverse angles and styles while maintaining the same quality in object placement and text-image consistency (One row pointed by the green arrows in Fig. 12 in the main paper.
Using game
D Visualization of Common Scenes
Qualitative Common Scenes. In the above experiments, primarily validate the controllability of our method for multi-instance generation in game scenes. However, this does not imply that our approach is limited to game scenarios. To further verify its capabilities, we design three sets of diverse scenes: (1) four common simple scenes; (2) two indoor scenes; and (3) three scenes featuring instances of the same type but with different color attributes. Without changing any hyperparametors, generations are presented in Fig. [7] In common scenes, our method effectively mitigates instance overlap under ControlNet control, while indoor scenes, it handles varied layouts well. For the challenging task of differentiating attributes within identical instances, our approach assigns distinct properties accurately. However, for uncommon attributes like generating a red cat, our method struggles due to limitations inherent in the original SDXL model.
E Metric of User Study
We conduct user study on 50 scenes, each with 6 variants, generating 100 images per scene. A Gradio-based evaluation interface is designed, which randomly selects one image from 120 sets to create sub-evaluation system, with images presented anonymonsly. 23 participants independently rate the images based on the following scale:
5: All instances are accurately placed, and overall image quality is high.
4: One instance is missing or misplaced, or All are placed with lower quality,
3: Two or three instances are missing or misplaced, or placed with lower quality.
2: Three four instances missing misplaced, placed with lower quality.
1: Multiple instances are missing, with low overall quality.
This detailed rating system helps assess both the accuracy of instance placement and the quality of generated images, whether the generations are aligned with text prompts and sketch layouts.
F Transfer PB to Attend-and-Excite
To further validate the PB module, we integrated into the Attend-and Breendand-EEE and---nd---ee Exx-ixecte method Chefer et al.
2023), based on attention tuning using the SD V1.4 model. The results shown in Figure 13 Despite the
17

---

Published in Transactions on Machine Learning Research (02/2026)
Attend
A cat and frog.
A cat and bird.
PB+ Attend
Attend
A cat and a dog.
A bird and pig.
PB+ Attend
Figure 13: Visualizations for transferring PB to Attend-and-Exe Exxiite (Chefer et al. 2023). In most cases, both instances are successfully generated. The frog-leg cat and the bird-wing pig further demonstrate the effectiveness since they lack the layouts to separate the instances spatially.
limitations of SD V1.4, the PB module effectively balances embedding strength between the two instances in scenarios without layout guidance, enhancing their representation. In most cases, both instances are successfully generated. However, in some cases, the attributes of the two objects become entangled, Icading to artifacts such as a cat with frog legs or a pig with bird wings, due to the lack of spatial separation, which further demonstrates the effectiveness the PB module.
G Transfer T3-225 to T21 T21-Adapter
Gray sketch
T21- Adapter
T3-S25 T21- Adapter
Figure 14: Visualizations for transferring T T2I-Adapter Mouu et al. 2023 T effectively improves the T2I-Adapter's alignment with prompts and layouts in complex scenes, demonstrating its control capabilities across different models.
18

---

Published in Transactions on Machine Learning Research (02/2026)
To validate the general applicability of our approach beyond the ControlNet model, apply T to another controllable T21-Adapter (Mou et al. 2023) model. Although the T2I-Adapter performs best with detailed sketches, we use grayscale sketches for quick validation, which contain less semantic information. We keep the PB and CP modulcs unchanged, while the DT module is integrated into the SDXL main channel, similar to CP, as it can not be placed in a separate branch like in ControlNet. We use the same prompls and sketches from the main results (Fig. unchanged. The results are shown in Figure 14 TS-S2 effectively improves the T21-Adapter's alignment with prompts and layouts in complex scenes, demonstrating its control capabilities across different models. However, the generation quality still lags behind the ControlNet-based approach, indicating the need for parameter tuning specific to the T2l-Adapter's distribution and improved sketch imputs to align with the Adl-err Despite these limitations, the results show that T has promising generalizability and can effectively control both ControlNet and Adl-pep models.
the main paper) and Appendix [ with all other hyperparameters
Perception Modeling for Structural Assembly
To bridge the transition between twin structural concept representations, i.c., isometric and terrain images, we follow sketch2scene (Xu et al. 2024) as our backend to build 3D game scenes from concept art generations. Structurally Specified Assembly module constructs full 3D natural scenes by segmenting foreground assets from isometric images, generating terrain meshes via tomography from terrain images, and applying sketch- based textures. This supports accurate terrain modeling, rich asset composition, and scamless integration with standard engines.
Foreground Instances. We extract 2D isometric images of individual objects (e.g., buildings, houses, trees) from the instance segmentation map via Segment Anything (Kirillov et al. 2023), which are later used for 3D retricval or conditioning in the 3D generation. Assuming an isometric projection with a 45 yaw angle, we estimate object poses by applying homography warping (Hartley & Zisserman 2003) and analyzing their bounding boxes. The warped instance segmentation provides the object footprint, while the depth information enables projection into 3D space.
MeightMap Gencration. The isometric view offers sufficiont coverage to recover both geometry and appearance from a single terrain image. We reconstruct a watertight terrain mesh from the image using Depth Anything (Yang al. 2024) for depth estimation, followed by Poisson surface reconstruction. From the coarse depth, we obtain an isometric view depth map and compute the height as h dmax d. The height is rotated into bird's-eye view (BEV) via homography warping (Hartley & Zisserman 2003) to build the terrain mesh, and then the color-aligned mesh is segmented via Segment Anything (Kirillov et al. 2023) and Osprey (uaan et al. 2023) into categories (grass, rock, mud, road, etc.). The categories map is used to retrieve standard texture tiles to ensure high-res appearance in close-up views. For water regions, we lower the terrain and insert a water asset to maintain realistic elevation.
Procedural 3D Scene Synthesize. Using semantic and geometric data, we procedurally generate and render 3D scenes via game engines (c.g., Unity, Unreal). We adopt Unity for its terrain, vegetation, and ani- mation support. Heightmaps and splatmaps are mapped to Unity terrain. Vegetation is procedurally placed based on texture semantics (e.g., grass + flowers, rocks). Foreground objects are placed by retrieval (from
Objaverse via. CLIP) or generated using 2D-to-3D models HHong et al. 2023 Wei et al. 2024 hyperhuman 2024), ensuring style consistency with the reference.
3D scene generations following the sketch2scene (Xu et al. 2024) are shown in Figs. 19 and 20
19

---

Published in Transactions on Machine Learning Research (02/2026)
sketch
ControlNet
T2.
Gray sketch
s2s
Prompt: Isometrie view of game seene, volcanic terrain, lava rocky cliff', crupting volcano.
Prompt: Isometric view of game scene, jungle, ruined temple, vines, waterfalls, a big rock.
Prompt: Isometric view game scene, dim swamp. murky water, wooden cabin, gruss, a bridge.
Prompt: Isometric view game seene, tundra, frozen house,
Prompt: isometric view of gume scene, forest, lake, path.
wooden house..
Prompt: Isometric view of game scene, urban park, playground, fences, trees.
Prompt: Isometric view a game seene. rocky beach, coastal cliffs, stormy occan, lighthouse, caves.
Prompt: Isometric view of gume small village, snowy
frozen lake, abundoned
castle, path, pine trees.
Remove the phrase "Isometric view of game scene" from the prompts above, keeping all other parameters unchanged, resulting in the following outputs.
Figure 15: Example results from subset of the 50 complex scene compositions tested using hyperparameters identical to those used in the main results (Fig. S). (1) Two columns pointed by the blue arrows represent the struetural concept representations using the prefix terrain prompts and isometric prompts under the same random seed. (2) Two columns pointed by the red arrows represent the generations using colored and
grayscale sketches under the same random seed. (3) One row pointed by the green arrows indicates the generations without the fixed game scene phrase.
20

---

Published in Transactions on Machine Learning Research (02/2026)
sketch
ControlNet
T7-22
Gray sketch
T $25
Prompt: isometric view game seene, seaside, golden heach, forest with trees, blue sea, wooden Holiday houses, beach umbrellas.
Prompt: Isometric view of seene, beach, sea, cabin, fence,
Prompt: Isometric view of game snowy plain, river, houses, pines.
Prompt: Isometric view ofa game scene, desert, cliff, pyramid, palm tree,
Prompt: Isometric view of game scene, city. road, houses, garden.
Prompt: Isometric view of game scene, countryside, river, bridge, house.
Prompt: Isometric view of game scene, beach, tropical islands, occan, palm trees.
Prompt: Isometric view of game seene. Gobi desert, dark clouds, stoep clift, mountain slope, road, broken cabin.
Prompt: Isometric view ofa seene, medieval village, cobblestone road, castle, market stalls.
Figure 16: Example results from subset of the 50 complex scene compositions tested using byperparameters identical to those used in the main results (Fig. 7). (1) Two columns pointed by the blue arrows represent the structural concept representations using the prefix terrain prompts and isometric prompts under the same random seed. (1) Two columns pointed by the red arrows represent the generations using colored and grayscale sketches under the same random seed.
21

---

Published in Transactions on Machine Learning Research (02/2020)
Sketch
ControlNet
T--22
Gray sketch
T2-25
Prompt:
monkey bmanc holding pink the top of huge tree, sky is wide and blue.
(a)
Prompt: A painting dog riding a flving bievele. over bie city with vellowish full moon in the night sky.
Prompt: painting couple
holding a velloww umbrella in street rainy nicht womnn wenring white dress and the man is wearing blue suit.
Prompt: realistic photograph of couple bolding yellow umbrella in street on a rainy night. The woman
white dress the wearing blue
(6)
Prompt: In an indour scene. beige cetling, black oven, wooden table, red chatns.
Indeor
Prompt: In restrom
sink with
cubinet, w.wtee eapy a arrundd minros,
white tmilet,
stemaee racks towel rack
Prompt: In
walkiing
blue
(c) Multi-colon scenes
Prompt: vand
pray cat
beige
vellow ead
blaek Cil
white cat.
Prompt: a vard black cat.
white cat, aod Blue green cat.
Figure 17: Examples of generated scenes across different settings. (a) Common simple scenes demonstrating effective instance representations under ControlNet control. (b) Indoor scenes showcasing robust handling
of varied instance layouts. (c) Scenes with identical instances but different color attributes illustrate precise difforentiation of properties.
22

---

Published in Transactions on Machine Learning Research (02/2026)
(a)
(b)
K=0
K-2
f=t
8=0.5
K-f
K=2
B-1
B-1
k-2
K-2
8=1
B=1.5
k-3
K-2
B=1
B=2
K=4
K=2
B-
B-3
Figure 18: Another visual comparison of two hyper- paprr-parameters K and 3 suggests that setting K = 2 and 3 = 1 is lavorable choice. With 8 = 1, increasing K initially improves results but event adds noise. Fixing K = 2, higher & values yield stable and favorable outcomes.
Prompt: Isometric view of game scene, a plain, walk path, river, high mountain, houses.
Prompt: Isometric view
of game scene, field with snow, iced hills, winding road, troes, a red house.
Figure 19: 3D scene generation results with two examples from Fig. The 3D scene shows that terrain and objects can be generated as concept art, but improvements are needed in understanding concept art, par- ticularly in segmentation and depth estimation, which may not be accurately represented in the synthesized data. Additional 3D scenes available in the Appendix and the supplementary video materials.
23

---

Published in Transactions on Machine Learning Research (02/2026)
Prompt: Isometric
Prompt:
Isometric view of a game scene, a snowy plain, river, houses, pines.
view of gume scene, desert, cliff, pyramid, palm tree.
Figure 20: 3D scene generation results with two more examples from Fig. 16 terrain and objects can be generated concept art, but improvements are needed in understanding concept art, particularly in segmentation and depth estimation, which may not be accurately represented in the synthesized data.
The 3D scene shows that
24


Reviews:
[{"review_id": "YiqUNRNQlg", "reviewer": "Reviewer_yNzH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces *Training-free Triplet Tuning for Sketch-to-Scene (T3-S2S)*, a novel method to improve the controllability and accuracy of multi-instance scene generation from sketches. \n\n **Key Contributions:**\n- Theoretical Insights: The authors investigate cross-attention mechanisms and identify problems with prompt energy imbalance, leading to poor instance representation.\n- T3-S2S Framework: Combines three modules:\n  - Prompt Balance (PB): Adjusts keyword energy to improve instance token competition.\n  - Characteristics Priority (CP): Enhances instance prominence using TopK value matrices.\n  - Dense Tuning (DT): Refines attention maps for better instance contours.\n- Feature Sharing: A dual-branch prompt synthesis for creating isometric and terrain-view representations.\n- Empirical Validation: Demonstrates that T3-S2S outperforms existing models like ControlNet in generating detailed and accurate multi-instance scenes.\n\n **Strengths**:\n- Training-free: No additional data required, adaptable, and cost-effective.\n- Improved instance generation: Addresses issues of missing small and rare instances in previous methods.\n- Clear experimental results: Empirical evidence supports the effectiveness of the method.\n\n **Weaknesses**:\n- Limited scalability: Struggles with large, complex scenes that require detailed inter-instance relationships.\n- Lack of direct comparison: No detailed comparison with cutting-edge models for large-scale scene generation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "- Clear visual comparisons with baseline models (e.g., ControlNet) showing improved handling of smaller and complex instances.\n- CLIP-Score evaluations for global image quality, instance representation, and background accuracy show significant improvements.", "audience_yn": "Yes", "audience_explanation": "- The paper contributes to improving sketch-to-scene generation.\n- Game developers, filmmakers, and content creators can benefit from the advancements in automating concept art generation.\n- The training-free nature of the approach is relevant to those working on scalable AI systems.", "requested_changes": "1. **Clarify Handling of Large-Scale Scenes**:\n   - The paper mentions challenges with generating large-scale scenes, but this section could be expanded to explain *why* the method struggles with large scenes, particularly in maintaining complex relationships between multiple objects. \n   - The authors should provide insights into how the current model's limitations impact the synthesis of large, intricate scenes (e.g., scenes with overlapping objects or large amounts of detail) and potential improvements or directions for overcoming these challenges in future work.\n\n2. **More Detailed Hyperparameter Tuning Discussion**:\n   - The tuning of hyperparameters like **TopK** and **β** plays a crucial role in the performance of the model. The paper should include more explicit guidance on:\n     - How **TopK** and **β** values were selected for the experiments (e.g., grid search, manual selection).\n     - The impact of different values on the model’s performance across various types of scenes (e.g., small scenes vs. large scenes, or simple vs. complex prompts).\n     - A clearer discussion of how these parameters could be adjusted in practical applications to improve scene generation for different types of inputs.\n\n3. **Comparison with Cutting-Edge Methods for Large-Scale Scenes**:\n   - While the paper demonstrates improvements over existing methods, it lacks a direct comparison with recent state-of-the-art models for large-scale scene generation, especially in cases that involve complex inter-object relationships.\n   - It would strengthen the paper to add an explicit comparison of T3-S2S with recent large-scale scene generation methods to show its relative performance in generating high-fidelity multi-instance scenes.\n\n4. **Further Explanation of Dense Tuning Modifications**:\n   - The **Dense Tuning (DT)** module is key to improving performance, but the paper could elaborate more on why dense tuning at certain layers (\"down_block_2\" and \"mid_block_0\") yields the best results. A clearer explanation of why these layers were specifically chosen and how their tuning impacts performance would be beneficial.\n   - A deeper analysis of potential drawbacks or trade-offs when applying dense tuning to these layers could help readers better understand its limits and provide guidance for future applications.", "broader_impact_concerns": "- Bias in generation: While the method is training-free, biases could still emerge from base models like ControlNet. The paper should discuss potential fairness and bias issues in content generation.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zf9TNj3yId", "reviewer": "Reviewer_pQcf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "A training free method to synthesize a scene based on a sketch consist of three modules: Prompt Balance (PB), Characteristics Priority (CP), and Dense Tuning (DT). PB finds instance keywords in the prompt, replaces multi-word phrases with single-token embeddings, and rescales their norms to match the End-of-Text token, keeping the prompt energy balanced across instances. CP applies a per-channel Top-K selection on value activations to highlight instance-relevant features within sketch regions. DT modifies ControlNet with a lightweight Dense Diffusion step that sharpens attention at specific blocks without retraining. The method also uses a dual-branch framework to generate both isometric and terrain views, leading to more coherent and controllable terrain layouts.\n\nStrength:\n\n1. The method is training free require no additional data \n2. good theoretical justifications\n3. comprehensive ablations\n\nWeakness:\n\n1. limited analysis of failure cases\n2. quantitative evaluation not super impressive (I am not by any means an expert in this field so I don't know what significance do those number represents)\n3. the quantitative results only has 50 examples, and is not a publicly open dataset, which can cause some doubts.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Yes, figure 10 and table 1 shows the quantitative results while many other figures present qualitative results, which looks convincing.", "audience_yn": "Yes", "audience_explanation": "yes, for people working on controllable generation", "requested_changes": "The quantitative results is not solid enough. Could you expand you analysis to a higher scale for better statistical significance? Also would be great if there are some disclosure in how the eval set are constructed.", "broader_impact_concerns": "NA", "additional_comments": "This area is totally out of my expertise so I am especially unconfident about my judgement.", "extra_scores": {}, "extra_text": {}}, {"review_id": "cp0cYDGxER", "reviewer": "Reviewer_wShr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper addresses a critical challenge in AI-generated content creation: improving image/video generation without fine-tuning by enhancing the generation of items mentioned in sketches and prompts. The paper identifies key shortcomings in existing solutions, particularly the problem of forgetting low-energy key items from text and sketches, and proposes a solution to increase the energy of these low-energy keys during generation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "This paper makes a strong contribution to controllable image generation through a well-defined problem statement, rigorous theoretical analysis of cross-attention mechanisms, and an innovative three-module solution (Prompt Balance, Characteristics Priority, and Dense Tuning). The proposed approach is training-free, practical, and cost-effective, while maintaining compatibility with ControlNet’s sketch-following ability. The authors support their claims with solid experimental validation, including CLIP scores, user studies, ablations, and transfer tests to related models.", "audience_yn": "Yes", "audience_explanation": "Yes. The findings would be of clear interest to TMLR’s audience, particularly those working on controllable text-to-image generation, diffusion models, and cross-attention mechanisms. The paper addresses a well-motivated challenge—multi-instance generation—and proposes a modular, training-free solution with strong theoretical backing and practical utility. Researchers and practitioners in machine learning, computer vision, and generative modeling would benefit from both the methodological insights and the empirical results.", "requested_changes": "The main requested change is to provide a clear justification for applying Dense Tuning only at “down_block_2” and “mid_block_0” in ControlNet. To strengthen the paper, the authors should include empirical comparisons of different layer combinations, an analysis of the ControlNet feature hierarchy to explain why mid-level features are most effective, and a cost-benefit evaluation covering memory usage, inference time, and performance trade-offs. In addition, providing guidelines on how to adapt layer selection for other architectures would greatly improve reproducibility and generalizability.", "broader_impact_concerns": ".", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)