T$^3$-S2S: Training-free Triplet Tuning for Sketch to Scene Generation

Authors: Zhenhong Sun, Yifu Wang, Yonhon Ng, Yunfei Duan, Daoyi Dong, Hongdong Li, Pan Ji
Primary area: generative models
Keywords: sketch-to-scene generation, training-free diffusion model, cross-attention mechnism

Abstract:
Scene generation is crucial to many computer graphics applications. Recent advances in generative AI have streamlined sketch-to-image workflows, easing the workload for artists and designers in creating scene concept art. However, these methods often struggle with complex scenes with multiple detailed objects, sometimes missing small or uncommon instances.
In this paper, we propose a Training-free Triplet Tuning for Sketch-to-Scene (T$^3$-S2S) generation after reviewing the entire cross-attention mechanism. This scheme revitalizes the existing ControlNet model, enabling effective handling of multi-instance generations, involving prompt balance, characteristics prominence, and dense tuning. 
Specifically, this approach enhances keyword representation via the prompt balance module, reducing the risk of missing critical instances. It also includes a characteristics prominence module that highlights TopK indices in each channel, ensuring essential features are better represented based on token sketches. Additionally, it employs dense tuning to refine contour details in the attention map, compensating for instance-related regions.
Experiments validate that our triplet tuning approach substantially improves the performance of existing sketch-to-image models. It consistently generates detailed, multi-instance 2D images, closely adhering to the input prompts and enhancing visual quality in complex multi-instance scenes.

Decision: withdrawn
Reviewer scores: 6, 5, 5, 8

Reviews and discussion:
<thread>
creation date:
1730216571734

thread_type:
official_review

<thread_object>
creator: reviewer_wtqn

<content>
summary:
This paper proposes an approach, termed Triplet Tuning for Sketch-to-Scene (T3-S2S), a training-free method designed to enhance T2I generation capabilities in multi-instance scenarios, effectively mitigating challenges such as instance omission. The T3-S2S method introduces a prompt balance mechanism to automatically balance the energy of each instance within text embeddings and incorporates a characteristics prominence mechanism that strengthens cross-attention by highlighting Top-K indices within each channel, ensuring that essential features are more robustly represented based on token sketches. Methodologically, T3-S2S demonstrates certain innovations compared to previous approaches, and the results presented by the authors indicate that this method indeed yields notable improvements. Therefore, at this stage, I would rate this paper as [weak accept]. However, I believe the current experimental results are still insufficient; please see the question section for details. I look forward to the authors' responses.

strengths:
Methodologically, T3-S2S demonstrates certain innovations compared to previous approaches, and the results presented by the authors indicate that this method indeed yields notable improvements.

weaknesses:
I believe the current experimental results are still insufficient. First, the paper does not include comparisons with state-of-the-art (SOTA) methods on popular benchmarks. Additionally, it lacks quantitative comparison results, which are essential for a comprehensive evaluation of the method’s effectiveness.

questions:
(1) The prompt balance mechanism could also be applied to text-to-image models. I recommend testing the effectiveness of the prompt balance mechanism on some multi-instance text-to-image benchmarks. Specifically, you could compare it with training-free optimization algorithms designed for multi-instance T2I scenarios, such as AAE [1] and StructureDiff [2].

(2) When performing the characteristics prominence mechanism, the paper states that the optimal setting for K is 2. When the number of instances exceeds K, how does your characteristics prominence mechanism ensure that each instance is effectively enhanced? More specifically, how does it handle multiple instances that belong to the same category but have different fine-grained attributes? For example, when dealing with five cats—one red, one blue, one green, one black, and one white.

(3) I have some questions regarding specific details. When performing characteristics prominence, should the operation be applied on cross_attention(I, T) or on I + cross_attention(I, T), where I is the image feature and T is the text embedding?

(4) I believe further experiments on widely used multi-instance generation benchmarks are necessary to better demonstrate the effectiveness of your method.

[1] Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models.

[2] TRAINING-FREE STRUCTURED DIFFUSION GUIDANCE FOR COMPOSITIONAL TEXT-TO-IMAGE SYNTHESIS.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730370453030

thread_type:
official_review

<thread_object>
creator: reviewer_7juo

<content>
summary:
This paper identifies a phenomenon in sketch-to-scene generation where certain instance are not effectively represented in complex, multi-instance scenes. The authors attribute this issue to the imbalance of prompt energy and the homogeneity of value matrices. To address these challenges, the paper proposes a method that incorporates prompt balance, characteristic prominence, and dense tuning for sketch-to-image generation in complex scenes.

strengths:
1. The paper provides a thorough analysis of text embeddings and the value matrices in cross-attention, revealing the issues of prompt energy imbalance and value matrix homogeneity.

2. The proposed method seemly  addresses the challenges associated with sketch-to-scene generation involving multiple subjects.

weaknesses:
My primary concern lies with the clarity of the writing and the evaluation methodology:

1. The writing lacks clarity and the notation system is somewhat confusing.
2. Although the method does not require training, the optimization during inference may increase time demands, yet the paper does not report specific inference times.
3. The results are solely qualitative, lacking quantitative comparisons. The effectiveness of the proposed modules is not validated with measurable metrics, raising concerns about potential cherry-picking in qualitative assessments. While it may be challenging to establish reasonable quantitative benchmarks for this task, there even is no mention of conducting user studies to support the findings.

questions:
see weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730427326187

thread_type:
official_review

<thread_object>
creator: reviewer_jftx

<content>
summary:
This article primarily focuses on the task of sketch-to-scene generation. Specifically, for complex scenes with multiple detailed objects, previous methods sometimes miss small or uncommon instances. Addressing this issue, the article proposes a training-free triplet adjustment method for sketch-to-scene generation. This method can be directly applied to existing SDXL and ControlNet models, enabling them to effectively tackle the multi-instance generation problem, including prompt balancing, feature highlighting, and dense adjustment. The proposal is made after reviewing the entire cross-attention mechanism. This solution revitalizes the existing ControlNet model, allowing it to effectively handle multi-instance generation, involving prompt balancing, feature highlighting, and dense adjustment. Experiments demonstrate that this triplet adjustment method enhances the performance of existing sketch-to-image models, enabling the generation of detailed, multi-instance 2D images that closely follow input prompts and enhance visual quality in complex multi-instance scenes.

strengths:
1. This article addresses a very meaningful area, utilizing the approach of scene generation through sketches.

2. The method requires no training and can be seamlessly integrated into existing controllable generation models, thereby enhancing the models' generative performance.

weaknesses:
1. The experimental content of this article is somewhat lacking in depth. From the presented experimental results, it only includes the generation of isometric view of game scene using colored sketch lines as control inputs. This raises the question of whether the method proposed in this article has universal applicability to scene sketch generation tasks.

2. According to the description in this article, this training-free method at the feature level should be universally applicable to various controllable generation models. The method proposed in this article is based on ControlNet, but the article does not sufficiently discuss whether this approach would still be effective when integrated into other controllable generation models.

questions:
1. Since this method can enhance the performance of existing controllable generation models without the need for training, is it possible to include experimental analyses based on more diverse existing models, such as the T2I-Adapter (https://arxiv.org/abs/2302.08453)   that the article compares?
If additional experiments could be added to demonstrate this, it would better showcase the universal applicability of the method proposed in the article, significantly enhancing its contribution.

2. Can the method proposed in this article be applied to a wider range of scenarios? From the experimental results presented in the article, it seems that simply altering the prompt should be sufficient to produce a more diverse array of scene images. I hope to see more experimental outcomes. Is it possible to incorporate a wider variety of input types (for example, black and white hand-drawn sketches)? Can the output be expanded to include a richer variety of scene images (for example, real-world images), not just limited to the isometric view of game scenes?
Particularly, the experimental results in Figure 6 of the article suggest that the method's control over the style of generated images should stem solely from the specific input prompt "Isometric view of game scene." Therefore, if the article could showcase more use cases across a variety of application scenarios, it would greatly increase the value of the method proposed in the article.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730706796984

thread_type:
official_review

<thread_object>
creator: reviewer_s5su

<content>
summary:
The paper proposes a Training-free Triplet Tuning (T³-S2S) method for sketch-to-scene generation, aiming to enhance the quality of generated scenes from sketches without additional training. The authors identify challenges in existing diffusion models related to imbalanced prompt energy and value homogeneity in the cross-attention mechanism, which lead to missing or coupled instances in complex scenes. To address these issues, they introduce two strategies: Prompt Balance and Characteristics Prominence. Additionally, they incorporate Dense Tuning from Dense Diffusion to refine attention maps. The proposed method is evaluated qualitatively on game scene generation tasks, demonstrating improvements in generating detailed, multi-instance scenes that align with input sketches and prompts.

strengths:
1. The proposed Prompt Balance and Characteristics Prominence strategies contribute to enhancing the quality of generated scenes, particularly in handling complex scenes with multiple instances.
2. The method is relatively simple and does not require additional training, making it efficient and practical.

weaknesses:
1. The proposed Prompt Balance and Characteristics Prominence strategies appear to be incremental improvements on existing techniques such as TopK methods. Dense Tuning is adapted from Dense Diffusion. Therefore, the paper may lean more towards engineering optimization rather than presenting significant novel techniques.
2. The experimental comparisons use models based on different versions of Stable Diffusion, with the proposed method and T2I Adapter using SDXL (a more advanced model), while Dense Diffusion is based on SD v1.5. Since SDXL offers higher generation quality than SD1.5, this results in an unfair comparison. Considering that this work draws inspiration from Dense Diffusion, it would be more appropriate to either adapt Dense Diffusion to SDXL or apply the proposed method to SD1.5 to ensure a fair evaluation.
3. The experiments focus primarily on game scenes, with limited variety in the types of scenes presented. Some examples are repeated in the paper, and the lack of diverse examples may not fully demonstrate the method's generalizability to other contexts. 
4. The paper focuses heavily on qualitative analysis and lacks quantitative experiments. 
5. Line 360 and 361: citations of SDXL and ControlNet are incorrect.

questions:
1. I wonder whether the proposed method is effective for other types of scenes. Exploring and presenting results on different scene categories would be beneficial.
2. I suggest to incorporate user studies to enhance the robustness of the experimental validation.
3. The appendix seems to be directly attached after the references. It might be more appropriate to format the appendix according to the conference guidelines, ensuring it is properly integrated or submitted as a supplementary document as required.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732075605996

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response

comment:
Dear reviewers and meta reviewers,

Thank you for your valuable feedback (we refer Reviewer s5sU as **R1**, JFtx as **R2**, 7JUo as **R3**, wtqN as **R4**).
We are glad to see your positive comments on our method, e.g., "Contribute to the quality" (**R1**, **R2**), "Handling complex scenes" (**R1**), "Relatively simple" (**R1**), "Training-free" (**R1**, **R2**), "Efficient" (**R1**), "Practical"(**R1**), "Addresses a very meaningful area" (**R2**), "Seamlessly integrated"(**R2**), "Provide a thorough analysis" (**R3**), "Addresses the challenges of S2S" (**R3**), "Certain innovations" (**R4**) and "Indeed yields notable improvements" (**R4**).

According to your valuable comments and suggestions, we have thoroughly revised our manuscript by adding more details and studies, as part of the ICLR revision process. In particular, we created and tested 20 complex scene sketches, each with over four sub-prompts, covering various terrains (plains, mountains, deserts, glaciers, cities) and diverse elements (rivers, bridges, stones, castles). We conducted quantitative evaluations tailored to sketch-to-scene scenarios using both CLIP-Score and user studies. Additionally, we included results for diverse scenes to validate multi-scenario applicability, including: a) common simple scenes; b) indoor scenes; and c) scenes featuring instances of the same type but with different color attributes. Through these quantitative results and visualizations, our approach demonstrates superior performance, achieving higher fidelity and precision in aligning text prompts and sketch layouts for multi-instance synthesis.

We understand that this may add to your workload, but we sincerely appreciate your time and efforts in reviewing our improvements based on your feedback. Please also find our specific response to each reviewer below.

Best regards,

Authors

***
`Additionally, the changes have been highlighted using the blue font in the revised paper, and we will release our code in the camera-ready version. The details of the revision are summarized as follows:`

* We have implemented quantitative experiments and user studies using commonly used metrics in **"Metrics" of Section 5.1, "Quantitative Evaluation" of Section 5.2, and Appendix E**, both of which demonstrate good improvements of our T$^3$-S2S approach over the ControlNet model (**R1/R3/R4**).

* We have added more results for different sketches of complex scene composition in **Figures 12 and 13 of Appendix C**, each with more than four sub-prompts, encompassing various terrains (plains, mountains, deserts, tundra, cities) and diverse instances (rivers, bridges, stones, castles) (**R1/R2**).

* We have added more results of diverse scenes in **Figure 14 of Appendix D** to verify multi-scenario applicability, including (1) four common simple scenes; (2) two indoor scenes; and (3) three scenes featuring instances of the same type but with different color attributes (**R1/R2/R4**).

* We have experimented with transferring T$^3$-S2S to T2I-Adapter in **Figure 11 of Appendix G**, and have validated the general applicability and performance improvements using our approach beyond the ControlNet model (**R2**).

* We have experimented with transferring the Prompt Balance module to Attend-and-Excite in **Figure 10 of Appendix F**, to further validate the effectiveness of the PB module (**R4**).

* We have applied the SDXL model to Dense Diffusion for a fair comparison in **the "Baselines" of Section 5.1 and Figure 6** (**R1**).

* We have fixed several typos, such as citations and the tense (**R1/R3**).

* We have moved the Top K analysis from the main text to **Appendix B**.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732766698618

thread_type:
official_comment

<thread_object>
creator: area_chair_zcs2

<content>
title:
Reviewer feedback on rebuttal

comment:
Dear Reviewers,

As the discussion period will end next week, please take some time to read the authors' rebuttal and provide feedback as soon as possible. For reviewers s5sU, JFtx, and 7JUo, did the author rebuttal address your concerns, and do you have further questions?

Thanks,
Area Chair
</content>
</thread_object>

</thread>
<thread>
creation date:
1733047207185

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response and Appreciation

comment:
Dear Reviewers, AC, and SAC,

Thank you all for your valuable comments on our paper. As the rebuttal process draws to a close, we are delighted that our responses have been able to address most of the reviewers' concerns, as no further questions have been raised. We would also like to sincerely thank the AC for actively facilitating discussions during this process.

We deeply appreciate your time and effort in guiding us through this journey.

Thank you once again for your support and feedback.

Best regards,

Authors
</content>
</thread_object>

</thread>


Metareview: