SG-Adapter: Enhancing Text-to-Image Generation with Scene Graph Guidance

Authors: Guibao Shen, Luozhou Wang, Ziyang Mai, Jiantao Lin, Wenhang Ge, Chaozhe Zhang, Xin Tao, Pengfei Wan, Di ZHANG, Guangyong Chen, Yijun Li, Ying-Cong Chen
Primary area: generative models
Keywords: diffusion models, scene graph, image generation

Abstract:
Recent advancements in text-to-image generation have been propelled by the development of diffusion models and multi-modality learning. However, since text is typically represented sequentially in these models, it often falls short in providing accurate contextualization and structural control. So the generated images do not consistently align with human expectations, especially in complex scenarios involving multiple objects and relationships. In this paper, we introduce the Scene Graph Adapter(SG-Adapter), leveraging the structured representation of scene graphs to rectify inaccuracies in the original text embeddings. The SG-Adapter's explicit and non-fully connected graph representation greatly improves the fully connected, transformer-based text representations. This enhancement is particularly notable in maintaining precise correspondence in scenarios involving multiple relationships. To address the challenges posed by low-quality annotated datasets like Visual Genome, we have manually curated a highly clean, multi-relational scene graph-image paired dataset MultiRels. Furthermore, we design three metrics derived from GPT-4V to effectively and thoroughly measure the correspondence between images and scene graphs. Both qualitative and quantitative results validate the efficacy of our approach in controlling the correspondence in multiple relationships.

Decision: reject
Reviewer scores: 6, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1730521309563

thread_type:
official_review

<thread_object>
creator: reviewer_wq1r

<content>
summary:
This paper presents a Scene Graph Adapter for text-to-image generation that leverages scene graph to refine CLIP text embeddings. Additionally, a dataset of 309 images with relational annotations is proposed for evaluation.

strengths:
1. The paper contributes a scene graph dataset with relational annotations, which adds value to the community.
2. The authors include their code in the supplemental materials.
3. Human evaluations are conducted to assess the effectiveness of the proposed method.

weaknesses:
1. Experiments are conducted only on the proposed dataset, which is relatively small and may not provide a robust evaluation.
2. The method uses a Scene Graph to refine CLIP text embeddings, which may be overly simplistic. I think the adapter should also change the parameters of the Stable Diffusion.
3. The latest method compared in Table 1 is from 2023. Including more recent methods would strengthen the comparative analysis.
4. The authors note that fine-tuning Stable Diffusion on a small dataset can degrade FID (shown in Table 1). Fine-tuning on a larger dataset, such as Flickr30k Scene Graph, is recommended.
5. Human evaluation is conducted on only 20 cases (from the proposed test set). Expanding the test set could improve the reliability of these results.

questions:
See the detail in the weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730605103641

thread_type:
official_review

<thread_object>
creator: reviewer_bw61

<content>
summary:
This paper hopes to optimize the generation of entity interaction in text-to-image generation by introducing the scene graph structure into the adapter. The method is simple and clear and has achieved significant improvement compared to the baseline.

strengths:
- The motivation is straightforward. The description of the problem and method is simple, clear and understandable.
 - Entity interaction information is introduced through scene graph to optimize the generation of entity interaction, which is relatively direct.
 - Judging from the demo and numerical comparison results, there are good results.

weaknesses:
- The novelty in method design is limited. It only performs an attention on the text embedding based on the scene graph relationship. It can even be considered as a replacement of the key and value values in the cross attention structure. Simplicity is not a disadvantage. If it is simple, direct and works, the author can provide some discussions to analyze the most important factors behind it.
Is the adapter structure important or is the scene graph information the most important? For example, in addition to the adapter, can simply injecting scene graph information through cross attention also obtain similar results?
And whether there are some situations where scene graph information is not applicable.
 - The experimental evaluation of entity interaction generation is not comprehensive enough. The scene graph detection method itself will also have a very obvious long-tail phenomenon for entity relationships. For entity interaction image generation, this should also be considered. Distinguishing and explaining according to the detection frequency in the training dataset will make the evaluation of the model more comprehensive and convincing. The author can try to divide buckets according to the frequency of scene graph detection, and finally give the results of each bucket and the average result.

questions:
-  The currently shown results (Figure 3) seem to have at most two main bodies interacting with their accessories. 1) In the case of more main bodies, such as a crowd surrounding or three people facing away from each other; 2) For some pairs that may not be common in the training set, such as pandas riding on horsebacks, what is the generation effect like? This helps evaluate whether the sg-adapter has learned to generate general interaction relationships or just high-frequency combinations in the training set.
 - The effect of scene graph detection may not be 100% accurate. What will be the effect when the detection result conflicts with the meaning of the text itself? Will it be more serious for some freely input long texts? For example, when users use the text-to-image function, they often add a long string of magic prompts, such as 4K, realistic, specific lenses, etc. These may all have an impact on scene graph detection.
 - How many parameters need to be fine-tuned in the current adapter part? Will the size of the number of parameters affect the final effect? The analysis of this part may make the article more complete.

In general, I think the problem solved by this paper is quite important. However, I feel that the current method is not general enough and may be a temporary technique. When there is a specific need for interaction generation in a specific scene, fine-tuning based on the SG Adapter will have some improvement. But it does not fundamentally solve the problem of entity interaction. Because I think the introduction of the SG adapter will lead to the problem that entity interaction adds the long-tail problem of scene graph detection on top of the long-tail problem of interaction relationships in the training set itself. The impact on the community may be limited. If the author has more thoughts and ideas on this aspect, we can discuss them.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730700958687

thread_type:
official_review

<thread_object>
creator: reviewer_3tge

<content>
summary:
To correctly align text-image consistency for text-to-image task, especially in complex scenarios involving multiple objects and relationships, this paper proposes SG-Adapter to leverage the structured representation of scene graphs to rectify inaccuracies in the original text embeddings. A highly clean, multi-relational scene graph-image paired dataset MultiRels is constructed to address the challenges posed by low-quality annotated datasets. Qualitative and quantitative experiments validate the efficacy of the proposed method in controlling the correspondence in multiple relationships.

strengths:
a. The proposed SG-adapter is effective in correcting the incorrect contextualization in text embeddings and enhancing the structural
semantics generation capabilities of current text-to-image models.

b. Both qualitative and quantitative experiments demonstrate SG-Adapter outperforms compared SOTA methods.

weaknesses:
a. The format of citations in this paper need be rectified.

b. This method requires the construction of a high-quality dataset, and it is mainly effective for the relationships within the constructed dataset, which will limit its generalization ability.

c. The quantitative results are insufficient and the author should provide more results to prove the its effectiveness.

questions:
Can SG-Adapter solves the color bind problems?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730738459744

thread_type:
official_review

<thread_object>
creator: reviewer_eblh

<content>
summary:
This paper proposes SGAdapter which solves the problem of using sequential representations for input text in recent text-to-image generation methods. They augment their approach with scene-graph-to-image generation due to the graph representations. SGAdapter introduces a small-scale adapter between StableDiffusion and the CLIP encoder for this task. They also identify the problem of casual attention masks for scene graphs. They curate a new dataset for scene-graph-to-image generation. The experiments show that SGAdapter can generate realistic images from more complicated input sentences than baselines.

strengths:
1. The method is simple and straightforward in solving the identified problem.
2. The empirical improvement is significant.

weaknesses:
1. Some figures are not clear.
2. Some confusing notations.

questions:
1. L40 missing space between "CLIP model" and the citation.
2. The $\mathbf{M}^{\tau}$ in Eq. 3 can be named differently, e.g. triplet-triplet mask, to distinguish it apart from the token-triplet attention mask in Eq. 5.
3. For Fig. 2, I suggest putting some of the introduced notations in Fig. 2 to help the reader understand the paper, e.g. model $f$, embedding $\mathbf{e}, \mathbf{w}$, etc. Fig. 2 can also be improved by using more arrows between features or modules to make it more concise and clear. For example, the inputs and outputs on the right side of the figure should be indicated with arrows. Redundant word in L233 Eq (equation) 4. The visualization of the matrix and the given scene graph example do not match. There are 2 triplets with 6 tokens in the input scene graph. I think it is a good opportunity to show what is $\mathbf{M}^{sg}$ and $\mathbf{M}^{\tau}$ for the inputs here.
4. L267, $\mathbf{M}$ or $M$? 
5. In Eq. 5, what is the difference between $\mathbf{M}^{sg}$ or $M$?
5. I am confused about Tab. 2 and L413, is it about $\mathbf{M}^{\tau}$ or token-triplrt mask $\mathbf{M}^{sg}$. I thought $\mathbf{M}^{\tau}$ is for solving the problem of causal masks.
6. Fig. 1, I assume the add signs mean StableDiffusion + SGAdapter and Caption + Scene graphs. I think it can made clearer as initially I was not sure whether you were replacing captions with scene graphs or using both.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733195808578

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We sincerely thank all reviewers for their valuable comments and insightful feedback. We appreciate that reviewers recognise both the experimental results(reviewer EbLH-"**improvement is significant**", 3Tge-"**outperforms SOTA methods**", Bw61-"**good results**") and motivation(Bw61-"**motivation is straightforward**", EbLH-"**method is straightforward**") of SG-Adapter. We have revised our paper based on the reviewers' comments, with changes highlighted in blue in the updated PDF. Before the detailed response to each reviewer, we kindly request that the reviewers and the Area Chair review this blog content first. It primarily addresses potential misunderstandings and common concerns regarding our paper. For specific questions, we have provided detailed responses in each reviewer's section.

## 1.Citation Format Issue (for reviewer EbLH and 3Tge)
We appreciate reviewer EbLH and 3Tge for their careful review and we have corrected the format of citations in our revised manuscript.

## 2.Experiment Results on Large Scale Dataset(for reviewer 3Tge and wQ1r)
**We have included experiment results on large scale dataset flickr30k[1] in our initial manuscript Section 4.6, Table 3 and Figure 4**. Furthermore, during the rebuttal period, we extend such results by combining 2 more baselines and 3 more metrics to comprehensively demonstrate the generalization performance of SG-Adaptor. We believe that the updated Table 3 as follows could fairly show the effectiveness of our methods:
| **Method** | **SG2IM[2]** | **PasteGAN[3]** | **SGDiff[4]** | **SceneGenie[5]** | **R3CD(AAAI24)[6]** | **ISGC(24)[7]** | **SG-Adapter** |
|------------|--------------|-----------------|---------------|-------------------|---------------------|-----------------|----------------|
| **FID** $\downarrow$ | 99.1 | 79.1 | 36.2 | 62.4 | 32.9 | 38.1 | **25.1** |
| **Inception Score** $\uparrow$ | 8.20 | 12.3 | 17.8 | 21.5 | 19.5 | 30.2 | **57.8** |
| **SG-IoU** $\uparrow$ | 0.085 | 0.091 | 0.122 | - | - | - | **0.413** |
| **Entity-IoU** $\uparrow$ | 0.297 | 0.382 | 0.436 | - | - | - | **0.729** |
| **Relation-IoU** $\uparrow$ | 0.253 | 0.297 | 0.394 | - | - | - | **0.681** |

For R3CD, ISGC and SceneGenie, we just report the inception score and FID from their paper since they do not provide the code.

## 3.Generation of specified application scenarios(for reviewer 3Tge and Bw61)
To validate the generalization performance of our SG-Adapter, **we successfully generate every specified case and scenario as the reviews requests**: case "panda riding on horse" and scenario "crowded scenarios" for reviewer Bw61; scenario "color bind problem" for 3Tge. We have present detailed results in Appendix A.7 Figure 7, Figure 8 and Figure 9 in the revised manuscirpt.


[1] Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence model.  
[2] Image generation from scene graphs.  
[3] Pastegan: A semiparametric method to generate image from scene graph.  
[4] Diffusion-based scene graph to image generation with masked contrastive pre-training.  
[5] Scene graph to image synthesis via knowledge consensus.  
[6] R3cd: Scene graph to image generation with relation-aware compositional contrastive control diffusion.  
[7] Scene graph to image synthesis: Integrating clip guidance with graph conditioning in diffusion models.
</content>
</thread_object>

</thread>


Metareview:
metareview:
While both R1 and R2 viewed the paper marginally above the acceptance threshold (based on the simpleness and effectiveness of the method in correcting the incorrect contextualization in text embeddings and enhancing the structural semantics generation capabilities), they shared concerns about poor presentation (i.e., unclear figures, confusing notation, and citation format). R2 also questioned the need for high-quality datasets and asked for more quantitative results (which are addressed by the authors).  R3 and R4 leaned toward rejection of the paper due to its limited novelty, insufficient experimental evaluation, and narrow generalization. With the introduction of SG-Adapter and a new dataset, the method is considered incremental and overly simplistic, failing to provide a fundamental advancement in the field. While the authors responded to both R3 and R4 in addressing the above concerns, both reviewers read the responses and decided marginally below the acceptance threshold. Weighing all the factors and the review quality, I tend to agree with R3 and R4 and rate the paper marginally below the acceptance threshold.