IFAdapter: Instance feature control for grounded Text-to-Image Generation

Authors: Yinwei Wu, Xianpan Zhou, bing ma, Xuefeng Su, Kai Ma, Xinchao Wang
Primary area: generative models
Keywords: Generative diffusion models, Layout to image generation

Abstract:
While Text-to-Image (T2I) diffusion models excel at generating visually appealing images of individual instances, they struggle to accurately position and control the features generation of multiple instances. The Layout-to-Image (L2I) task was introduced to address the positioning challenges by incorporating bounding boxes as spatial control signals, but it still falls short in generating precise instance features. To address this Instance Feature Generation (IFG) task, we introduce the Instance Feature Adapter (IFAdapter). The IFAdapter enhances feature depiction by incorporating additional appearance tokens and utilizing an Instance Semantic Map to align instance-level features with spatial locations. The IFAdapter guides the diffusion process in a plug-and-play module, making it adaptable to various community models. For evaluation, we contribute an IFG benchmark and develop a verification pipeline to objectively compare models’ abilities to generate instances with accurate positioning and features. Experimental results demonstrate that IFAdapter outperforms other models in both quantitative and qualitative evaluations.

Decision: reject
Reviewer scores: 6, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1729756348796

thread_type:
official_review

<thread_object>
creator: reviewer_e5vg

<content>
summary:
The authors aim to tackle the challenge of achieving controllability in generating precise instance features. To do this, they introduce the Instance Feature Adapter (IFAdapter), designed for instance-level positioning and feature representation. Specifically, the IFAdapter employs learnable appearance queries that extract instance-specific feature information from descriptions, creating appearance tokens that complement EoT tokens. Additionally, the IFAdapter constructs a 2D semantic map to link instance features with designated spatial locations, providing enhanced spatial guidance. In areas where multiple instances overlap, a gated semantic fusion mechanism is utilized to mitigate feature confusion.

To validate their approach, the authors have created a new dataset, referred to as the COCO IFG benchmark. They leverage existing state-of-the-art Vision Language Models (VLMs) for annotation, resulting in a dataset with detailed instance-level descriptions. Experimental results indicate that the proposed plug-and-play component surpasses baseline models in both quantitative and qualitative assessments.

strengths:
1. A plug-and-play component that can be integrated with various existing models to enhance layout control capabilities.
2. The paper also includes a user study, offering valuable insights into the proposed method.

weaknesses:
1. The quality of the proposed dataset has not been evaluated. It appears that all ground truths (GTs) are generated by existing Vision Language Models (VLMs). A human-level quality assessment would be beneficial for greater impact within the community.
2. The assertion that the proposed component can “seamlessly empower various community models with layout control capabilities without retraining” (l.113) may be misleading. The IFAdapter is fundamentally a training-based method, and the phrase “without retraining” only holds true when applied to spaces closely aligned with COCO IFG, as the IFAdapter does not demonstrate zero-shot capabilities in this paper.
3. The semantic-instance map does not appear to be novel. Please refer to "BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion" (ICCV 2023) and other zero-shot L2I methods for comparison.
4. The appearance tokens show only minor improvements in Table 3. Additional explanations regarding this observation would be appreciated.

questions:
Please kindly provide visual comparisons between the ground truth (GT) and the generated images.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730614345056

thread_type:
official_review

<thread_object>
creator: reviewer_nx7q

<content>
summary:
- This work tackles instance Instance Feature Generation (IFG) task, i.e. train a model that can generate images given a global caption, spatial locations and detailed local captions as conditioning.
- Authors introduce IFAdapter, a plug-and-play module for improving IFG.
- The IFAdapter first extracts a fixed number of appearance tokens from a detailed instance caption and its spatial location. Next, a 2D map called Instance Semantic Map (ISM) is constructed using the bounding boxes of instances to aid the adherence of the model to spatial location conditions.
- IFAdapter is architecture agnostic and can be incorporated into existing open-source text to image models and authors show its effectiveness on the newly introduced IFG Benchmark constructed from the COCO dataset.

strengths:
- The perceiver like Resampler design to extract fixed set of appearance tokens is novel and effective. This addresses the problem of only utilizing the EoT token from the text encoders. 
- The gated semantic fusion to address multiple overlapping instances is very useful for layout to image generation methods. 
- The proposed method is simple and is architecture agnostic.

weaknesses:
- Authors claim IFG as their first contribution. But this task has already been introduced in InstanceDiffusion [Wang et. al 2024c]. InstanceDiffusion uses detailed instance captions in addition to the global caption. What is the difference between their setup and IFG?
- The experimental section needs heavy work to make this work ready for publication. With exponential progress in generative modeling, it is hard to control the experimental settings with other models, especially the training data. But that doesn't mean all other settings can be held constant to properly understand the contributions. InstanceDiffusion and GLIGEN use SD1.5 as their base generation model but authors use SDXL an already powerful base generator. This makes it hard to understand the improvements in Table 1 and 2. I recommend authors report numbers with SD1.5 as the base generator or retrain InstanceDiffusion with SDXL (since their code is available) to properly support their claims.
- Authors introduce a new benchmark and evaluation metric for this task. Why can't they use the evaluation setup and metrics as InstanceDiffusion? If authors find flaws in InstanceDiffusion's setup, I recommend authors point it out and discuss the advantages of the IFG Benchmark (setup) and IFS Rate (metric). There is no point in creating multiple new benchmarks when existing ones are already rigorous. For IFS Rate authors use Grounding DINO whereas InstanceDiffusion uses YOLO. Please compare with InstanceDiffusion using their exact setup (COCO and LVIS val set and their metrics) to support your claims.
- Authors claim that the a lightweight network $f$ provides an "importance" score for location (x,y) in the ISM construction and use it to compute $D(x,y)$. Please show qualitative or quantitative evidence that the network $f$ infact does what is claimed in the paper. While the idea sounds reasonable, I suspect how $f$ learns to predict the right "importance" scores without supervision.

questions:
A few suggestions to improve the paper. 
- Please remove point 3 in contributions. Comprehensive experiments are not a contribution but are rather required to support the claims made in the paper. 
- The related works section has not been used correctly in this work according to my opinion. Authors just cite all relevant work but fail to differentiate their work from the literature. Please discuss how IFG is different from prior work and how their method is different from previously proposed methods in the related works section.
- If authors believe local CLIP score is suboptimal. I would recommend authors show (quantiatively) why VLMs are better than CLIP for this task. Please refrain from introducing a new metric and benchmark unless absolutely necessary.
- L77-78 is a repetition. 

I'm willing to improve my rating if authors address the weakness section satisfactorily.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730627880031

thread_type:
official_review

<thread_object>
creator: reviewer_v57v

<content>
summary:
This paper presents Instance Feature Adapter (IFA) for layout-to-image generation. The key insight of IFA is to incorporate additional appearance tokens corresponded to different objects, so as to steer the T2I models to generate images that convey precise layout information and text semantics. To achieve this, IFA first leverages learnable instance tokens to aggregate the textual and bounding box information of the specific objects. To cope with the feature leakage problems, IFA further introduce Instance Semantic Map strategy to reallocate the spatial area for different semantics, so as to alleviate the feature conflicts between different objects during external feature tokens injection process. A new benchmark is proposed, the visual improvement over different baselines is significant. Further, the proposed method is a plug-and-play module, which can be adapted to various community models.

strengths:
1. The paper is well written and easy to follow. 
2. The designed approaches efficiently incorporate the external object semantics and layout information into the generation process. The proposed Appearance Tokens aggregate the textual semantic and bbox information with learnable tokens. The proposed Instance Semantic Map accurately reallocates the spatial area of different objects, and solves the semantic fusion and feature leakage problem. 
3. The illustrated visual results are impressive, which shows clear superiority against competing baselines.
4. The proposed method is compatible with various community models.

weaknesses:
1. The design module is supposed to interface with the text encoders, and both Appearance Tokens and Instance Semantic Map introduce the attention mechanism. Will it be computational costly during the inference process. There should be a detailed discussion.
2. The size and shape of different objects seem unstable when applying IFA to different community models (Fig. 4), is it caused by the re-weight strategy from Instance Semantic Maps?
3. The L2I problem is not a novel task, and the main novelty mainly lies in the implementation detail of the layout incorporation strategy, which may not bring significantly inspirations to the community.

questions:
Is the proposed model able to generate images with some intricate prompts. For example, a blue dog with red legs running on a colorful river, (with dog, legs, river assigned to different boxes). I want to see some limitations of the proposed method, or in other words, I want to know how IFA copes with the semantic issues which may inherit from the base model given out-of-domain prompts and instructions. I would love to revise my rating after further discussion with the authors.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730680146653

thread_type:
official_review

<thread_object>
creator: reviewer_mfku

<content>
summary:
This paper aims to improve the capability of Text-to-Image diffusion models in generating precise features and positioning multiple instances in images. The proposed IFAdapter enhances feature accuracy by integrating additional appearance tokens and constructing an instance semantic map, ensuring that each instance's features align accurately with its spatial location. The IFAdapter is designed as a flexible, plug-and-play module, allowing enhanced control without retraining.

strengths:
+ The authors propose an important task, namely Instance Feature Generation. In addition, the authors also provide a benchmark and a verification pipeline
+ The proposed method seems to achieve good results. Qualitative and quantitative experiments demonstrate the effectiveness of the proposed method.

weaknesses:
-	Compared with instance diffusion and dense diffusion, this paper shows a small number of objects and does not show the situation where the object is denser.
-	The details of the method are not explained clearly. The details of the dataset construction and the baseline setting are not presented clearly. Ablation experiments lack a basic baseline presentation.
-	Authors should carefully check for errors in the text. For example, a sentence appears twice in succession in Introduction.

questions:
1. Why IFAdapter enables it to be seamlessly applied to various community models. For example, appearance queries are trained , how do the authors ensure that the feature has sufficient generalization capabilities.
2. Why can appearance-related features be extracted from the bounding box through Fourier and MLP, especially when the model inference cannot obtain image input?
3. Is the model-free method based on the same backbone as the proposed method? Are all the comparison methods that require training trained on the provided dataset? The VLM used to annotate the proposed dataset is the same as the VLM used in the evaluation metric, which may be unfair to methods that are not trained on the proposed dataset.
4. VLMs are likely to produce hallucinations. How do the authors ensure that the annotations of the provided dataset are free of hallucinations?
</content>
</thread_object>

</thread>


Metareview:
metareview:
Summary
This paper studies the problem of controllable image generation where the scene layout can be specified using a global caption, instance locations, and instance specific captions. The authors propose an adapter based technique called IFA that fuses such instance conditioning with a pretrained text to image backbone. The authors propose a benchmark, IFG, to evaluate their work.

Strengths
1. IFA is architecture agnostic which makes it generally and widely applicable to many text to image models.
2. The authors address a critical limitation of current methods that only use the last token from the text encoders to condition the image generation.
3. The method section in the paper is written well and explains the approach

Weaknesses
1. The stated contributions in the paper are inaccurate and required revisions after review. There are also factually vague statements (L143 in reference to conditioning on instance prompts which InstanceDiffusion does). Not having a clear sense of contributions in the paper is not acceptable.
2. The authors do not state any evidence for why IFG is a necessary benchmark compared to prior instance level generation benchmarks -- different metric, different VLM from prior work. They only talked about this difference in the rebuttal and not in the main paper. Given that this is a major contribution and the setting used for evaluating against all prior work, I believe this is a serious omission. The author rebuttal also does not answer this question raised by Nx7Q.
3. Comparisons in the main paper used a different text to image backbone across methods. This made it hard to make a fair comparison and assessment of improvements, if any. The authors did fix this in the rebuttal comments.

Justification
The paper's method section is well written. However, the exact contributions, relation to related work, reason for benchmark, experimental setup need revision (weaknesses above) and thus, it is better that this paper be resubmitted and go through the review process again.