IFAdapter: Instance feature control for grounded Text-to-Image Generation

## Abstract

Abstract While Text-to-Image (T2I) diffusion models excel at generating visually appealing images of individual instances, they struggle to accurately position and control the features generation of multiple instances. The Layout-to-Image (L2I) task was introduced to address the positioning challenges by incorporating bounding boxes as spatial control signals, but it still falls short in generating precise instance features.
In response, we propose the Instance Feature Generation (IFG) task, which aims to ensure both positional accuracy and feature fidelity in generated instances.
To address the IFG task, we introduce the Instance Feature Adapter (IFAdapter). The IFAdapter enhances feature depiction by incorporating additional appearance tokens and utilizing an Instance Semantic Map to align instance-level features with spatial locations. The IFAdapter guides the diffusion process as a plug-and-play module, making it adaptable to various community models.
For evaluation, we contribute an IFG benchmark and develop a verification pipeline to objectively compare models’ abilities to generate instances with accurate positioning and features.
Experimental results demonstrate that IFAdapter outperforms other models in both quantitative and qualitative evaluations.
Project Page: https://ifadapter.github.io/ .

## 1 Introduction

The advent of diffusion models has revolutionized the field of Text-to-Image (T2I) synthesis  (Ho et al., ; Podell et al., ; Baldridge et al., ; Betker et al., ; Rombach et al., ; Yang et al., ).
Despite their exceptional performance in generating high-quality images of single objects, these models remain limited in composing multiple objects into an exquisite image.
There are two key challenges underscore this limitation: 1) The inability of natural language in conveying precise spatial information impedes expression of user intent to the model, resulting in poor image composition in the generated images. 2) Relying solely on a given text prompt describing the attributes of multiple objects, existing models often fails to bind the detailed features to the correct object instances ().

Recent advancements in the Layout-to-Image (L2I) task (Li et al., ; Wang et al., ; Zhou et al., ; Kim et al., ; Bar-Tal et al., ) have partially mitigated such limitation and achieved precise instance-level position control by incorporating bounding boxes as spatial signals.
However, in terms of instance feature generation, most state-of-the-art (SOTA) L2I methods can only accurately depict coarse features of an instance (e.g., color attribution), while struggling to generate more complex, fine-grained features, as shown in Figure 1(a). This shortcoming limits the models’ applicability in scenarios such as graphic design and art design, where local high-grade details are essential. To simultaneously track the improvement of layout accuracy and feature generation accuracy, we introduce a more challenging task, termed Instance Feature Generation (IFG) task.
We found that existing T2I methods do not perform satisfactorily on the IFG task. Upon experiment and analysis, we observe that existing T2I methods do not perform satisfactorily on the IFG task and ascribe this phenomenon to two restrictions: 1) Insufficient detailed descriptions: Most L2I methods rely solely on category labels as descriptions for instances during training. This approach causes samples with detailed descriptions to become out-of-distribution during inference. 2) Insufficient feature information: Existing designs mostly use a single contextualized token to guide the feature generation of each instance. Although this token effectively captures the coarse semantics of the instance (Chen et al., ), it is limited in generating high-frequency appearance features.

In this work, we propose the Instance Feature Adapter (IFAdapter) to address the aforementioned restrictions.
First, to address issues related to the training data, we utilize existing SOTA Vision-Language Models (VLMs) for annotation, generating a dataset with detailed instance-level descriptions.
Subsequently, we implement two meticulously designed components to address the challenges of instance positioning and feature representation. 1) Appearance Tokens: To address the loss of detailed feature information in instances, the IFAdapter introduces novel learnable appearance queries. These queries extract instance-specific feature information from descriptions, forming appearance tokens that work alongside $EoT$ tokens, thereby enabling more precise control over the generation of instance features;
2) Instance Semantic Map: In contrast to sequence-to-2D grounding conditions (Li et al., ; Wang et al., ), IFAdapter constructs a 2D semantic map to correlate instance features with designated spatial locations. This map-like condition provides enhanced spatial guidance, reinforcing the spatial prior and preventing the leakage of instance features.
In regions where multiple instances overlap, a gated semantic fusion mechanism is employed to resolve feature confusion.
The IFAdapter integrates the semantic map only within a subset of cross-attention layers (Vaswani, ) in the diffusion model. This loose coupling allows the IFAdapter to function as a plug-and-play component, enabling its instance-level control capabilities to be transferred across various community models without requiring retraining, as illustrated in 1(b).

For evaluation, previous L2I benchmarks primarily focused on instance position accuracy, overlooking instance feature accuracy, which limits their ability to fully assess model performance on the IFG task. To address this limitation, we introduce the COCO-IFG benchmark, designed to evaluate models based on both positional accuracy and precise instance feature generation. Additionally, to overcome the limitations of existing object detection methods, which are incapable of detecting instance features, we integrate SOTA VLMs to facilitate instance feature detection, establishing an objective verification pipeline. Comprehensive experiments on the benchmark demonstrate that IFAdapter significantly enhances instance feature generation accuracy while maintaining precise position accuracy.

The contributions of this work are as follows:

- 1.
We propose the Instance Feature Generation task to address the challenges of positional and feature accuracy in multi-instance generation using diffusion models. In addition, we introduce the COCO IFG benchmark and verification pipeline to evaluate and compare model performance.
- 2.
We propose IFAdapter, which utilizes novel appearance tokens and instance semantic map to enhance diffusion models’ depiction of instances, enabling high-fidelity instance feature generation.
- 3.
Comprehensive experiments demonstrate that our model outperforms the baselines in both quantitative and qualitative evaluations.
- 4.
The IFAdapter is designed as a plug-and-play component, enabling it to seamlessly empower various community models with layout control capabilities without retraining.

## 2 Related Work

Layout-to-Image Generation.
In the early stages, Layout-to-Image (Layout-to-Image) works primarily hinged on Generative Adversarial Networks (GANs) (Sun & Wu, ; ; Li et al., ; He et al., ; Wang et al., ; Sylvain et al., ).
Novel modules and techniques have been proposed to address specific challenges in existing methods, such as object-to-object relations (He et al., ; Sylvain et al., ), object appearance (Sun & Wu, ; He et al., ), and handling interactions between bounding boxes (Sylvain et al., ; Li et al., ; Wang et al., ).
Nevertheless, withthe rising tide of diffusion-based methods in the generative field, incorporating diffusion techniques into Layout-to-Image methods (Cheng et al., ; Zheng et al., ; Li et al., ; Wang et al., ; Zhou et al., ; ; Xie et al., ; Xiao et al., ; Chen et al., ; Yang et al., ; Avrahami et al., ) has led to significant improvements in the quality, diversity, and controllability of generated images.
LayoutDiffusion (Zheng et al., ) constructs a structural image patch with region information and transforms it into a unified form fused with the layout.
LayoutDiffuse (Cheng et al., ) proposed an adapter based on layout attention and task-aware prompts.
MIGC (Zhou et al., ; ) utilizes an instance enhancement attention mechanism for precise shading.
GLIGEN (Li et al., ) and InstanceDiffusion (Wang et al., ) pioneer Layout-to-Image generation of open-sets by using the conceptual knowledge of SD pretrained models to inject object locations and appearance into features through custom attention.
Some works explore the training-free method for Layout-to-Image generation by guiding the attention maps of the diffusion model during the generative process (Xiao et al., ; Chen et al., ) or implementing spatial constraints in the denoising process (Xie et al., ).

Controllable Diffusion Models.
The emergence of diffusion models has significantly propelled advancements in the field of image generation.
Controllable Diffusion Models utilize a wide variety of control conditions to generate images with specific content, leading to a proliferation of applications.
Semantic control enables precise manipulation of image attributes or features in the generation process by referencing text (Rombach et al., ; Saharia et al., ; Ramesh et al., ; Chen et al., ) or images (Tang et al., ; Saharia et al., ).
Spatial control provides fine-grained control over the content in specific regions, such as segmentation-guided (Bar-Tal et al., ; Couairon et al., ; Wu et al., ), sketch-guided (Voynov et al., ), and depth-guided methods (Kim et al., ).
Recent efforts have concentrated on integrating these spatial control conditions into a unified framework for text-to-image generation, including approaches such as ControlNet (Zhang et al., ; Zhao et al., ), Composer (Huang et al., ), and Adapter-based (Mou et al., ) methods.
ID and style control emphasize maintaining the consistency of user-specified identity or style in generated images, tuning-based methods guide diffusion models to generate the specified content by fine-tuning (Hu et al., ; Ruiz et al., ), while tuning-free methods (Ye et al., ; Huang et al., ; Wang et al., ; Li et al., ; Hertz et al., ; Wang et al., ) injecting coded condition embedding in the denoising process.

## 3 Approach

### 3.1 Preliminaries

Latent Diffusion Model.
Our method is applied over a pretrained T2I diffusion model, more specifically, a T2I latent diffusion model (LDM) (Rombach et al., ).
The generation process of the LDM can be regarded as stepwise denoising from a initial Gaussian noise $z\sim\mathcal{N}(0,I)$ , conditioned on a textual prompt $y$. The training objective is to minimize the following LDM loss:

$$ $\mathcal{L}_{LDM}=\mathbb{E}_{z\sim\mathcal{N}(0,I),y,t}[||\epsilon-\epsilon_{ \theta}(z_{t},t,E(y))||^{2}_{2}],$ (1) $$

where the $\epsilon_{\theta}$ is parameterized as a UNet (Ronneberger et al., ) and $t$ is the denoising timestep. $E$ is a pretrained text encoder, used to encode $y$ into text embeddings.

Cross Attention.
In the LDM, text embeddings guide the direction of generation via cross attention (Vaswani, ), which can be represented using the following equation:

$$ $\text{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V},\mathbf{M})=\text{Softmax}( \frac{\mathbf{Q}\mathbf{K}^{\top}}{\sqrt{d}}+\mathbf{M})\mathbf{V},$ (2) $$

the $Q$ represents the image latent code obtained after being projected through a MLP (Multi-Layer Perceptron), while $K$ and $V$ are similarly derived from text embeddings. $\mathbf{M}$ is a mask used to adjust attention scores, and $d$ represents the dimensionality of the hidden vector, which helps stabilize the training process.

### 3.2 Problem Definition

In the Instance Feature Generation task, the LDM requires additional conditioning on a set of local descriptors $c=\{(\mathbf{r}_{1},\mathbf{l}_{1}),\ldots,(\mathbf{r}_{n},\mathbf{l}_{n})\}$.
$r_{i}$ represents the designated generation position for the $i$-th instance, in $[{x,y,w,h}]$ form. $l_{i}$ is the corresponding phrase that describes the features of the $i$-th instance.
Our method differs from others in that $l_{i}$ incorporates detailed, extended descriptions of the instance, including aspects such as mixed colors, complex textures. With $c$ serving as auxiliary conditions, the LDM should be able to generate instances with high fidelity in both position and features.

### 3.3 IFAdapter

In this work, the IFAdapter is designed to control the generation of instance features and positions. We employ the open-source Stable Diffusion (SD) (Rombach et al., ; Podell et al., ) as the base model. To address the issue of instance feature loss, we introduce appearance tokens as a supplement to the high-frequency information, as discussed in Sec. 3.3.1.
Furthermore, to incorporate a stronger spatial prior for more accurate control over position and features, we use appearance tokens to construct an instance semantic map that guides the generation process, as elaborated in Sec. 3.3.2.

Figure: Figure 2: Structure of proposed IFAdapter. (a) The generation process of appearance tokens. For simplicity, we use the generation process of one instance (the corgi) as example. (b) The construction process of the Instance Semantic Map.
Refer to caption: x2.png

#### 3.3.1 Appearance Tokens

L2I SD enables the generation of grounded instances by incorporating local descriptions and location as additional conditions. Existing approaches (Li et al., ; Zhou et al., ; Wang et al., ) typically utilize the contextualized token (the End of Text, $EoT$ token) produced by the pretrained CLIP text encoder (Radford et al., ) to guide the generation of instance features.
Although the $EoT$ token plays a crucial role in foreground generation, it is primarily used to generate coarse structural content (Wu et al., ; Chen et al., ) and requires additional tokens to complement high-frequency details.
As a result, existing L2I methods that discard all other tokens are unable to generate detailed instance features.
One naive mitigation approach would be to use all tokens (77 in total) generated by the CLIP text encoder as instance-level conditions. However, this method would significantly increase the memory requirements during both inference and training. Moreover, these 77 tokens include a substantial amount of padding tokens, which do not contribute to the generation.
While removing padding tokens can reduce computational costs, this strategy is incompatible with batch training due to the varying lengths of the descriptions.
To address this, we propose compressing the feature information into a small set of appearance tokens and utilizing these tokens to complement the $EoT$ token.

Drawing inspiration from the Perceiver design (Ye et al., ; Alayrac et al., ), we employ a set of learnable appearance queries to interact with instance description embeddings through cross attention, thereby extracting feature information and forming appearance tokens, as shown in Fig. 2 (a).
It is worth noting that the appearance queries only interact with word tokens, thus converting descriptions of arbitrary length into fixed-length appearance tokens. In addition, to obtain text features of different entangled granularities, the query tokens also interact with text features in the shallower layers of the text encoder.
By combining the instance’s appearance tokens with their corresponding location embeddings, appearance tokens $\mathbf{h^{l}}\in\mathbb{R}^{L\times d}$ are obtained from layer $l$, where $L$ denotes the number of appearance tokens.
This process can be expressed using the following formula:

$$ $\displaystyle\mathbf{H}$ $\displaystyle=[\mathbf{h^{l_{1}}},\dots,\mathbf{h^{l_{k}}}]$ (3) $\displaystyle\text{where}\quad\mathbf{h^{l}}$ $\displaystyle=\text{Resampler}(\mathbf{Q_{a}},\mathbf{K^{l}},\mathbf{V^{l}})+ \text{MLP}(\text{Fourier}(\boldsymbol{r})).$ $$

For the sake of clarity, we use the generation of appearance tokens for a single instance as an example. The Resampler is adapted from Perceiver, composed of multiple transformer blocks. $Q_{a}$ represents the appearance queries, while $\mathbf{K^{l}}$ and $\mathbf{V^{l}}$ are obtained by projecting the text features extracted from the $l$-th layer of the text encoder. The Fourier is the Fourier embedding (Mildenhall et al., ), combined with a MLP to project $r$ to the feature dimension. Finally, the appearance tokens at $k$ different granularities are concatenated into $\mathbf{H}\in\mathbb{R}^{(kL)\times d}$ to serve as the generation guidance for each instance.

#### 3.3.2 Instance Semantic Map-Guided Generation

Along with ensuring the generation of detailed instance features, the IFG task also requires instances to be generated at designated locations. Previous method (Li et al., ) uses sequential grounding tokens as conditions, which lack robust spatial correspondence, potentially leading to issues such as feature misplacement and leakage.
Therefore, in our work, we introduce a map called the Instance Semantic Map (ISM) as a stronger guiding signal.
Since the generation of all instances is guided by the ISM, two major considerations must be addressed when constructing the map: (1) generating detailed and accurate features for each instance while avoiding feature leakage, and (2) managing overlapping regions where multiple instances are present. To address these concerns, we first generate each instance in isolation and then aggregate them in the overlapping regions. The following sections will provide a detailed explanation of these processes.

Per-instance Feature Generation.
Avoiding interference from extraneous features is crucial for the precise generation of high-quality instance details.
To achieve this objective, we first generate the semantic map of each instance individually. Specifically, for the $i$-th instance, we transform its corresponding location $\mathbf{r}_{i}$ into the following mask $\mathbf{m}_{i}$:

$$ $\left.\mathbf{m}_{i}(x,y)=\left\{\begin{matrix}0&\mathrm{if~{}}[x,y]\in \mathcal{R}_{i}\\ -\infty&\mathrm{if~{}}[x,y]\notin\mathcal{R}_{i}\end{matrix}\right.\right.,$ (4) $$

where $\mathcal{R}_{i}$ represents the coordinates within the region indicated by $\mathbf{r}_{i}$. By employing Eq. 2, we can obtain the semantic map $s_{i}$ for the $i$-th instance:

$$ $s_{i}=\text{Attention}(\mathbf{Q},\mathbf{K_{i}},\mathbf{V_{i}},\mathbf{m_{i}}),$ (5) $$

where $K_{i}$ and $V_{i}$ are projected from the concatenation of the appearance tokens $\mathbf{H}$ and $EoT$ token of $i$-th instance, the Q is derived from the image latent code.

Gated Semantic Fusion.
After obtaining the semantic maps for each instance, the next step is to blend these maps to derive the final ISM, as shown in Fig. 2 (b).
A critical issue to consider during the map integration process is how to represent the features of a latent pixel when it is associated with multiple instances. Previous method (Jia et al., ) average the features of multiple instances. While this approach is simple, it may lead to feature conflicts between different instances. Intuitively, the visual features in regions where multiple instances overlap should be dominated by the instance closest to the observer (i.e., the one with the smallest depth). Therefore, the weights of different instances in overlapping regions should vary.
For clarity, we use the feature integration at pixel location $(x,y)$ as an example. The features of each instance are first projected into a scalar representing importance through a trainable lightweight network $f$. Then, the Softmax operation normalizes the importance across different instances, yielding their respective weights. This process can be described by the following equation:

$$ $[w_{1}{(x,y)},\dots,w_{n}{(x,y)}]=\text{Softmax}(f(s_{1}(x,y)),\dots,f(s_{n}(x ,y))),$ (6) $$

where $w_{i}{(x,y)}$ denotes the weight of instance $i$ at location $(x,y)$.

In addition to the instance feature, the size of the instance also influences its weight. This design is motivated by the following consideration: when the region of a small instance is completely covered by a larger instance, it is necessary to prevent the smaller instance from being “assimilated” due to the inclusion of excessive irrelevant features.
Therefore, the proportion of the area occupied by the instance in the foreground is also considered, with smaller instance being assigned greater weight.
Using the instance features and their respective weights, the final representation for a latent pixel position $(x,y)$ is obtained using the following formula:

$$ $\mathbf{D}(x,y)=\sum_{i}w_{i}(x,y)\cdot\text{Sigmoid}(\frac{|\bigcup_{j}^{n}a_ {j}|}{|a_{i}|})\cdot s_{i}(x,y),$ (7) $$

where $a_{i}$ represents the area occupied by instance $i$. After the aforementioned steps, the ISM is constructed.
Finally, ISM interacts through the following duplicate cross attention layers (Ye et al., ) to guide the generation of salient regions:

$$ $\text{Attn}=\text{Attention}(\mathbf{Q},\mathbf{K},\mathbf{V},0)+\text{tanh}( \lambda)\cdot(1-\mathcal{M}_{bg})\odot\mathbf{D},$ (8) $$

where $\mathcal{M}_{bg}$ is a binary mask with the background area set to 1, and $\lambda$ is a trainable parameter initialized to 0 to prevent pattern collapse during the initial training phase.

### 3.4 Learning Procedure

During training, we freeze the parameters of the SD, training only the IFAdapter. The loss function used for training is the LDM loss with instance-level condition incorporated:

$$ $\mathcal{L}_{IFA}=\mathbb{E}_{z\sim\mathcal{N}(0,I),y,t}[||\epsilon-\epsilon_{ \theta}(z_{t},t,E(y)),c||^{2}_{2}]$ (9) $$

To enable our method to perform classifier-free guidance (CFG) (Ho & Salimans, ) during the inference phase, we randomly set the global condition $y$ and local condition $c$ to 0 during training.

## 4 Experiments

### 4.1 Implementation Details

We described the basic setup for training our model. For more details, please refer to the appendix.

Training dataset.
We use the COCO2014 (Lin et al., ) dataset and a 1 million subset from LAION 5B (Schuhmann et al., ) as our data sources. Following previous methods (Wang et al., ; Zhou et al., ), we utilize Grounding-DINO (Liu et al., ) and RAM (Zhang et al., ) to annotate the instance positions within the images. We then employ the state-of-the-art visual language models (VLMs) QWen (Bai et al., ) and InternVL (Chen et al., ) to generate captions for the images and individual instance.
Training details. We use SDXL (Podell et al., ), known for its strong detail generation capabilities, as our base model. The IFAdapter is applied to a subset of SDXL’s mid-layers and decoder layers, which significantly contribute to foreground generation. We trained the IFAdapter using the AdamW (Loshchilov et al., ) optimizer with a learning rate of 0.0001 for 100,000 steps and a batch size of 160. During training, there was a 15% chance of dropping the local description and a 30% chance of dropping the global caption. For inference, we used the EulerDiscreteScheduler (Karras et al., ) with 30 sample steps and set the classifier-free guidance (CFG) scale to 7.5.

### 4.2 Experimental Setup

Baselines. We compared our approach with previous SOTA L2I methods, including training-based methods InstanceDiffusion (Wang et al., ), MIGC (Zhou et al., ), and GLIGEN (Li et al., ), as well as the training-free methods DenseDiffusion (Kim et al., ) and MultiDiffusion (Bar-Tal et al., ).

Evaluation dataset.
Following the previous setup (Li et al., ; Zhou et al., ; Wang et al., ), we constructed the COCO IFG benchmark on the standard COCO2014 dataset. Specifically, we annotate the locations and local descriptions in the validation set using the same approach as in the training data. Each method is required to generate 1,000 images for validation.

Evaluation Metrics.
For the validation of the IFG task, it is imperative that the model generates instances with accurate features at the appropriate locations.

- •
Instance Feature Success Rate. To verify spatial accuracy and description-instance consistency, we propose the Instance Feature Success (IFS) rate. The calculation of the IFS rate involves two steps. Step 1, Spatial accuracy verification: We begin by using Grounding-DINO to detect the positions of each instance. Next, we compute the Intersection over Union (IoU) between the detected positions and the Ground Truth (GT) positions, selecting the GT with the highest IoU as the corresponding match for that instance. If the highest IoU is less than 0.5, the instance generation is considered unsuccessful.
Step 2, Local feature accuracy verification: Previous methods (Avrahami et al., ; Zhou et al., ) primarily employ local CLIP for verifying local features. However, CLIP focuses on overall semantics and is not well-suited for capturing fine visual details (Yuksekgonul et al., ). Therefore, we utilize VLMs in conjunction with the prompt engineering technique to achieve more precise verification of local details. For each local region identified in Step 1, we prompt the VLMs to determine whether the content within the cropped region aligns with the corresponding description. If the VLM confirms that the content matches the prompt, the instance is marked as successful. The Instance Foreground Success (IFS) rate is then calculated as the ratio of successful instances to the total number of instances. Additionally, we report the Grounding-DINO Average Precision (AP) score to independently validate the positional accuracy of instance location generation.
- •
Fréchet Inception Distance (FID). FID (Heusel et al., ) measures image quality by calculating the feature similarity between generated and real images. We compute the FID using the validation set of COCO2017.
- •
Global CLIP Score.
The global caption of the image primarily describes the overall semantics of the image. Therefore, we use the CLIP score to evaluate Image-Caption Consistency.

### 4.3 Comparison

#### 4.3.1 Quantitative analysis.

**Table 1: Evaluation on COCO IFG benchmark. To perform a more rigorous and comprehensive experiment for calculating the IFS rate, we utilize three different VLMs. For spatial accuracy, we report the Grounding-DINO AP. To assess overall image quality, we measure the CLIP score and FID. The $\uparrow$ indicates that a higher value is better, while $\downarrow$ signifies the opposite.**
| Methods | IFS Rate(%) | Spatial(%) | Quality |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| QwenVL $\uparrow$ | InternVL $\uparrow$ | CogVL $\uparrow$ | AP $\uparrow$ | CLIP $\uparrow$ | FID $\downarrow$ |  |
| Real images | 92.8 | 82.2 | 69.9 | 75.3 | - | - |
| InstanceDiffusion | 69.6 | 49.7 | 38.2 | 43.1 | 23.3 | 26.8 |
| GLIGEN | 44.8 | 25.8 | 17.5 | 18.4 | 23.5 | 29.7 |
| MIGC | 62.8 | 40.7 | 27.5 | 32.5 | 22.9 | 26.0 |
| MultiDiffuion | 58.1 | 47.0 | 34.2 | 36.9 | 22.8 | 28.3 |
| DenseDiffusion | 38.7 | 26.0 | 19.7 | 22.2 | 20.1 | 29.9 |
| Ours | 79.7 | 68.6 | 61.0 | 49.0 | 25.1 | 22.0 |

Tab. 1 presents our qualitative results on the IFG benchmark, including metrics of IFS Rate, Spatial accuracy, and the Image Quality.

IFS Rate. To calculate the IFS rate, we utilize three state-of-the-art (SOTA) vision-language models (VLMs): QWenVL (Bai et al., ), InternVL (Chen et al., ), and CogVL (Wang et al., ). This multi-model approach ensures a more comprehensive and rigorous validation. As shown in Tab 1, our model outperforms the baseline models in all three IFS rate metrics. The introduction of appearance tokens and the incorporation of dense instance descriptions in training have significantly enhanced our model’s ability to generate accurate instance details. It is worth noting that InstanceDiffusion achieves a higher IFS rate compared to other baselines. This is likely due to its training dataset also contains dense instance-level descriptions. This observation further underscores the necessity of high-quality instance-level annotations.

Spatial Accuracy. As can be observed from Tab 1, IFAdapter achieves the best results in Grounding-DINO AP. This success can be attributed to our map-guided generation design, which incorporates additional spatial priors, leading to more accurate generation of instance locations.

Figure: Figure 3: Qualitative results. We compare the models’ ability to generate instances with different types of features, including mixed colors, varied materials, and intricate textures.
Refer to caption: x3.png

Image Quality. As shown in Table 1, our method demonstrates a higher CLIP Score, indicating that enhancing local details contributes to the simultaneous improvement of image-caption consistency. Additionally, our method achieves a lower FID, suggesting that the images generated by our approach are of higher quality compared to the baselines. We attribute this improvement to the adapter-like design of our model, which enables spatial control without significantly compromising image quality.

#### 4.3.2 Qualitative analysis.

In Fig. 1(a), we present generation results for a scene with multiple complex instances. We further evaluate the models’ ability to generate instances with diverse features in Fig. 3. As shown, our method demonstrates the highest level of fidelity across various types of instance details.

### 4.4 User Study.

**Table 2: Results of user study. We conducted a user study to evaluate the spatial generation accuracy, instance detail generation effectiveness, and aesthetic index of the L2I methods. Evaluators were provided with the image layout and the corresponding image, and they were asked to rate the aforementioned three dimensions on a scale of 0 to 5. A score of 0 represents the lowest rating, while 5 represents the highest rating. We also reported the user preference rate (Pref. Rate), which represents the proportion of the highest scores obtained by the methods.**
| Methods | Spatial | Instance Details | Aesthetics |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Score $\uparrow$ | Pref. Rate $\uparrow$ | Score $\uparrow$ | Pref. Rate $\uparrow$ | Score $\uparrow$ | Pref. Rate $\uparrow$ |  |
| InstanceDiffusion | 4.44 | 44.4% | 3.82 | 33.3% | 2.99 | 14.8% |
| GLIGEN | 3.96 | 14.2% | 2.54 | 3.7% | 2.44 | 3.7% |
| MIGC | 4.30 | 33.3% | 3.39 | 7.4% | 2.54 | 3.7% |
| Ours | 4.85 | 88.9% | 4.69 | 88.9% | 4.10 | 96.2% |

Although VLMs can verify instance details to a certain extent, a gap remains compared to human perception. Therefore, we invited professional annotators for further validation.

Setup. We conducted a study comprising 270 questions, each associated with a randomly sampled generated image. Evaluators were asked to rate image quality, instance location accuracy, and instance details. In total, 30 valid responses were collected, yielding 7,290 ratings.

Results. As seen in Tab. 2, our method achieves the highest scores and user preference rate across all three dimensions. Notably, the trends in these dimensions are consistent with those in Table 1, further demonstrating the effectiveness of VLM validation.

### 4.5 Integration with Community Models

Figure: Figure 4: The IFAdapter can seamlessly integrate with community diffusion models.
Refer to caption: x4.png

Thanks to the plug-and-play design of the IFAdapter, it can impose spatial control on pretrained diffusion models without significantly compromising the style or quality of the generated images. This capability enables the IFAdapter to be effectively integrated with various community diffusion models and LoRAs (Hu et al., ).
As illustrated in Fig. 4, we applied IFAdapter to several community models, including PixlArt (NeriJS, ), LeLo-LEGO (LordJia, ), Claymation (DoctorDiffusion, ), and BluePencil (blue_pen5805, ). The generated images not only adhere to the specified layouts but also accurately reflect the respective styles.

### 4.6 Ablation Study

In our method, appearance tokens are introduced to address the shortcomings of $EoT$ tokens in generating high-frequency details. This ablation study primarily explores the roles of these two token types in instance generation.

Figure: Figure 5: Qualitative results of variants of IFAdapter.
Refer to caption: x5.png

appearance tokens. As observed in Tab. 3, the removal of appearance tokens leads to a decrease in the model’s IFS rate and FID, indicating a loss of detailed features. Furthermore, as illustrated in Fig. 5, the images generated without appearance tokens exhibit instance feature mismatches, further demonstrating that appearance tokens are primarily responsible for generating high-frequency appearance features.

$EoT$ token. As shown in Table 5, the IFS rate significantly decreases when generating without the $EoT$ token. This is primarily because the $EoT$ token is responsible for generating the coarse semantics of instances. Additionally, Fig. 5 indicates that removing the $EoT$ token results in semantic-level issues, such as instance category errors and instance omissions.

**Table 3: Quantitative results of variants of IFAdapter.**
| appearance tokens | EoT token | IFS Rate(%) | Spatial(%) | Quality |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| QwenVL $\uparrow$ | InternVL $\uparrow$ | CogVL $\uparrow$ | AP $\uparrow$ | CLIP $\uparrow$ | FID $\downarrow$ |  |  |
|  | ✓ | 69.6 | 63.9 | 53.5 | 45.9 | 24.1 | 27.2 |
| ✓ |  | 29.9 | 16.2 | 12.0 | 12.3 | 24.3 | 44.7 |
| ✓ | ✓ | 79.7 | 68.6 | 61.0 | 49.0 | 25.1 | 22.0 |

## 5 Conclusion

In this work, we propose IFAdapter to exert fine-grained, instance-level control on pretrained Stable Diffusion models. We enhance the model’s ability to generate detailed instance features by introducing Appearance Tokens. By utilizing Appearance Tokens to construct an instance semantic map, we align instance-level features with spatial locations, thereby achieving robust spatial control. Both qualitative and quantitative results demonstrate that our method excels in generating detailed instance features. Furthermore, due to its plug-and-play nature, IFAdapter can be seamlessly integrated with community models as a plugin without the need for retraining.

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Why IFAdapter enables it to be seamlessly applied to various community models. For example, appearance queries are trained , how do the authors ensure that the feature has sufficient generalization capabilities.
2. Why can appearance-related features be extracted from the bounding box through Fourier and MLP, especially when the model inference cannot obtain image input?
3. Is the model-free method based on the same backbone as the proposed method? Are all the comparison methods that require training trained on the provided dataset? The VLM used to annotate the proposed dataset is the same as the VLM used in the evaluation metric, which may be unfair to methods that are not trained on the proposed dataset.
4. VLMs are likely to produce hallucinations. How do the authors ensure that the annotations of the provided dataset are free of hallucinations?
soundness: 3
strengths: + The authors propose an important task, namely Instance Feature Generation. In addition, the authors also provide a benchmark and a verification pipeline
+ The proposed method seems to achieve good results. Qualitative and quantitative experiments demonstrate the effectiveness of the proposed method.
summary: This paper aims to improve the capability of Text-to-Image diffusion models in generating precise features and positioning multiple instances in images. The proposed IFAdapter enhances feature accuracy by integrating additional appearance tokens and constructing an instance semantic map, ensuring that each instance's features align accurately with its spatial location. The IFAdapter is designed as a flexible, plug-and-play module, allowing enhanced control without retraining.
weaknesses: -	Compared with instance diffusion and dense diffusion, this paper shows a small number of objects and does not show the situation where the object is denser.
-	The details of the method are not explained clearly. The details of the dataset construction and the baseline setting are not presented clearly. Ablation experiments lack a basic baseline presentation.
-	Authors should carefully check for errors in the text. For example, a sentence appears twice in succession in Introduction.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Is the proposed model able to generate images with some intricate prompts. For example, a blue dog with red legs running on a colorful river, (with dog, legs, river assigned to different boxes). I want to see some limitations of the proposed method, or in other words, I want to know how IFA copes with the semantic issues which may inherit from the base model given out-of-domain prompts and instructions. I would love to revise my rating after further discussion with the authors.
soundness: 4
strengths: 1. The paper is well written and easy to follow. 
2. The designed approaches efficiently incorporate the external object semantics and layout information into the generation process. The proposed Appearance Tokens aggregate the textual semantic and bbox information with learnable tokens. The proposed Instance Semantic Map accurately reallocates the spatial area of different objects, and solves the semantic fusion and feature leakage problem. 
3. The illustrated visual results are impressive, which shows clear superiority against competing baselines.
4. The proposed method is compatible with various community models.
summary: This paper presents Instance Feature Adapter (IFA) for layout-to-image generation. The key insight of IFA is to incorporate additional appearance tokens corresponded to different objects, so as to steer the T2I models to generate images that convey precise layout information and text semantics. To achieve this, IFA first leverages learnable instance tokens to aggregate the textual and bounding box information of the specific objects. To cope with the feature leakage problems, IFA further introduce Instance Semantic Map strategy to reallocate the spatial area for different semantics, so as to alleviate the feature conflicts between different objects during external feature tokens injection process. A new benchmark is proposed, the visual improvement over different baselines is significant. Further, the proposed method is a plug-and-play module, which can be adapted to various community models.
weaknesses: 1. The design module is supposed to interface with the text encoders, and both Appearance Tokens and Instance Semantic Map introduce the attention mechanism. Will it be computational costly during the inference process. There should be a detailed discussion.
2. The size and shape of different objects seem unstable when applying IFA to different community models (Fig. 4), is it caused by the re-weight strategy from Instance Semantic Maps?
3. The L2I problem is not a novel task, and the main novelty mainly lies in the implementation detail of the layout incorporation strategy, which may not bring significantly inspirations to the community.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: A few suggestions to improve the paper. 
- Please remove point 3 in contributions. Comprehensive experiments are not a contribution but are rather required to support the claims made in the paper. 
- The related works section has not been used correctly in this work according to my opinion. Authors just cite all relevant work but fail to differentiate their work from the literature. Please discuss how IFG is different from prior work and how their method is different from previously proposed methods in the related works section.
- If authors believe local CLIP score is suboptimal. I would recommend authors show (quantiatively) why VLMs are better than CLIP for this task. Please refrain from introducing a new metric and benchmark unless absolutely necessary.
- L77-78 is a repetition. 

I'm willing to improve my rating if authors address the weakness section satisfactorily.
soundness: 2
strengths: - The perceiver like Resampler design to extract fixed set of appearance tokens is novel and effective. This addresses the problem of only utilizing the EoT token from the text encoders. 
- The gated semantic fusion to address multiple overlapping instances is very useful for layout to image generation methods. 
- The proposed method is simple and is architecture agnostic.
summary: - This work tackles instance Instance Feature Generation (IFG) task, i.e. train a model that can generate images given a global caption, spatial locations and detailed local captions as conditioning.
- Authors introduce IFAdapter, a plug-and-play module for improving IFG.
- The IFAdapter first extracts a fixed number of appearance tokens from a detailed instance caption and its spatial location. Next, a 2D map called Instance Semantic Map (ISM) is constructed using the bounding boxes of instances to aid the adherence of the model to spatial location conditions.
- IFAdapter is architecture agnostic and can be incorporated into existing open-source text to image models and authors show its effectiveness on the newly introduced IFG Benchmark constructed from the COCO dataset.
weaknesses: - Authors claim IFG as their first contribution. But this task has already been introduced in InstanceDiffusion [Wang et. al 2024c]. InstanceDiffusion uses detailed instance captions in addition to the global caption. What is the difference between their setup and IFG?
- The experimental section needs heavy work to make this work ready for publication. With exponential progress in generative modeling, it is hard to control the experimental settings with other models, especially the training data. But that doesn't mean all other settings can be held constant to properly understand the contributions. InstanceDiffusion and GLIGEN use SD1.5 as their base generation model but authors use SDXL an already powerful base generator. This makes it hard to understand the improvements in Table 1 and 2. I recommend authors report numbers with SD1.5 as the base generator or retrain InstanceDiffusion with SDXL (since their code is available) to properly support their claims.
- Authors introduce a new benchmark and evaluation metric for this task. Why can't they use the evaluation setup and metrics as InstanceDiffusion? If authors find flaws in InstanceDiffusion's setup, I recommend authors point it out and discuss the advantages of the IFG Benchmark (setup) and IFS Rate (metric). There is no point in creating multiple new benchmarks when existing ones are already rigorous. For IFS Rate authors use Grounding DINO whereas InstanceDiffusion uses YOLO. Please compare with InstanceDiffusion using their exact setup (COCO and LVIS val set and their metrics) to support your claims.
- Authors claim that the a lightweight network $f$ provides an "importance" score for location (x,y) in the ISM construction and use it to compute $D(x,y)$. Please show qualitative or quantitative evidence that the network $f$ infact does what is claimed in the paper. While the idea sounds reasonable, I suspect how $f$ learns to predict the right "importance" scores without supervision.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Please kindly provide visual comparisons between the ground truth (GT) and the generated images.
soundness: 3
strengths: 1. A plug-and-play component that can be integrated with various existing models to enhance layout control capabilities.
2. The paper also includes a user study, offering valuable insights into the proposed method.
summary: The authors aim to tackle the challenge of achieving controllability in generating precise instance features. To do this, they introduce the Instance Feature Adapter (IFAdapter), designed for instance-level positioning and feature representation. Specifically, the IFAdapter employs learnable appearance queries that extract instance-specific feature information from descriptions, creating appearance tokens that complement EoT tokens. Additionally, the IFAdapter constructs a 2D semantic map to link instance features with designated spatial locations, providing enhanced spatial guidance. In areas where multiple instances overlap, a gated semantic fusion mechanism is utilized to mitigate feature confusion.

To validate their approach, the authors have created a new dataset, referred to as the COCO IFG benchmark. They leverage existing state-of-the-art Vision Language Models (VLMs) for annotation, resulting in a dataset with detailed instance-level descriptions. Experimental results indicate that the proposed plug-and-play component surpasses baseline models in both quantitative and qualitative assessments.
weaknesses: 1. The quality of the proposed dataset has not been evaluated. It appears that all ground truths (GTs) are generated by existing Vision Language Models (VLMs). A human-level quality assessment would be beneficial for greater impact within the community.
2. The assertion that the proposed component can “seamlessly empower various community models with layout control capabilities without retraining” (l.113) may be misleading. The IFAdapter is fundamentally a training-based method, and the phrase “without retraining” only holds true when applied to spaces closely aligned with COCO IFG, as the IFAdapter does not demonstrate zero-shot capabilities in this paper.
3. The semantic-instance map does not appear to be novel. Please refer to "BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion" (ICCV 2023) and other zero-shot L2I methods for comparison.
4. The appearance tokens show only minor improvements in Table 3. Additional explanations regarding this observation would be appreciated.

Meta-review: Summary
This paper studies the problem of controllable image generation where the scene layout can be specified using a global caption, instance locations, and instance specific captions. The authors propose an adapter based technique called IFA that fuses such instance conditioning with a pretrained text to image backbone. The authors propose a benchmark, IFG, to evaluate their work.

Strengths
1. IFA is architecture agnostic which makes it generally and widely applicable to many text to image models.
2. The authors address a critical limitation of current methods that only use the last token from the text encoders to condition the image generation.
3. The method section in the paper is written well and explains the approach

Weaknesses
1. The stated contributions in the paper are inaccurate and required revisions after review. There are also factually vague statements (L143 in reference to conditioning on instance prompts which InstanceDiffusion does). Not having a clear sense of contributions in the paper is not acceptable.
2. The authors do not state any evidence for why IFG is a necessary benchmark compared to prior instance level generation benchmarks -- different metric, different VLM from prior work. They only talked about this difference in the rebuttal and not in the main paper. Given that this is a major contribution and the setting used for evaluating against all prior work, I believe this is a serious omission. The author rebuttal also does not answer this question raised by Nx7Q.
3. Comparisons in the main paper used a different text to image backbone across methods. This made it hard to make a fair comparison and assessment of improvements, if any. The authors did fix this in the rebuttal comments.

Justification
The paper's method section is well written. However, the exact contributions, relation to related work, reason for benchmark, experimental setup need revision (weaknesses above) and thus, it is better that this paper be resubmitted and go through the review process again.

Decision: Reject