Hummingbird: High Fidelity Image Generation via Multimodal Context Alignment
Authors: Minh-Quan Le
Gaurav Mittal
Tianjian Meng
A S M Iftekhar
Vishwas Suryanarayanan
Barun Patra
Dimitris Samaras
Mei Chen
Venue: ICLR 2025 Poster (2025)
Keywords: multimodal
diffusion model
image generation
lora
mllm
stable diffusion
mme
hoi
tta
Primary area: generative models
TL;DR: We introduce Hummingbird, the first diffusion-based image generator to synthesize high fidelity images guided by multimodal context of reference image and text guidance while maintaining high diversity.

Abstract: While diffusion models are powerful in generating high-quality, diverse synthetic data for object-centric tasks, existing methods struggle with scene-aware tasks such as Visual Question Answering (VQA) and Human-Object Interaction (HOI) Reasoning, where it is critical to preserve scene attributes in generated images consistent with a multimodal context, i.e. a reference image with accompanying text guidance query. To address this, we introduce **Hummingbird**, the first diffusion-based image generator which, given a multimodal context, generates highly diverse images w.r.t. the reference image while ensuring high fidelity by accurately preserving scene attributes, such as object interactions and spatial relationships from the text guidance. Hummingbird employs a novel Multimodal Context Evaluator that simultaneously optimizes our formulated Global Semantic and Fine-grained Consistency Rewards to ensure generated images preserve the scene attributes of reference images in relation to the text guidance while maintaining diversity. As the first model to address the task of maintaining both diversity and fidelity given a multimodal context, we introduce a new benchmark formulation incorporating MME Perception and Bongard HOI datasets. Benchmark experiments show Hummingbird outperforms all existing methods by achieving superior fidelity while maintaining diversity, validating Hummingbird's potential as a robust multimodal context-aligned image generator in complex visual tasks. Project page: https://roar-ai.github.io/hummingbird

Published as a conference paper at ICLR 2025
HUMMINGBIRD: HIGH FIDELITY IMAGE GENERATION VIA MULTIMODAL CONTEXT ALIGNMENT
Minh-Quan Le Gaurav Mittal Tianjian Meng', A SSM Iftekhar', Vishwas Suryanarayanan' Barun Patra', Dimitris Samaras", Mei Chen' Microsofft "Stony Brook University
ABSTRACT
While diffusion models are powerful in generating high-quality, diverse synthetic data for object-centric tasks, existing methods struggle with scene-aware tasks such as Visual Question Answering (VQA) and Human-Object Interaction (HOI) Reasoning, where it is critical to preserve scene attributes generated images consistent with a multimodal context, i.e. reference image with accompany- ing text guidance query. To address this, we introduce Hummingbird, the first diffusion-based image generator which, given a multimodal context, generates highly diverse images w.r.t. the reference image while ensuring high fidelity by accurately preserving scene attributes, such as object interactions and spatial re- lationships from the text guidance. Hummingbird employs novel Multimodal Context Evaluator that simultaneously optimizes our formulated Global Semantic and Fine- tine-granned Consistency Rewards to ensure generated images preserve the scene attributes reference images in relation to the text guidance while main- taining diversity. As the first model to address the task of maintaining both diver- sity and fidelity given a multimodal context, we introduce a new benchmark for- mulation incorporating MME Perception and Bongard HOI datasets. Benchmark experiments show Hummingbird outperforms all existing methods by achieving superior fidelity while maintaining diversity, validating Hummingbird's potential as robust multimodal context-aligned image generator in complex visual tasks. Project page: https: //roar ai github.......... io/ wo/hummingho.
INTRODUCTION
In recent years, diffusion models (Ho al. 2020 Rombach et al., (022) have emerged as power- ful tools for image generation, offering impressive capabilities in creating high-quality and diverse synthetic data. This synthetic data has proven valuable in various applications, particularly for object-centric image classification tasks (Shu al. 2022 Feng al. (023). However, for scene- aware tasks like Visual Question Answering (VQA) (Goyal et al. 2017] Antol et al.) 2015) and Human Humman-Object Interaction (HOI) Reasoning (Jiang et al., 2022: Olutan et al.] (220),
essential that the generated images accurately preserve scene attributes relevant to the task, as specified by the accompanying text queries. These attributes may include numerical attributes (e.g., object count), physical properties (e.g., color), spatial relationships (c.g., relative positions of objects), object in- teractions (c.g., how objects interact with each other), or scene-level details. Although defining an all- all-exhaustive list of such attributes is intractable, tasks like VQA and HOI, through their open-ended questions (Antol et al. 2015), enable to capture the relevant semantic attributes effectively.
While existing image generation methods have made great strides in improving the diversity (defined as how different the generated images are from a given reference image and from each other), they often fall short in maintaining high fidelity. Given a text query as guidance pertaining to scene attributes in relation to given reference image (together referred to as multimodal context), we define fidelity how truthfully the image generator preserve those attributes in the generated image. Existing methods, including state-of-the-art (SOTA) diffusion models like Stable Diffusion XL (SDXL) (Pdell et al.] 2024), Image Translation techniques like Boomerang (Luzi al. 2024
Textual Inversion (Gal et al.] 2023) Trabucco et al. 2024), and Image Variation (Xu et al.
2023
A Equul contribution. This work was done as Minh-Quan's internship project Microsoft.

---

Published as a conference paper at ICLR 2025
Text Suidance: Are there six people?
Reference there a red hat?
Is there, a bottle?
Reference [Ans: Yes] Generated [Ans: Yes]
Generated (Ane
Reference [And:
Generated (Are: Yee)
(a) Numerical Attribute
Physical Attribute (Color)
(c) Object Existence
Text Suidonoe: Is the dog under the pool? Reference (Are Ns]
Is this general store outdoor?
Is person swinging or holding a tennis racket?
Generated [Ans: No]
Reference (ArE Yes] Generated [Anz: Yes]
Reffrencc: [Ans: swing]
Generated [Ans: swing!
FEE RANS HEAD TABE
(d) Spatial Relationship
Visual Attribute (Scene)
(f) Human- Numan-Object Interaction Reasening
(g)
Diversity/Fidelity Comparison
Reference Image
I2T2I SDXL
Image Variation
Image Tronslation
Textual Inversion
Hummingbird
v Diversity X Fidelity
v Diversity Fideliny
X Diversity Fidelity
v Diversity Fidelity
v Diversity v Fideliity
Text Guidence: Is the clock above people?
Context Description from MILM: The image shows city street scene with clock mounted on a building. The clock is positioned obove the street level, where pedestrians walking. The street lined with shops and buildings, and there people walking the sidewalk. clock cleanly visible serves as a prominent feature the image.
Figure 1: Hummingbird aligns generated image with multimodal context input (reference image text guidance) ensuring the synthetic image is diverse w.r.t. reference image while exhibiting high fidelity (i.e. preserves the scene attribute from reference image in relation to text guidance). (a-f) By doing so, for VQA and HOI Reasoning, Hummingbird enables the answer to the question in the text guidance remain consistent between the reference and generated images. (g) Hummingbird is also able to preserve both diversity and fidelity without the trade-off exhibited by existing methods.
often lose fine- areegra scene details as the embeddings do not have precise control on which scene attributes to capture. This lack of fidelity is particularly problematic in synthetic data generation for scene-aware tasks like VQA and HOI Reasoning, where preserving scene elements relation to text query is critical for high performance.
To achieve high fidelity while preserving diversity image generation w.r. multimodal context, introduce Hummingbird, the first diffusion- based general-purpose image generator designed to produce high-fidelity images guided by a multimodal context while maintaining diversity. Given a multimodal context, containing reference image and accompanying guidance, Hummingbird generates highly diverse images that differ from the reference image while accurately preserving the scene attributes referenced in the text guidance.
Hummingbird contains a novel Multimodal Context Evaluator that simultaneously maximizes our formulated Global Semantic and Fine-grained Consistency Rewards. Given the multimodal con- text input, Hummingbird crafts an instruction prompt to provide to Multimodal Large Language Model (MLLM) (Liu al., 2024 Chen et al., (2024) to obtain text-based Context Description con- taining the necessary scene attributes to focus on for image generation. Hummingbird then finetunes the SDXL. diffusion model using the rewards from the Multimodal Context Evaluator based on the alignment between the generated image and the MILM Context Description. While SDXL enables generating images with high diversity (i.e. visually different from the reference image), the rewards encourage the fine-tuning process to align the generated image closely with the scene attributes pro- vided in the multimodal context to achieve high fidelity. Unlike existing methods, Hummingbird is able to balance both diversity and fidelity in generated images which is a prerequisite for leveraging synthetic data for scene-aware tasks like VQA and HOI Reasoning. Figure [ (a-f) illustrates how, given text guidance (a question) and reference image, the generated image remains visually diverse
2

---

Published as a conference paper at ICLR 2025
w.r. reference image while preserving the specified scene attributes, thus ensuring that the answer to the question in the text guidance remains consistent between the reference and generated images.
As the first approach to address the task of multimodal context-aware image generation with high fidelity and diversity, we also introduce new benchmark formulation to evaluate different methods on this task. Our benchmark leverages the MME Perception benchmark (Fu et al. (024) to perform Dest-Time&A Augmentation (TTA) (Shanmugam et al.) [021] Kim et al., 2020) with real and generated synthetic images evaluate the ability of method to maintain fidelity on scene attributes related to spatial existence, count, position, color, and scene, We further leverage Bongard Human-Object Interaction (HOI) (Tiang et al. 2022) dataset to perform Test-time Prompt Tuning (TPT) (Shu al, -bbjectt
2022) to test a method's ability to maintain fidelity when focusing on sophisticated interactions. Finally, we compute a method's ability to maintain diversity using feature-based dis- tance metrics between the reference and generated images. Experiments show that Hummingbird able to outperform all existing methods on MME Perception and Bongard HOI (i.e. provide the best fidelity) while achieving high diversity. Hummingbird also outperforms all other methods con- sistently on ImageNet and its OOD variants. This validates its effectiveness on large and diverse datasets from both scene- aware and object-centric tasks. Figure [ (g) demonstrates that Humming- bird is able to preserve diversity and fidelity without the trade-off exhibited by existing methods.
Our contributions are as follows:
We introduce Hummingbird, the first fiffusion-basee image generator to synthesize high fidelity images guided by multimodal context of reference image and text guidance while maintaining high diversity.
We develop novel Multimodal Context Evaluator that simultancously maximizes our for- mulated Global Semantic and Fine-grained Consistency Rewards during end-to-end fine- tning. This enables the diffusion model to focus on both local and global scene attributes.
Hummingbird outperforms existing methods on MME Perception and Bongard HOI datasets as part of newly formulated benchmark that evaluates the ability to generate images with both high fidelity and high diversity given a multimodal context.
RELATED WORK
Popular diffusion-based image generators, such SDXL (Podell al. 2024), have made tremen- dous progress in producing high-quality diverse images. But these methods struggle to fully trans- late complex scene details from text to image, especially if it involves preserving specific scene attribute w.r.t. a reference image. We can observe this limitation in Figure [g] where the image generated based text description of reference image (image-to-text-to-maaa (I2T2T)) cannot preserve the scene elements from reference image (poor fidelity).
Image Variation methods like Xu et al. (2023); Feng al. (2023); Zhang al. (022): Le et al. 2004); Graikos et al. 2024)); Belagali et al. (2024). use an image encoder to extract embeddings to condition diffusion models. While this helps retain high-level semantics and allow changes to style and other low-level features, these methods fail to preserve specific scene attributes of a reference image due to a lack of precise control over which detail to emphasize. Moreover, addition of noise during diffusion process introduces randomness which causes significant deviation from the reference image, again resulting in poor fidelity (Figure (g)).
Image Translation Image Editing techniques, such Boomerang Luzi et al. 2024) and Control- Net (Zhang et al., 2023), add noise to reference input image to create variations around the original reference image in latent space. Since they aim to generate multiple local samples, they fail to pro- duce sufficiently diverse images (Figure (g)). Moreover, the added noise during forward diffusion step often distorts removes the details specific scene attributes in the reference image.
Textual Inversion methods (Gal et al. 2023 Trabucco et al. 2024 Ahn et al. 2024 Nguyen et al. 2024 encode new concepts as "pseudo-word words" in text embedding space guide image generation. Although effective for style transfer, these methods struggle with tasks where a single reference image is involved, such as TTA, to generate multiple synthetic images. When fine-tuning with just one reference image, these methods lead to images containing irrelevant scene attributes and fail to preserve the attributes from the original reference image (leading to poor fidelity, Figure (g))
3

---

Published as a conference paper at ICLR 2025
Multimodal Context M
Text Guidance
Reference
Frozen
LoRA Fine-tuning
Grodient Flow
Tmage Enthodding Ie
Are there dogs appeer image?
CLIP Image Encoder
Generated
Multimodal Context Evaluator

 imageelerrrodee
MLLM
UNet Denoiser 6g
Glabal Semontic Reward
elomants and contect the
mage to recttl cnnver
cuattioe:
Fine-groined Corsistency Reward
The ardwer [eessars)
[oseatioo)
There ore two people siting
dags nariin in prossy field
CLIP Text Encoder
descmppton is rected to question beng caked [FINSI)
shearsing the The
Text Enchending
of differeet agse and breeds and enjoyin tame sundberss.
Pronpt Template
Context Description C
Given text guidance g and reference image (multimodal context M), Hummingbird crafts an instruction prompt to feed to MLLM and obtain Context Description C. them embeds x and C via CLIP feed to UNet Denoiser of SDXL generate image & To improve the fidelity of & w.r.t. MM while preserving diversity, Hummingbird introduced Multimodal Context Evaluator to simultaneously maximize novel rewards Global Semantic and Fnne-Grined Consistency Rewards - to align & with scene attributes provided in M.
Figure 2: Model Overview:
PRELIMINARIES
Latent Diffusion Models (LDM) (Podell et al. 2024 Rombach et al. 2022 operate a com- pressed latent space rather than directly in pixel space. Using VAE (Kingma 2013) encoder E, they encode an input image X ~ Pdata into a lower-dimensional latent variable Zo. on which the diffusion process applied. The model iteratively corrupts Zo with noise and learns reverse this process to generate new samples coherent with the original data distribution. The training objective minimizes the error between the noisy latent sample time and the denoised prediction:
Cample 1,, (00
[||e <s +)]l"]
(1)
where eg (Z,, t) represents UNet denoiser's (Ho et al. 2020 Rombach et al. 2022) predicted noise.
Denoising Diffusion Implicit Models (DDIM) (Song al. 2021) accelerate the sampling process
diffusion models while maintaining high-quality samples. DDIM adjusts the reverse diffusion
process by directly predicting the latent variable at the next timestep using deterministic sampling, which allows for faster convergence. The DDIM update step is defined as:
Zt-1 = vat-1 2:
/1 aneo (Zi,t) + /1 - au-1 eo (z,,
(2)
aa
METHOD
4.1
MODEL OVERVIEW
Figure 2 provides overview of the end-to-end training setup of Hummingbird. Let M {x, g] denote the multimodal context fed as input to Hummingbird. MM comprises of the reference image and text guidance g for Hummingbird to generate image & that is visually diverse w.r.t. x while faithfully preserves the scene attributes in relation to g. Depending on the task, can be a question or set of annotations accompanying x. During training, g additionally consists the ground truth (such answer to the question or correct annotation) which is unavailable during evaluation.
We feed the multimodal context M MILLM to generate a text-based Context Description, For this, need to provide M an instruction prompt to MLLM such that it guides the MLLM on which scene attributes to focus on in the reference image x in relation to the text guidance g. We devise prompt template p g
to transform g E M into instruction prompt P. We then feed P along with x as the multimodal instruction prompt to the MLLM to obtain C. Figure provides sample of one such prompt template. Please refer Appendix Corr more samples.
provides a detailed description of x, capturing the scene attributes specified by g, such as object relationships, their interactions, other spatial attributes. We pass C through CLIP text encoder
4

---

Published as a conference paper at ICLR 2025
UNet Denoiser en
BLIP-2 QFrmer Multimodal Context Evoluator
Leaned Queres Qwnne
Image Takene I (
Image Encoder CLIP ViT-L/14
Cross Attention
Globol Semontic
Generated X
Reward
Text Takena T.keva
Context Description C
Text Encoder
BERT Base
Bidiractional SA
Fine-grained Consistency Reward
IIM Classifier
Frozen
LoRA Fine-tuning
Gradient Flow
Figure 3: Hummingbird's Multimodal Context Evaluator leverages pre-trained BLIP-2 QFormer: It simultaneously maximizes our novel Global Semantic and Fine-grained Consistency Rewards to align generated image x with Content Description corresponding to multimodal context M.
(Radford et al. 2021 convert it into text embedding Te. In parallel, also encode x using CLIP image encoder to generate the image embedding Ie comprising the semantic features of x.
We feed the embeddings Ie and Te to the UNet denoiser Eg SDXL (Podell al. 2024) generate image &. While & exhibits diversity (i.e. visually different from x), we fine-tune the denoiser via LoRA (Hu al. 2022) to enhance the fidelity of & (i.e. preserve scene attributes occurring in in relation to To facilitate the fine-tuning, we introduce Multimodal Context Evaluator which simultaneously maximizes our two novel rewards Global Semantic and Fine Tine-Grained Consistency Rewards to align & with the scene attributes provided in the multimodal context M (Figure 3
4.2 FINE-TUNING WITH MULTIMODAL CONTEXT REWARDS
Multimodal Context Evaluator. We design our Multimodal Context Evaluator using the multi- modal representation connector in pre- pre-trained vision-languae models (VLMs). Specifically, we leverage the pre-trained QFormer from BLIP-2 (Li et al. 2023) due to its unique ability to cap- ture both global and local alignment between image-text pairs. We use BLIP-2's image encoder to extract the image tokens Tenkens (x) for the image & generated by SDXL. (Figure 3). In parallel, we use BLIP-2's BERT-based (Kenton & Toutanova] 2019) text encoder to extract the text tokens Trokens (including the representative token T(cLs)) ) for the text-based Context Description C obtained from the MILMM We then utilize the learned queries Qlearned from the pre-trained BLIP-2 QFomeer to extract the visual representation token sequence Z via the Cross-Attention layer as,
Z CrossAttention (tearned- Itnkens (x))
(3)
Global Semantic Reward. Using the visual representation tokens and representative text token TieLs1, compute cosine similarity between each visual token Z, and TieLsI and define Global Semantic Reward, Rglobal, as the maximum of these computed cosine similarities,
2TT TTeus)
Reglobal = max
12 I
Through the above formulation, the Global Semantic Reward, Rglobal, ensures the overall scene context in the generated image aligns with that in the textual context description by maximizing their global feature similarity.
Fine-Grained Consistency Reward. While Rglobal allows to align the global semantics of the gen- erated image & with that of the textual context description C, it is not sufficient to maintain optimal fidelity as SDXL. can still omit capturing key scene attributes specified g. We therefore intro- duce Fine-Grained Consistency Reward, Rime grained. that helps to complement Rellobal and captures the multimodal context more comprehensively in a fine-grained manner. For this, we leverage the bi-directional self-attention along with the image-text matching (ITM) classifier of pre-trained BLIP-2 Former to attend to all the visual tokens Z and Context Description text tokens Tiokens with each other. The ITM classifier outputs two logits: one for the positive match (with index j = 1) and one for the negative match (with index = 0). the training of Hummingbird, positive pairs are defined as the generated image and its corresponding context description within the same training
5

---

Published as a conference paper at ICLR 2025
Algorithm Multimodal Context Rewards Fine-tuning
Require: Pre-trained UNet denoiser cæ; original data distribution Pasta: context descriptor MLLM.
Ensure: ea converges and minimizes Lital.
Activate UNet denoiser cg; freeze context rewards Rglobal, Raine- gruieed context descriptor MLLM
while Leaal not converged do
Sample multimodal context input fx. ~ Pdatai t =T
P p(g)
b. Create instruction prompt P from text guidance and prompt template p > Extract context description from MLLM
C- MLLM(x, P)
Extract image embedding Io of reference image X
Extract text embedding To of description C
while do
P perform DDIM denoising
7 f v .I -T Ta..)
+ VT- a1-11-1 6o (Zz, Ie: To,
ar-1
var
end while
D(z))
VAF Decoder decodes latent zo pixel space
Creat Ryveal + A2 Rane-gnine))
Backward Lima and update eg for last K steps.
end while
batch. We select the logit corresponding to the positive match j = 1 of the classifier as Riine-grained grained,
Rine grained = ITM ClassifierrBiddrectiiioo SA(2, 2, Teokens))
grained
(5)
Raie grained therefore captures the fine-grained multimodal image-text alignment between and C. Loss Function. We combine Reglobal and graine to compute loss Loaal to fine-tune denoiser cØ.
Lusa (A1 Relobal + Xg R line grained);
66)
where d and X2 are the hyperparameters balancing the contribution from the two rewards.
Training Procedure. During training, we freeze all components of Hummingbird except UNet denoiser Eg in SDXL which we fine-tune using LoRA (Figure and 3) Additionally, text guidance comprises the ground truth corresponding to the reference image in relation to the input text query. (
an overview of the training procedure. We optimize the UNet denoiser Eo in SDXL. using DDIM (Song et al.] 2021) scheduler, guided by the two rewards Rglohal and Rinn-grineed For each training step. we extract the image embedding Ie of the reference image and text embedding To of context description
to condition the generation process. We obtain the generated image via 25-steps DDIM and VAE decoder D from latent Zo to pixel space &, which we feed to Multimodal Context Evaluator along with C. Since each timestep in the denoising process differentiable, we compute the gradient to update parameters in denoiser Eg through chain rule,
OC 00
ØR
0%
I
00-1 VI-area Te.t) /1 at-1 Eo (2,, Ie, To:
Var
0x
00
Evaluation. During evaluation, we remove the Multimodal Context Evaluator with final output being the generated image & (Appendix[D D Figure D.. For fair evaluation, text guidance g no longer includes the ground truth corresponding to the reference image in relation to the input text query.
5
EXPERIMENT
5.1
BENCHMARK FORMULATION
To evaluate Hummingbird's effectiveness as a multimodal context-aligned image generator, we in- troduce benchmark focused on two key criteria: fidelity, which measures how accurately the gen- crated images preserve scene attributes as per the multimodal context, and diversity, which assesses how distinct the generate images are from each other well from the reference image.
For fidelity, we build our evaluation framework around three main considerations: Applicability, ensuring generated images enhance model performance when combined with real data; Efficiency, emphasizing computationally light and resource-efficien protocols; and Fairness, promoting stan- dardized test-time evaluation to mitigate biases caused by varying training processes.
Based on these principles, we adopt two evaluation settings: (1) VQA Benchmark for MLLMs using Test-Time Augmentation (TTA) (Shanmugam et al. 202T), and (2) Human- Interaction (HOI)

---

Published as a conference paper at ICLR 2025
Reasoning using Test-time Prompt Tuning (TPT) from [Shu et al. (2022) Both settings leverage real and synthetic data to boost performance while allowing computational efficiency. TTA uses pre- trained MLLMs without requiring additional training for fair comparison, while TPT fine-tunes prompt embeddings for quick convergence.
For diversity, we conduct two experiments using CLIP ViT-G/14 (Radford al. 2021 features. First, we compute the Euclidean distance between the generated and reference images to quantify how distinct the generated images are from their reference counterparts. Second, we generate images using 20 different random seeds and compute the average pairwise Euclidean distance across all generated images, providing measure of intra-set diversity.
Datasets and Metrics. For the VQA benchmark, we fine-tune Hummingbird on VQAv2 /ooya al., 2017) and GQA (Hudson & Manning 2019), then evaluate using TTA on MME Perception (Fu et al. (024). common benchmark for assessing SOTA MLLMs. Our benchmark covers MME Perception tasks related to Existence, Count, Position, Color, and Scene (more discussion on this in Appendix A). We generate synthetic images using the test image
reference and pair them with corresponding questions as text guidance. We then feed the real (reference) and generated images to MLLMs along with yes/no questions to extract the logit for the next token [yes] / [no]. [nn]]. We determine the final predicted token by averaging the logits and comparing it to the ground truth (yes/no). Using the paired yes and no questions for each test image, we report Accuracy (ACC), which measures the correctness of individual question predictions, and Accuracy+ (ACC+), which measures the joint correctness of the question pair.
For HOI Reasoning, we fine-tune Hummingbird on Bongard-HO (Jiang 2022] training set and evaluate on associated test sets using TPT from Shu al. (2022). Following the setup, given query test image with support sets of positive (e.g.. riding bicycle) and negative (e.g., not riding bicycle) images, we generate synthetic images for support images as reference using corresponding label as text guidance. We then use the augmented support sets to optimize a pair of prompt embeddings that contrast each other and predict the human-object interaction in query image by comparing its feature similarity to the two optimized prompts. Please refer to Shu et al. (2022) for more details. We use Accuracy as a measure of model's 's ability to correctly predict these human-object interactions.
Method Comparisons. We compare Hummingbird against representative techniques from four groups SOTA image generation techniques (covered in Section [ (I) customized T21 diffusion models (referred to as 12T2I SDXL), (2) Image Variation, (3) Image Translation/Editinn and (4) Textual Inversion, as well as data augmentation methods like RandAugment (Cubuk et al. 2020).
Object-Centrie Benchmark. In addition to scene-aware tasks like VQA and HOI Reasoning, also evaluate Hummingbird on an object-centric benchmark to demonstrate its versatility. Specifi- cally, we fine-tune the UNet denoiser the ImageNet training set (Deng et al. 2009). and perform TPT (Shu et al.] (2022) using real and generated images on the ImageNet test set and four out-of- distribution (OOD) datasets: ImageNet-A (Hendrycks et al. (022b), ImageNet- V2 (Recht et al.) 2019). ImageNet-R (Hendrycks et al. 2021a), and ImageNet-Sketch (Wang et al. 2019). This en- ables to assess the robustness of Hummingbird under natural distribution shifts. We use Top-1 accuracy which measures the correctness of classifying test images.
5.2 IMPLEMENTATION DETAILS
For Hummingbird, we use SDXL Base 1.0 which is standard pre-trained diffusion-based image generation model. We further employ CLIP ViT-G/14 as the image encoder and both CLIP-L/14 & CLIP-G/14 the text encoders (Radford et al., 2021). We perform LoRA fine-tuning with 11M trainable parameters (=s 0.46% of total 2.6B parameters) on 8 NVIDIA 1100 80GB GPUs using AdamW (Loshchilov & Hutter 2019) optimizer, learning rate of 5e 6, and gradient accumulation steps of 8. Please refer to Appendix Q for more details.
5.3
COMPARISON WITH EXISTING METHIODS ON THE BENCIIARK FORMULATION
VQA benchmark. To evaluate Hummingbird for VQA using MME Perception, we experiment with SOTA MLLMs including LLaVA v1.6 (Liu et al. 2024) and Intern 2.0 (Chen et al. 2024). Table I shows that Hummingbird significantly improves both ACC and ACC+ compared to all ex- isting methods on both LLaVA v1.6 and Intern VL 2.0 consistently across all tasks. The ability of Hummingbird's generated images to enhance MILMM performance on complex VQA tasks validates
7

---

Published as a conference paper at ICLR 2025
Table 1: Comparison for VQA benchmark on MME Perception using Test-Time Augmenta- tion (TTA), Hummingbird outperforms SOTA image generation and augmentation techniques con-
sistently across all MME tasks, when evaluating with different MLLMs.
MLLM
Method
Existence
Count
Position
Color
Scene
Name
ACC
ACC+
ACC
ACC+
ACC
ACC+
ACC
ACC-
ACC+
Real only
95.00
90.00
81.67
66.67
76.67
53.33
93.33
86.67
86.00
72,50
Real RandAuggent
93.33
86.67
78.33
60.00
78.35
56.67
91.67
83.33
85,50
72.00
Cabuk et al. 2020
1.007
-333
1331
16.67
0334
1106
1333
10.50
19.50
Real + Image Variation
88.33
76.67
71.67
46.67
76.67
56.67
85.00
73.33
86.50
73.00
Xuu et.ttal, 2023
13.8
1000
1 1,2000
0.334
8.33
13334
1..00
1..50
LLaVA
Real + Imagg Translation
93.33
86.67
78.33
60.00
80.00
60.00
93.33
86.67
86.75
73.00
Liu
v1.6 7B
2024
[uzz ct ml 2024
-331
1334
16.67
1.335
0.600
0.775
1,950
Real + Textual Inversion
86.67
73.33
71.67
43.33
73.33
50.00
85.00
70.00
85,25
71.00
Gal et al. 2023
8,33
16.87
10.00
23.34
1,3
0,3
8.33
16.07
0,735
1 1.350
Real 12T21 SDXL
96.67
93.33
81.67
66.67
75.00
50.00
88.33
76.67
85.50
72.50
Podell ct al. 2024
1.60
-5.33
1.167
0.933
1500
10.00
0.0.9
Real Humminghird
96.67
93.33
83.33
70.00
81.67
66.67
95.00
93.33
87.75
74.00
1.87
3,33
1.66
13333
5.500
13.34
1,67
1.666
1.775
1..50
Real only
96.67
93.33
73.33
50.00
76.67
60.00
91.67
83.33
85.00
70.00
Real RandAngment
95.00
90.00
76.67
66.67
76.67
60.00
88.33
76.67
84.75
69.50
Cabuk al. 2020
1.67
-333
1.334
16.67
13334
1.6.6
4.0 0.225
1,0.50
Real Imuge Variation
91.67
83.33
73.33
53.33
70.00
46.67
78.33
60.00
85.25
70.50
Xu et al. 2023
5.000
10.00
13.35
1.667
13.33
13.34
1.23.3
1.2.5
1.0.5
InternVL VL 2.0
Real Imaae Translation
85.00
70.00
75.00
50.00
78.33
63.33
90.00
80,00
85,50
70.50
Chen et al. 2024
Luzi al [024]
11.67
133339
1.167
1.566
0333
4.667
1.333
1:0.0
1.0.0
Real Textual Inversion
83.33
66.67
70.00
46.67
61.67
40.00
75.00
56.67
84.25
68.50
Gal et al. 2023
13.34
16266
1333
1.335
15:00
20.00
1. 16667
16.6
1.755
1.30
Real 12T21 SDXL
93.33
86:67
78.33
56.67
65.00
43,33
95.00
90,00
84.75
70.50
[Podell et al. 2024
3.34
6.60
1,500
16.67
11.67
11:67
73333
1.007
1 0.25
1.030
Real Humminngbied
98.33
96.67
86.67
73.33
78.33
63.33
98.33
96.67
86.25
71,00
1.66
3.34
1
13.34
1 23.33
1
1.66
3.33
0.06
13.34
1.25
1 1.00
Table 2: Comparison Human- Objjct- Interaction (HOI) Reasoning using Test-time Prompt Tun-
ing (TPT). Hummingbird outperforms SOTA methods on all 4 test splits of Bongard- HOI dataset.
Splits
Method
Average
seen act.,
unseen act.,
seen oct., unsocn ob).
unseen act., unseen ohj.
obj.
oby.
RandAugment (Cabuk al
2020
2020
66.39
68.50
65.98
65:48
66.59
Image Variation
20255
57.86 $888
59.71
56.07
55.58 19900
57.31 19228
Image" Translation 1 ef di 2124
66.22 1
68.13 0.37
66.09 0.111
66.12 1:04
66.64 1.005
Textual Inversion Gal ot 2023
55.18
11.21
59.16
19.34
55,30
10.68
54,16 11.32
55.95
10.64
12T21 SDXL (Podei
67.26 0.887
69.25
67.23
125
65.76 0.22
67.38
Humminghira
68.14
70.95
68.28 2.30
67.56
68.73 2.14
that Hummingbird can generate images faithful to the input multimodal context. The improvement is particularly prominent for tasks needing a fine-grained understanding of the scene as in Count and Position. For LLaVA v1.6, Hummingbird boosts Position ACC by 5% and ACC+ by 13.34%, while for Intern VL 2.0, it significantly improves both Count ACC and Color ACC+ by 13.34% and Count ACC+ by 23.33%. This also highlights Hummingbird's 's generalizability across different MLLMs.
HOI Reasoning. We experiment with CLIP-ResNet50 (Radford et al. 2021 evaluate Hum- mingbird on Bongard-HOI using TPT. Table 2 shows that Hummingbird outperforms all existing augmentation/image generation methods consistently across all test splits, achieving the highest av- erage accuracy of 68.73% (+2.14% over data augmentation- based baseline). While VQA on MME Perception helps to demonstrate Hummingbird's effectiveness on spatial scene attributes, Hum- mingbird's 's superior performance on Bongard- HOI demonstrates the method's ability to generate high-fidelity augmentations also for novel interaction- scene attributes.
Object-Centric benchmarks. Table 3 shows the evaluation of Hummingbird on object-centric benchmarks, including ImageNet and its OOD variants (ImageNet-A, ImageNet-V2, ImageNet-R, and ImageNet-Skette SSeech) using TPT. We can observe that Hummingbird consistently outperforms all other image generation/augmentatioo methods across all object-centric benchmarks. This validates Hummingbird's versatility by being effective on both scene-aware and object-centric tasks.
Qualitative Study. Figure 4 provides qualitative comparison between Hummingbird and other im- age generation methods across different scene-aware tasks from MME Perception and HOI Rea- soning benchmarks. Hummingbird consistently surpasses all other approaches, achieving higher fidelity w.r.t. the input multimodal context (reference image along with text guidance) while main-
8

---

Published as a conference paper at ICLR 2025
Table 3: Hummingbird outperforms SOTA generation/augmmenttion methods on object-centric
datasets including out-of-distribttion (OOD) variants showing its robustness to distribution shifts.
Method
ImageNet
ImageNet-A
ImageNet-V2 V2 ImageNet-R
ImageNet-Sk.
Average
OOD Avg.
58.10
53.90
42.26
Real RandAugment
59.40 1:30
27.34 +:453
55. 1.20
$6.80
1.90
34.50 1:000
46.65 4.35
43.46 1.266
Reall Image Variation
60,80 1:270
31.06 1:825
$5.50 2:50
58.30 1400
37,10 13300
48.71
1644
45.69 14480
Real Image Transhation
61.90
32.14 19.33
16.2013.00
59.60
37.20
49.41 1:15
46.29 1:5.9
Real Testual Inversion
60.10
1
35.50
47.87
1.561
44.81 1:401
Real + 12T21 SDNL
61.20-13.10
31.56 1:775
56.20
13220
58.50 1.00
38.10 1.000
19.11 6.855
46.09 1:529
Real + HHmminngiee
62.60 4.50
32,85
10.84
56.50
3.30
60.20 6.30
38,80
5.30
50.19
7:93
47.09 6.2.
Reference Image
12T2I SbXo
Image Variation
Image Translation
Textual Inversion
Humninghird
people image? (Count Yes]
CO
Guidence: Is
the left of the lamp? (Posttion/ Anst
Text Guidonce: Is person holding or painging hennis nacket? (HOT Reosening/ / Ans: holding)
Figure 4: Generated image comparison between Hummingbird and SOTA methods on MME Percep- tion and HOI Reasoning. Hummingbird achieves highest fidelity while maintaining high diversity. Table 4: Euclidean distance (diversity) between features of real and generated images, and among generated images with varying random seeds. Higher is better. Bold denotes best and underline is second best. While achieving best fidelity (Table 1 Hummingbird provides second best diversity.
Comparison Type
Augmentt
12T21
Image Vartation
Image Translation
Textual Inversion
Humminghird fine-tuingg / fine-tuiin
Reference (real) vs genernted Among generated
36,84
36
18.51
27.14
25.85
20.39
26.46
26.08
26.87
taining high diversity w.r.t the reference image. For example, in Figure 4 (Row 1), the reference image depicts 4 people, and Hummingbird successfully maintains this count in the generated image. In contrast, 12T21 SDXL., Image Translation, and Textual Inversion generate images with 6, 5, and 2 people, respectively. Image Variation struggles even with image quality, with an estimated people count ranging between and Please refer to Appendix for more qualitative results.
5.4 ABLATION STUDY
Diversity Analysis. Table shows that Hummingbird (after fine-tuning) achieves the second- highest Euclidean distance score, both w.r.t the reference image and among generated images, slightly behind 12T2I SDXL, while securing the highest fidelity as shown in Tables [ and [ Fig- ure iillustrates Hummingbird's high diversity in generated images across different random seeds, validating its ability to provide high fidelity w.r.t multimodal context while preserving diversity.
Effectiveness Multimodal Context Rewards. Table 5 shows an ablation to evaluate the impact of Global Semantic Rglobal and GrainGr Consistency Reward Rinc-grained of Hummingbird. We use LLaVA 1.6 7B for evaluation while considering both LLaVA 1.6 7B and InternVL 2.0 8B as MLLM for Context Deseription. Table 5 (Row 1 and 5) show that Hummingbird achieves the best performance when both rewards are applied. Performance reduces in absence of either reward, especially tasks requiring detailed multimodal context preservation (e.g. for Position and Count).
Impact of MLLM as Context Descriptor. Table 5 also helps to analyze the effect of using dif- ferent MILMS obtain Context Description C. We consider LLaVA 1.6 7B and Intern 2.0

---

Published as a conference paper at ICLR 2025
Table 5: Ablation Multimodal Context Rewards, MLLM Context Descriptor, and Fine-tuning.
Bold is overall best, blue best per Context Descriptor, gray is baseline without fine-tuning.
MILMM Name
Contect Descriptor
Reward
Existence
Count
Position
Color
Sccne
Ry
R Ree-pid
ACC
ACC-
ACC
ACC+
ACC
ACC+
ACC
NCC+
ACC
ACC+
96.67
93.33
83.33
71.00
$1.67
66.67
95,00
93.33
87.75
74.00
LLaVA
96.67
93.33
83.33
70.00
50.00
63.33
95,00
93.33
87.25
73.50
2IZ4
66.67
51.67
66.67
93.35
90000
87.50
74.00
75.33
56.67
90:00
v1.6
70,00
50.00
03:33
87.25
IntemVL 2,088 Cless et al 2024
98.33
96.67
85.00
70.00
78.33
60.00
95.00
93.33
87.00
73.00
98.33
96.67
X3.33
66.67
78. 33
60.00
91.00
90000
86.
73.00
95.67
93:33
67
66, 87
76.87
$6.67
90,00
86.67
86:
86.50
72:50
Reference Image
Seed 0
Sead
Seed
Seed
Seed 4
Text Guidomoc Are there three ginsffes? (Count Arat No)
Text Guidence: Io there skateboard with nod wheels? [Coler / Ane Yes]
Figure 5: Hummingbird exhibits high diversity across different random seeds while producing high- fidelity images each time w.r.t multimodal context (reference image + text guidance).
Text Guidance: Are there six people appear in this image?
Reference Image
No training
Iter 45k
Iter 48k
Iter 50k
Count:
Count: X
Count: 8
Count: 6 v
Count: 6 v
Figure 6: Fine-tuning with Multimodal Context Rewards improves fidelity in generated images. 8B for evaluation. Table 3 (Row and 5) show that training Hummingbird with C obtained from LLaVA performs better than that from Intern VL for Position and Scene tasks while C from Intern VL achieves better accuracy on Existence and Count tasks. This experiment emphasizes the importance of choosing a strong MILM to get Context Description to fine-tune Hummingbird.
Effectiveness of Fine-tuning. Table that fine-tuning SDXL Hummingbird leads to better performance consistently across all tasks compared to the baseline without finetuning ( ggay-shadedd Rows 4, 8). Figure 6 further illustrates how fine-tuning improves fidelity while preserving diver- sity (with reference image having 6 people and text guidance referencing Count attribute, generated image also has 6 people after fine-tuning while remaining visually diverse w.r.t. reference image).
6 CONCLUSION
We introduce Hummingbird, a novel diffusion-based image generation algorithm that provides both high fidelity and diversity when generating images guided by multimodal context, consisting of a reference image and accompanying text guidance. By incorporating a Multimodal Context Evaluator and leveraging Global Semantic and Fine-Grained Consistency Rewards, Hummingbird ensures that the generated images accurately preserve specified scene attributes, such as object interactions and spatial relationships, while maintaining visual diversity. Our comprehensive experiments demon- strate that Hummingbird outperforms SOTA methods across multiple benchmarks, including VQA and HOI Reasoning, as well as object-centric benchmarks. The results validate Hummingbird's ability to generate nigh-iideity, multimodal context-aligned images that improve scene-aware task performance while maintaining diversity of generated images. Hummingbird sets a new standard for scene-aware image generation, with potential applications for wide range of multimodal tasks.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was carried out during Minh-Quant internship Microsoft, and Dimitris Samaras was supported in part by NSF grants IIS-2123920 and IIS-2212046 during the preparation of the manuscript.
REFERENCES
Namhyuk Ahn, Junsoo Lee, Chunggi Lee, Kunhee Kim, Daesik Kim, Seung-Hun Nam, and Kibeom Hong. Dreamstyler: Paint by style inversion with sikt-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 674-681, 2024.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 2425-2433 2015.
Varun Belagali, Srikar Yellapragada, Alexandros Graikos, Saarthak Kapse, Zilinghan Li, Tarak Nath Nandi, Ravi K Madduri, Prateek Prasanna, Joel Saltz, and Dimitris Samaras. Gen-sis: Generative self-augmentation improves self-supervised learning. arXiv preprint urXiv:2412.01672. 2024.
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24185-24198, 2024.
Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoe Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition workshops, pp. 702-703, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and patlern recognition, pp. 248-255. Icee, 2009.
Hugging Face. Diffusers: State-of-the-art diffussinn models for image generation.
https:/
hucgingface cttcco//ccccoooo /oos boocs/diie di difffusrss/iddex 2023. Accessed: 2024-09-26.
Chun-Mei Feng, Kai Yu, Yong Liu, Salman Khan, and Wangmeng Zuo. Diverse data augmentation with diffusions for effective test-time prompt tuning. In Proceedings of the IEEE/CVF Interna- lional Conference on Computer Vision, pp. 2704-27 24,4,77 2023.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024. URL httphtts///axxiv.rrggee/o
1336613399
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or. An image is worth one word: Personalizing text-to-image generation using textual inversion. In The Eleventh International Conference Learning Representations, 2023. URL
net hwt.//ooummmm/mo Foo.ormmiiiNAggOOOTrrrro
Yash Goyal. Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 6904-6913, 2017.
Alexandros Graikos, Srikar Yellapragada, Minh- Le, Saarthak Kapse, Prateek Prasanna, Joel Saltz, and Dimitris Samaras. Learned representation-guidee diffusion models for large-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 3532-8542. 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
11

---

Published as a conference paper at ICLR 2025
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A criti- cal analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 8340-8349, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEF/CVF conference on computer vision and pattern recogni- fion, pp. 15262-15271, 2021b.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-688 6851, 2020.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhie Zhu, Yuanzhi Li. Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference on Learning Representations, 2022. URL. https:://ppenreviit
net tetefforum
id=nzevKeeFyff
Drew Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 6700-6709, 2019.
Huaizu Jiang, Xiaojian Ma, Weili Nie, Zhiding Yu, Yuke Zhu, and Anima Anandkumar. Bongard- hoi: Benchmarking few-shot visual reasoning for human-object interactions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 19056-19065, 2022.
Jacob Devlin Ming-We Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of nLacL-HLT, volume 1, pp. 2. Minneapolis, Minnesota, 2019.
Ildoo Kim, Younghoon Kim, and Sungwoong Kim. Learning loss for test-time augmentation. Ad- vances in neural information processing systems, 33:4163 41744, 2020.
Diederik P Kingma. Auto-encoding variational bayes. arXiv preprint
2013.
Munn-Quan Le, Alexandros Graikos, Srikar Yellapragada, Rajarsi Gupta, Joel Saltz, and Dimitris Samaras. oo-brush: Controllable large image synthesis with diffusion models in infinite dimen- sions. In European Conference on Computer Vision, pp. 385-388--- 401. Springer, 2024.
Junnan Li, Dongxu Li. Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference machine learning, pp. 19730-19742. PMLR, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recogni- tion, pp. 26296-2666 26306, 2024.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019. URL http::://ppeeeeeeee...//// net. www.mfrtt/eeeeeeeeeoooooo iid=
Bkg6RiCqY7
Lorenzo Luzi, Paul Mayer, Josue Casco-Rodriguez, Ali Siahkoohi, and Richard Baraniuk. Boomerang: Local sampling image manifolds using diffusion models. Transactions Machine Learning Research, 2024. ISSN 1835-8856. URL httpps://peenreree..nnt
forum? win.mmiddd-ddThnnn-m NNW1
Thao Nguyen, Yuheng Li, Utkarsh Ojha, and Yong Jae Lee. Visual instruction inversion: Image editing via image prompting. Advances in Neural Information Processing Systems, 36, 2024.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, al. Pytorch: An imperative style, high- performacce deep learning library. In Advances in Neural Information Processing Systems, 2019.
volume 32, pp.
12

---

Published as a conference paper at ICLR 2025
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for resh-resolttion image synthesis. In The Twelfth International Conference Learning Representations, 2024. URL net forum? wtwp://wwwnnn///ffm21dd-ei-- 5228xqqf
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceed- ings of the International Conference on Machine Learning (ICML), 2021.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pp. 5389-5400. 5400..MLR. PMLR, 2019.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Divya Shanmugam, Davis Blalock, Guha Balakrishnan, and John Guttag. Better aggregation in test-time augmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1214-1223, 2021.
Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in Iasion-language mod- els. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL htt.::tt////ooom htttt::://ppenevieew net forum? id=e8PVEkSa4Fq
Yang Song, Jascha Sohl- Dick-Dicsttein, Durk P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- lional Conference on Learning Representations, 2021.
Brandon Trabucco, Kyle Doherty, Max Gurinas, and Ruslan Salakhutdinov. Effective data aug- mentation with diffusion models. In The Twelfth International Conference Learning Repre- sentations, 2024. URL /ttptpps///peeeeviee www.//n.e/eteeee..eee wnttt//orouniddd12222222a22
Oytun Ulutan, A S M Iftekhar, and BS Manjunath. Vsgnet: Spatial attention network for detecting human object interactions using graph convolutions. Proceedings the the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13617-13626, 2020.
Haohan Wang, songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representa- tions by penalizing local predictive power. Advances in Neural Information Processing Systems, 32, 2019.
Xingqian Xu, Zhangyang Wang, Gong Zhang, Kai Wang, and Humphrey Shi. Versatile diffusion: Text, images and variations all in one diffusion model. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 7754-7765, 2023.
Lvmin Zhang, Anyi Rao, and Mancesh Agrawala. Adding conditional control text-to-image diffusion models. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 3836-3847, 2023.
Yuechen Zhang, Jinbo Xing, Eric Lo. and Jiaya Jia. Real-world image variation by aligning diffusion inversion chain. Advances in Neural Information Processing Systems, 36, 2024.
13

---

Published as a conference paper at ICLR 2025
APPENDIX
A DISCUSSION: SCOPE AND ETHICS
In this work, we evaluate our method six core scene-aware tasks: existence, count, position, color, scene, and HOI reasoning. We select these tasks as they represent core aspects of multimodal un- derstanding which are essential for many applications. Meanwhile, we do not extend our evaluation to more complex reasoning tasks, such as numerical calculations or code generation, because SOTA diffusion models like SDXL are not yet capable of handling these tasks effectively. Fine-tuning alone cannot overcome the fundamental limitations of these models in generating images that re- quire symbolic logic or complex reasoning. Additionally, we avoid tasks with ethical concerns, such as generating images of specific individuals (e.g., for celebrity recognition task), to mitigate risks related to privacy and misuse. Our goal was to ensure that our approach focuses on technically feasi- ble and responsible AI applications. Expanding to other tasks will require significant advancements in diffusion model capabilities and careful consideration of ethical implications.
B LIMITATIONS AND FUTURE WORK
While our Multimodal Context Evaluator proves effective in enhancing the fidelity of generated images and maintaining diversity, Hummingbird is built using pre-trained diffusion models such as SDXL and MILMs like LLaVA, it inherently shares the limitations of these foundation models. Hummingbird still faces challenges with complex reasoning tasks such as numerical calculations or code generation due to the symbolic logic limitations inherent to SDXL. Additionally, during inference, the MILMM context descriptor occasionally generates incorrect information or ambiguous descriptions initially, which can lead to lower fidelity in the generated images. Figure 7 further illustrates these observations.
Hummingbird currently focuses on single attributes like count, position, and color as part of the multimodal context. This is because this task alone poses significant challenges to existing methods, which Hummingbird effectively addresses. A potential direction for future work is to broaden the applicability of Hummingbird to synthesize images with multiple scene attributes in the multimodal context as part of compositional reasoning tasks.
Tragge
Hamminghird
Trage
Harninghidd
Trage
Hamningbird
Text Suidunce: Ts of jituure equi 247
Text there three
Text fuédinnce
the
The
Descngtton
mage
hand drean "iht margle with uheled as and The cres of right trangle can caloa ened using the farmul Area (bese height// farmula
with
contro ploosd enit. The remotes
bow bressoh add with table with a striped resleclath The solad gams shed with leno wedges and apnelle nuts, spoon edge
plceed
white and aspear to be for controling elechrone devices,
cane, base Anes (8 6 2/2+22 24 Therefere, the area of the right tringge in the picture squel to 24.
possibly telensian or sound aystem rumber 7000" The nemorca
tioned the bed.
with one on the left side and the other on the night side.
(a) Fatlure due to SDXL's symbolic legic limitations
(b) Failure due to MLLM's context description (c) Failure due to ambigueus context description tion
Figure 7: Failure cases of Hummingbird. (a) Our method fails due to the symbolic logic limitation of existing pre-trained SDXL. (b) Initially incorrect descriptions generated by MLLMS lead to low fidelity of generated images. (c) Context description generated by MLLMs is ambiguous and does not directly relate to the text guidance, the spoon can be both inside or outside the bowl.
C PROMPT TEMPLATES
Figure (a-c) showcases the prompt templates used by Hummingbird to fine-tune diffusion models specifically on each task including VQA, HOI Reasoning, and Object-Centric benchmarks. It's worth noting that we designed the prompt such that it provides detailed instruction to MLLMs on
14

---

Published as a conference paper at ICLR 2025
which scene attributes to focus. We also evaluate the effectiveness of our designed prompt templates by fine-tuning Hummingbird with a generic prompt as illustrated in Figure 8 (d). Table indicates that without using our designed prompt template, the MLLM is not properly instructed to generate specific context description thus leading to reduced performance after fine-tuning on MME tasks.
We believe that when using a generic prompt, MI.MM is not able receive sufficient grounding about the multimodal context leading to information loss on key scene attributes.
Table 6: Effectiveness of the prompt template on finetunning Hummingbird on MME Perception.
MILM
Hummingbird
Existence
Count
Position
Color
Scene
Name
ACC
ACC+
ACC
ACC+
ACC
ACC+
ACC
ACC+
ACC
ACC+
LLaVA (Liu al. 2024
w/ prompt template
96.67
93.33
83,33
70.00
81,67
66.67
95.00
93.33
87.75
74.00
91.67
83.33
75.00
56.67
81.67
63.33
91.67
83.33
87.25
73.00
w/ generic prompt
13.00
110000
InteraVL
w/ prompt template
98.33
96.67
86.67
73.33
78.33
63.33
98.33
96.67
86.25
71.00
91.67
83.33
80.00
60.00
71.67
50.00
91.67
83.33
84.50
69.00
Chen al. 2024
w/ generic prompt
6,66
6.67
13.33
4 6.66
1 13.34

 (nngge/yFrrvvdde concise description of key diements and context the image to directly answer the questions (osestiin)) The crswwr (Xsavve))
t.ios
Are. There. four dagal
appoor in image?
Answest
Ersure the description cleor, concise, and directly related to the question being asked 

Guidance g
Prompt Template
a) VQA

 VProiie Provide concise description of key elements context of the image. foous the interaction between (Sub ser) and [Ubrect) onswer is (Subject) son) (cbsect).
Bublactt person
Obsect:: hernis racket Action: Swing
Ersure the description tion clear, concise, and directly related to the question being asked 

Text Guidonce g
Prompt Template
b) HOI Reasoning
Queatiion: Ansswass American chomeleon, onole, Anolis corsiionnsis

 Provien concise description main object in image to directly unswer the question: [qseatiion] The cnswer [>r swer).
Ensure the description clear, concise, and directly related to the question being asked 

Guidonce g
Prompt Template
c) Object-Centric

 cimage VnGiven the image, generate a coption which coptures all the key aspects of the image. It should not contain anything that not present in the image. key cspect con be count of entities presest, entifics that are neteworthy, relatiorchips betwoon objects or informetive descriptions of the presented image. 

d) Generic prompt
Figure 8: Prompt templates (a-c) used by Hummingbird to fine-tune the diffusion model on each task including VQA. HOI Reasoning, and Object Centric benchmarks. The generic prompt (d) is also included to evaluate the effectiveness of prompt template.
D INFERENCE PIPELINE
In the inference pipeline of Hummingbird (Figure 9). the text guidance g includes only the question corresponding to the reference image x. The answer is excluded for fair evaluation. Moreover, we remove Multimodal Context Evaluator, and the generated image & is the final output.
15

---

Published as a conference paper at ICLR 2025
Multimodal Context M
Text Guidance g
Reference X
Image Embedding Typ
CLIP Image Encoder
Are there four days appear in this image?
Generated X
UNet Denoiser Eg
MLLM

 limage>\/Previde concise description of key elements and context of the image directly question: (question)
The image shows
dogs rumning gressy field. are least four dogs visible in the image. The dogs are of different sizes and breeds, and enjoying themselves es they run arcund. In the backyround, there are two people sitting on bench, watching dogs play.
CLIP Text Encoder
Ensure the description clear, cancise, directly neloted no the question being asked. 

they appear
Text Embedding Te
Prompt Template y
Context Description C
Figure 9: Inference pipeline of Hummingbird
Referance Image
Seed 0
Seed
Seed
Seed 3
Seed 4
Are there three giroffes?
Context Description from MLLM: The image shows two giraffes stonding in grassy area within fenced enclosure. The giraffes are positioned near
rock formation, The enclosure appears be part of zoo or wildlife park, as indicated by the presence of fence and the man-made structures.
Is there skateboard with red wheels?
Context Description tion from MLL.M: The image shows riding skateboard with red wheels, He is wearing red shirt and jeans, and appears to in a food court or market area, as there are food stands and people walking around in the background. The man also holding cell phone his hand.
Figure 10: Examples of context description from MILMM in the inference pipeline where answers are not included in text guidance.
E ABLATION STUDY ON BLIP-2 QFORMER
Our design choice to leverage BLIP-2 Qormer Hummingbird the multimodal context evaluator evaluatrr facilitates the formulation of our novel Global Semantic and Fine Fine-grained Consistency Rewards. These rewards enable Hummingbird to be effective across all tasks as seen in Table ? On replace with a less powerful multimodal context encoder such CLIP VIT-G/14, we can only implement the global semantic reward as the cosine similarity between the text features and generated image features. As result, while the setting maintain performance on coarse-level tasks such Scene and Existence, there a noticeable decline on fine- fine-graine grained tasks like Count and Position. This demonstrates the effectiveness of design choices in Hummingbird and shows that using less
16

---

Published as a conference paper at ICLR 2025
powerful alternatives, without the ability to provide both global and fine- tne-grained alignment, affects the fidelity of generated images.
Table 7: Effectiveness of the prompt template fine-tuning Hummingbird on MME Perception.
MLLM Name
Hummingbird
Existence
Count
Position
Color
Scene
ACC
ACC+
ACC
ACC+
ACC
ACC+
ACC
ACC+
ACC
ACC+
L.LVV
w/ our Evaluator
96.67
93.33
83.33
70.00
81.67
66.67
95.00
93.33
87.75
74.00
Liu et al. 2024
v1.6 7B
w/ CLIP
96.67
93.33
81.67
66.67
80.00
63.33
95.00
90.00
87.75
73.50
1.466
1.333
1.447
1.334
1.333
1..50
InternVI. VL
w/ our Evaluator
98.33
96.67
86,67
73.33
78.33
63.33
98.33
96,67
86.25
71.00
8B
98.33
96.67
81.67
70.00
76.67
60.00
96.67
93.33
86.00
71.00
Chen 2024
w/ CLIP
5,00
1,333
1
1333
1.46
1.334
1..25
F ADDITIONAL EVALUATION ON MME ARTWORK
To explore the method's ability to work on tasks involving more nuanced or abstract text guidance beyond factual scene attributes, we evaluate Hummingbird
additional task of MME Artwork. This task focuses on image style attributes that are more nuanced/abstract such as the following question-answer pair Question: "Does this artwork exist in the form of mosaic?", Answer: "No".
Table 8 summarizes the evaluation. We can observe that Hummingbird outperforms all existing methods on both ACC and ACC+, implying its higher effectiveness generating images with high fidelity (in this case, image style preservation) compared to existing methods. This provides evi- dence that Hummingbird can generalize tasks involving abstract/nuanced attributes such image style. Figure I1 further shows qualitative comparison between image generation methods on the MME Artwork task.
Table Comparison on Artwork benchmark and Visual Reasoning task. Hummingbird outperforms
SOTA image generation and augmentation techniques.
Method
Real only
RandAngment
Image Variation
Image Translation
Textual Inversion
12T21 SDXL
Hummingbird
Artwork ACC
69.50
69.25
69.00
67.00
66.75
68.00
70.25
Artwork
41.00
40.00
37,50
38.00
Reasoning ACC
69.29
67.36
69.29
69.29
67.14
72,14
72.86
Reasoning ACC+
42.86
40.00
41.40
40.00
37.14
47.14
48.57
Reference Tmage
I272I SDXL
Image Varistion
Image Translorian/Editingg
Teversion
Hummingbird
Text Suidonce:
Does this ortwork exist the form of sculpture?
Context Description from MLLM: The image depicts a relief sculpture, which is form of art that corved into a flat surfacee such as wall or a panel. figures. including women and mon, who are inf interacting with each other. The figures ore, depicted realistic
The sculpture fectures style, with attention detail their clothing and expressions.
Qualitative comparison on the Artwork task between image generation method. Hum mingbird can preserve both diversity and fidelity of the reference image in a more abstract domain.
Figure 11:
G ADDITIONAL EVALUATION ON MME COMMONSENSE REASONING
We have additionally performed our evaluation to more complex tasks such as Visual Reasoning using the MME Commonsense Reasoning benchmark. Results in Table 8 highhlight Hummingbird's
17

---

Published as a conference paper at ICLR 2025
ability to generalize effectively across diverse domains and complex reasoning tasks, demonstrating its broader applicability. Figure I2 further shows qualitative comparison between image generation methods on the MME Commonsense Reasoning task.
Reference Image
I2T2I SDXL
Image Veriation
Image Transkation/Editing
Textual Inversion
Hummingbird
V Diversity Fidelity
v Diversity Fidelity
v/ Fidelity
v Diversity Fidelity
v Diversity // Fidelity
Guidonce: toilet guide sign.
go
Context Description from MILA): The inage shows sign with male and female symbol, indicating the presence of restrooms. The sign mounted on wall, and the male symbol on the left nide of the sign. The context suggests that the sinn is providing guidance for restrooms, with the male symool pointing the left. Therefore, if you one mon, you should go to the toilet on the left.
Figure 12: Qualitative comparison on the Commonsense Reasoning task between image generation method. Humminghird can preserve both diversity and fidelity of the reference image in a more abstract domain.
H FID SCORES
We compute FID scores for Hummingbird and the different baselines (traditional augmentation and image generation methods) and tabulate the numbers in Table 9 FID is a valuable metrie for assess- ing the quality of generated images and how closely the distribution of generated images matches the real distribution. However, FID does account for the diversity among the generated images, which is critical aspect of the task our work targets (i.e., how can we generate high fidelity images, preserving certain scene attributes, while still maintaining high diversity?). We also illustrate the shortcomings of FID for the task in Figure 13 where we compare generated images across methods. We observe that RandAugment and Image Translation achieve lower FIID scores than Humming- diversity by only minimally changing the input
bird (w/ finetuning) because they compromise image, allowing their generated image distribution to be much closer to the real distribution. While Hummingbird has higher FID score than RandAugment and Image Translation, Figure 13 shows that it able to preserve the scene attribute w.r.t multimodal context while generating an image that significantly different from than original input image. Therefore, it accomplishes the targeted task more effectively, with both high fidelity and high diversity.
Table 9: FID scores of traditional augmentation and image generation methods. Lower is better.
Humminghird
Method
RandAngment
12T21 SDXI.
Image Variation
Image Translation
Textual Inversion
x fine-runing
fine munin
score
Reference
RendAuyment
12T27
Irage Variation
TransiotivVddtiig
Textual Invension
Humnisgbird
Diversity v/ Fidelty
v Diversity Fidelity
v Diveraity x Fideiit
Diversty v/ Ficelity
Diversity Fideliy
v/ Diversity Fidelity
Figure 13: While RandAugment and Image Translation achieve lower FID scores, Hummingbird balances fidelity and diversity effectively.
18

---

Published as a conference paper at ICLR 2025
1
USER STUDY
We conduct user study where we create a survey form with 50 questions (10 questions per MME Perception task). In each survey question, we show users reference image, a related question, and a generated image each from two different methods (baseline I2T21 SDXL vs Hummingbird), We ask users to select the generated images(s) (either one or both or neither of them) that preserve the attribute referred to by the question in relation to the reference image. If an image is selected, denotes high fidelity in generation. We collect form responses from 70 people for this study. We compute the percentage of total generated images for each method that were selected by the users as measure of fidelity. Table summarizzs the results and shows that Hummingbird significantly outperforms 12T2I SDXL in terms of fidelity across all tasks on the MME Perception benchmark. We have some examples of survey questions in Figure 14
Table 10: User study MME tasks to evaluate the fidelity of generated images by 12T21 SDXL Hummingbird.
Method
Existence
Count
Position
Color
Scene
Average
12T21 SDXL
63.71 Hummingbird
44.43 72.29
40.00 59.57
46.86
87.86 90.00
56.57 76.06
81.29
77.14
Figure 14: Some examples of our survey questions to evaluate the fidelity of generated images from 12T21 SDXL and Hummingbird.
J TRAINING PERFORMANCE ON BONGARD HOI DATASET
Following the existing method (Shu et al. 2022), we conduct an additional experiment by training a ResNet50 (He et al. 2016) model on the Bongard- HOI (Jiang al. 2022) training set with tradi- tional augmentation and Hummingbird generated images. We compare the performance with other image generation methods, using the same number of training iterations. As shown in Table [I] Hummingbird consistently outperforms all the baselines across all test splits. In the paper, as dis- cussed in Section [.1] we focus primarily on test-time evaluation because it eliminates the variability introduced by model training due to multiple external variables such as model architecture, data dis- tribution, and training configurations, and allows for a fairer comparison where the evaluation setup remains fixed.
K RANDOM SEEDS SELECTION ANALYSIS
We conduct an additional experiment, varying the number of random seeds from 10 to 100. The results are presented in the boxplot in Figure L3 which shows the distribution of the mean L2 distances of generated image features from Hummingbird across different numbers of seeds.
The figure demonstrates that the difference in the distribution of the diversity (L2) scores across the different numbers of random seeds is statistically insignificant. So while it is helpful to increase the number of seeds for improved confidence, we observe that it stabilizes at 20 random seeds. This analysis suggests that using 20 random seeds also suffices to capture the diversity of generated images without significantly affecting the robustness of the analysis.
19

---

Published as a conference paper at ICLR 2025
Table 11: Comparison on Human-Object Interaction (HOI) Reasoning by training CNN-baseline ResNet50 with image augmentation and generation methods. Hummingbird outperforms SOTA methods on all 4 test splits of Bongard-HOl dataset.
Test Splits
Method
Average
seen unseen obj.
unnocn act
obj.
obj.
obj.
CNN-baselin (ResNe:50)
50.03
49.89
19.77
50.01
49.92
RandAugment (Cubuk et
2020
51.07 1:104
5
51.79
51.73
51.43
Image Variation Xu
1:75
41,29
18660
41:25 18776
Image Translation Laa v s
2024
46.60 1.43
46.94
2:95
46.38
1339
46.50 13.51
46.51
13331
Textual Inversion
Gal
37.67
37.52
38.12
38,06
37.84
(Podd
52.15
2214
Humminghira
33.71
348
$3.55
3,6
$3.69 1,992
53.41
3.40
$3.89 1:347
Diversity Random Seeds
40
Diversiy Score (MaanL Dicaane)
10 Seads
20 Seads
40 Sieds
60 Seeds Number of Random Seeds
80 Sewda
100 Seeds
Figure 15: Diversity analysis across varying numbers random seeds (10 to 100) using mean L2 distances of generated image features from Hummingbird. The box plot demonstrates consistent diversity scores as the number of seeds increases, indicating that performance stabilizes around 20 random seeds.
L FURTHER EXPLANATION OF MULTIMODAL CONTEXT EVALUATOR
The Global Semantic Reward, Rglobal, ensures alignment between the global semantic features of the generated image and the textual context description C. This reward leverages cosine similar- ity to measure the directional alignment between two feature vectors, which can be interpreted as maximizing the mutual information I(x. C) between the generated image & and the context descrip- tion C. Mutual information quantifies the dependency between the joint distribution p0 C) and marginal distributions. In conditional diffusion models, the likelihood po ($(C) of generating given C is proportional to the joint distribution:
() p(C)
Po (x|C)
where p(C) is the marginal probability of the context description, treated as a constant during opti- mization. By maximizing Rglobal, which aligns global semantic features, the model indirectly max- imizes the mutual information 1(x, C),, thereby enhancing the likelihood pe (x(C)) in the conditional diffusion model.
The Fine-Grained Consistency Reward, Rine-grainedd captures detailed multimodal alignment be- tween the generated image & and the textual context description C. It operates at token level, leveraging bidircctional self-attention and cross-attention mechanisms in the BLIP-2 QFommer, fol- lowed by the Image-Text Matching (ITM) classifier to maximize the alignment score.
20

---

Published as a conference paper at ICLR 2025
Self-Attention Text Tokens:
Text tokens Trokeus undergo self-attention, allowing interactions among words to capture intra-text dependencies:
Tattn = SelfAttention
(8)
Self-Attention on Image Tokens: Image tokens Z are derived from visual features of the generated
image & using cross-attention mechanism:
2 CrossAttentioni Clearned: Trokens (x)
These tokens then pass through self-attention to extract intra-image relationships:
Zattn SelfAttention(2)
(10)
Cross-Attente between Text and Image Tokens: The text tokens Tattu and image tokens Zatta interact through cross-attention focus on multimodal correlations:
CrossAttention Zattn)
(11)
ITM Classifier for Aligmment: The resulting multimodal features F are fed into the ITM classifier, which outputs two logits: one for positive match (j 1) and one for negative match (j 0). The positive class (j = 1) indicates strong alignment between the image-text pair, while the negative
class (j 0) indicates misalignment:
Riine-graied ITM.Classiiferr()pr
(12)
The ITM classifier predicts whether the generated image and the textual context description match. Maximizing the logit for the positive class = 1 corresponds to maximizing the log probability log p(&, C) of the joint distribution of image and text. This process aligns the fine-grat one-grained details in & with C, increasing the mutual information between the generated image and the text features.
Improving fine-grained relationships CLIP. While the CLIP Text Encoder, at times, struggles to accurately capture spatial features when processing longer sentences in the Multimodal Con- text Description, Hummingbird addresses this limitation by distilling the global semantic and fine- grained semantic rewards from BLIP-2 QFormer into a specific set of UNet denoiser layers, men- tioned in the implementation details under Appendix O (i.e., Q. V transformation layers including to q. to v, query, value). This strengthens the alignment between the generated image tokens (Q) and input text tokens from the Multimodal Context Description (K, V) in the cross-attention mech- anism of the UNet denoiser. As result, we obtain generated images with improved fidelity, partic- ularly w.r.t. spatial relationships, thereby helping to mitigate the shortcomings of vanilla CLIP Text Encoder in processing the long sentences of the Multimodal Context Description.
To illustrate further, a Context Description like "the dog under the pool" is processed in three steps: (1) self-attentios is applied to the text tokens (K, V), enabling spatial terms like "dog." "under," and "pool" to interact; (2) self-attention is applied to visual features represented by the generated image tokens (Q) to extract intra-image relationships (3) cross-attention aligns this text features with visual features. The resulting alignment scores are used to compute the mean and select the positive class for the reward. Our approach to distill this reward into the cross-attention layers therefore ensures that spatial relationships and other fine-grained attributes are effectively captured, improving the fidelity of generated images.
M THE CHOICE OF TEXT ENCODER IN SDXL AND BLIP-2 QFORMER
The choice of text encoder in our pipeline to leverage pre-trained models for their respective strengths. SDXL inherently uses the CLIP Text Encoder for its generative pipeline, as it is designed to process text embeddings aligned with the CLIP Image Encoder. In the Multimodal Context Eval- uator, we use the BLIP-2 QFormer, which is pre-trained with BERTTbased text encoder.
N TEXTUAL INVERSION FOR DATA AUGMENTATION
In our experiments, we applied Textual Inversion for data augmentation as follows: given a reference image, Textual Inversion learns text embedding that captures the context of the reference
21

---

Published as a conference paper at ICLR 2025
image (denoted <<oontet>>). This embedding is then used to generate multiple augmented images by employing the prompt: "a photo of <context>>". This approach allows Textual Inversion to create context-relevant augmentations for comparison in experiments.
O CONVERGENCE CURVE
To evaluate convergence, we monitor the training process using the Global Semantic Reward and Consistency Reward as criteria. Specifically, we observe the stabilization of these rewards over training iterations. Figure T6 presents the convergence curves for both rewards. il- lustrating their gradual increase followed by stabilization around 50k iterations. This steady state indicates that the model has learned to effectively align the generated images with the multimodal context.
Global Semantic Reward
Fine Grained Consistency Reward
0.90
0.58
0.37
0.75
0.35
Figure 16: Convergence curves of Global Semantic and Fine-Grained Consistency Rewards
P FIDELITY EVALUATION USING GPT-40
In addition the results above, we compute additional metrics for fidelity, which measure how well the model preserves scene attributes when generating new images from reference image. For this, we use GPT-4o (model version: 2024-05-13) 13) as the MLLM oracle for a VQA task on the MME Perception benchmark (Fu et al. 2024). We evaluate Hummingbird with and without fine-tuning process.
The MME dataset consists of Yes/No questions, with positive and negative question for every reference image. To measure fidelity, we measure the rate which the oracle's 's answer remains consistent across the reference and the generated image for every image in the dataset. We the experiment multiple times and report the average numbers in Table 12 We see that fine-tuning the base SDXL with our novel rewards results in average increase of 2.99% in fidelity.
Table 12: Fidelity between reference and generated images from Hummingbird with and without fine-tuning.
MLLM Oracle
Hummingbird
Fidelity on "Yes"
Fidelity on "No"
Overall Fidelity
GPT-40 Ver: 2024-05-13
w/o fine-tuning
68.33
70.55
71.18
fine-tuning
69.72 1.39
73.61 3.06
74.17 2.99
IMPLEMENTATION DETAILS
We implement Hummingbird using PyTorch (Paszke et al. 2019 and HuggingFace diffusers (Face) 2023 libraries. For the generative model, utilize the SDXL Base 1.0 which is standard and commonly used pre-trained diffusion model in natural images domain. In the pipeline, we employ CLIP ViT-G/14 as image encoder and both CLIP-L/14 & CLIP-G/14 text encoders (Radford et al. 2021). We perform LoRA fine-tuning on the following modules SDXL UNet denoiser includ- ing Q V transformation layers, fully-connected layers (to q. to v, query, value, ff .ff.ne.0.proo)
22

---

Published as a conference paper at ICLR 2025
with rank parameter 8, which results in 11M trainable parameters z 0.46% of total 2.6B pa- rameters. The fine-tuning is done on 8 NVIDIA A100 80GB GPUs using AdamW (Loshchilov & Hutter 2019) optimizer, a learning rate of 5e-6 and gradient accumulation steps of 8.
R ADDITIONAL QUALITATIVE RESULTS
Figure 10 showcases two examples of context description from MLLM in the inference pipeline where answers are not included text guidance. Figure 17 illustrates additional qualitative results highlighting the diversity and multimodal context fidelity between reference and synthetic images, as well as across images generated by Hummingbird with different random seeds. Figure [8 shows additional qualitative comparisons between Hummingbird and SOTA image generation methods on VQA and HOI Reasoning tasks.
23

---

Published as a conference paper at ICLR 2025
Image
Are there three giraffes? [Count Andi No]
Is there skateboard with red wheels? [Color Ansi Yes]
green the image? [Color Ans: Yes]
Is the refrigerator on the right side the picture? (Position / Ans: No]
Is there a bottle this image? [Existence Ans: Yes]
picture captured place of desert sand? (Scene Yes]
69
98
Is there only one bottle. in the image? [Count Ans: Yes]
Is the dog under the pool image? (Position No]
Figure 17:
Diversity and multimodal context fidelity between reference and synthetic image and across generated ones from Hummingbird with different random seeds.
24

---

Published as a conference paper at ICLR 2025
Reference Image
SDXL
Image Variation
Image Transkation/Editing
Textudl Inversion
Hummingbird
Are there only three people in the image? [Count Ard: No]
Is there a white bird the inage? (Color Ans: Yes]
AE
the on the right the lamp? (Position/ Andi No)
Is there car this image? [Existence/ Ars: Yes]
Is this pichure coptured in place of elevator shoft? [Scene Ardi Yes]
Are. there four dogs this image? [Count Ans: Yes]
feeding sheep? [HOI Reasoning Anst sheoring)
person holding swinging Reasoning holding]
Figure 18: Qualitative comparison between Hummingbird and other image generation methods on MME Perception and HOI Reasoning benchmarks.
25


Reviews:
[{"review_id": "XmQ80Z7oUC", "reviewer": "Reviewer_Ag2o", "summary": "The paper introduces Hummingbird, a diffusion-based image generator that aligns generated images with a multimodal context comprising a reference image and text guidance. The model combines Global Semantic and Fine-Grained Consistency Rewards by a Multimodal Context Evaluator, leveraging vision-language models (BLIP-2). Hummingbird generates high-fidelity images that preserve scene attributes while maintaining diversity, performing favorably against state-of-the-art (SOTA) methods in tasks such as Visual Question Answering (VQA) and Human-Object Interaction (HOI) Reasoning.", "questions": "### Questions\n1.\tHow does the ITM classifier select the positive class for computing the Fine-Grained Consistency Reward?\n2.\tWould the model maintain robust performance when using alternative, less powerful MLLMs or other multimodal context encoders in place of BLIP-2?\n3.\tCould the method be adapted for tasks involving more nuanced or abstract text guidance beyond factual scene attributes, such as visual structures (e.g., relative positioning of objects) or style?\n\n### Comments\n- Including failure cases or limitations would provide more completeness of the paper.\n- The paper would give more insights if the paper could outline about the future work.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1.\tInteresting framework. The use of the Multimodal Context Evaluator with reward mechanisms (Global Semantic and Fine-Grained Consistency) is a unique approach that successfully addresses both the fidelity and diversity.\n2.\tComprehensive Evaluation. The model is tested across various benchmarks and datasets, including VQAv2, GQA, and ImageNet, validating robustness under both scene-aware and object-centric tasks.\n3.\tPerformance Gains. Empirical results show that Hummingbird consistently performs favorably against the other SOTA methods in terms of accuracy and consistency for VQA and HOI tasks. This validates the effectiveness of the proposed method in downstream tasks.\n4.\tDetailed Analysis: The paper includes thorough ablation studies that explore the impact of individual components and different pretrained MLLMs.", "weaknesses": "1.\tClarity of the Fine-Grained Consistency Reward. How the ITM classifier's positive class is determined sholud be clarified further.  What does the class ‘j’ mean in equation (5)?\n2.\tLimitations are not discussed. It would be more insightful to discuss about the potential limitations and possible improvement of the idea.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jT5FdwAzrj", "reviewer": "Reviewer_g5Ub", "summary": "This paper proposes an image data augmentation pipeline based on diffusion models. Paired reference image and text guidance embeddings have been used into a diffusion model with LoRA to generate an image, and then the image can be optimized by a multimodal context evaluator who returns a global semantic reward and fine-grained consistency reward. Experimental results have been conducted to prove its effectiveness", "questions": "1. Could you provide further details on how to enhance the fidelity of generated images with respect to spatial relationships? While the CLIP Text Encoder is effective, it sometimes struggles to accurately capture spatial features when processing the longer sentences in the Context Description in Figure 2.\n2. when generating the x_hat, you use CLIP Image Encoder and CLIP Text Encoder. However, in the BLIP-2 module, you opt for the BeRT text encoder instead. Could you clarify the rationale behind this choice?\n3.  How is Textual Inversion, which fine-tunes a rarely used text embedding to learn novel concepts, being applied for data augmentation in your comparison experiments?\n4.  Regarding line 274, what criteria do you use for convergence? Additionally, could you present your convergence curve in experiment?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The first work applying diffusion models for image data augmentation.\n2. A pioneering study demonstrating the potential of synthetic data.\n3. Produces impressive results.", "weaknesses": "1. The writing needs improvement; for example, the introduction should clearly state that the research task focuses on data augmentation.\n2. Consider adding the following experiments: 1) evaluation of augmented image quality, such as using FID scores and user studies. 2) more assessment of the proposed augmentation's performance in training, not test-time. 3) Inclusion of a baseline in Table 4, such as \"random seed + stable diffusion,\" to compare data augmentation capabilities, as the vanilla diffusion model does have variety, and I think 20 random seeds are not enough.\n3. Other aspects mentioned in Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3VYgqtdTfS", "reviewer": "Reviewer_spMH", "summary": "The paper presents a new diffusion-based image generation method designed to address the challenge of maintaining both diversity and high fidelity in multimodal contexts. The main contributions are:\n\n1. Introducing Hummingbird, a diffusion model capable of generating high-fidelity and diverse images based on multimodal context (a reference image and text guidance).\n\n2. Proposing a novel Multimodal Context Evaluator that simultaneously maximizes global semantic and fine-grained consistency rewards, ensuring that the generated images maintain scene attributes from the multimodal context while preserving diversity.\n\n3. Presenting a new benchmark using the MME Perception and Bongard HOI datasets, demonstrating Hummingbird's superiority in generating high-fidelity and diverse images compared to existing methods.", "questions": "1. What is the basis for selecting the global semantic and fine-grained consistency rewards in the Multimodal Context Evaluator? Could more mathematical derivation or theoretical support be provided to explain the effectiveness of these reward mechanisms?\n\n2. The experiments primarily use the MME and Bongard HOI datasets. Could the performance of the method be validated on larger or more diverse datasets? This would be crucial to demonstrate the generalizability of the method.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "Originality: The paper introduces a new multimodal context alignment approach that balances diversity and fidelity. The introduction of a Multimodal Context Evaluator and reward mechanism demonstrates high originality.\n\nQuality: The experimental design is well-conducted, clearly validating the proposed method's effectiveness in maintaining diversity and high fidelity.\n\nSignificance: Generating high-fidelity and diverse images is crucial for many complex visual tasks, particularly those involving scene understanding. Hummingbird demonstrates excellent performance in this area.\n\nClarity: The paper is well-organized, with a natural flow between sections, and the experimental results clearly highlight the comparative advantages over existing methods.", "weaknesses": "1. Lack of comprehensive theoretical basis: While global semantic and fine-grained consistency rewards are proposed, there is a lack of detailed mathematical derivation or theoretical analysis, especially regarding why these rewards are effective in improving fidelity.\n\n2. Limited evaluation dataset diversity: The paper uses the MME and Bongard HOI datasets, but their representativeness may be limited, particularly regarding generalizing the model to broader scenarios. It is recommended to validate the method on more diverse datasets in future work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KaNedZYUaY", "reviewer": "Reviewer_WWqb", "summary": "The paper introduces Hummingbird, an image generation model that creates high-fidelity and diverse images aligned with multimodal context. It outperforms other methods on scene-aware tasks and uses a novel evaluator to optimize image generation.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1、High Fidelity and Diversity: Hummingbird generates images that are both diverse and maintain high fidelity to the multimodal context, which is crucial for complex visual tasks like VQA and HOI Reasoning.\n\n2、Novel Multimodal Context Evaluator: The model uses a new evaluator that optimizes Global Semantic and Fine-grained Consistency Rewards, ensuring that generated images accurately preserve scene attributes from the reference image and text guidance.\n\n3、Superior Performance: Benchmark experiments demonstrate that Hummingbird outperforms existing methods, showing its potential as a robust multimodal context-aligned image generator.", "weaknesses": "1、What is the use of using multimodal input as a condition? What are the benefits of using text as a condition compared to Stable Diffusion?\n\n2、The sophisticated Multimodal Context Evaluator and the fine-tuning process might imply high computational requirements.\n\n3、The performance of Hummingbird is likely to depend heavily on the quality and relevance of the multimodal context (reference image and text guidance) provided. In scenarios where the context is ambiguous or low-quality, the model's effectiveness may be compromised.\n\n4、While Hummingbird shows strong performance on VQA and HOI Reasoning tasks, the document does not provide evidence of its effectiveness on a broader range of tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)