Information Theoretic Text-to-Image Alignment
Authors: CHAO WANG
Giulio Franzese
Alessandro Finamore
Massimo Gallo
Pietro Michiardi
Venue: ICLR 2025 Poster (2025)
Keywords: Diffusion model
Text-image alignment
Mutual information
Primary area: generative models
TL;DR: A novel fine-tuning method for text-to-image generative diffusion models, that uses mutual information to align generated images to user intentions through natural prompts.

Abstract: Diffusion models for Text-to-Image (T2I) conditional generation have recently achieved
tremendous success. Yet, aligning these models with user’s intentions still involves a
laborious trial-and-error process, and this challenging alignment problem has attracted
considerable attention from the research community. In this work, instead of relying on
fine-grained linguistic analyses of prompts, human annotation, or auxiliary vision-language
models, we use Mutual Information (MI) to guide model alignment. In brief, our method
uses self-supervised fine-tuning and relies on a point-wise MI estimation between prompts
and images to create a synthetic fine-tuning set for improving model alignment. Our
analysis indicates that our method is superior to the state-of-the-art, yet it only requires
the pre-trained denoising network of the T2I model itself to estimate MI, and a simple
fine-tuning strategy that improves alignment while maintaining image quality. Code available at https://github.com/Chao0511/mitune.

Published as a conference paper ICLR 2025
INFORMATION THEORETIC TEXT-TO-IMAGE ALIGNMENT
Chao Wang 1.2 Giulio Franzese', Alessandro Finamore', Massimo Gallo", Pietro Michiardi EURECOM Huawei Technologies SASU, France?
'[chao. wang, giulic. pietro. wetro.micciardde /eereccmm 3 [wang chao3, alessandro. finamore, masaimo... gallo (aalloo@@hawee.com
ABSTRACT
Diffusion models for Text-to-Image (T21) conditional generation have recently achieved tremendous success. Yet, aligning these models with user's intentions still involves a laborious trial-and-error process, and this challenging alignment problem has attracted considerable attention from the research community. In this work, instead of relying on fine-grained linguistic analyses of prompts, human annotation, or auxiliary vision-langue -anguage models, we use Mutual Information (MI) to guide model alignment. In brief, our method uses self-supervised fine-tuning and relies on a point-wise MI estimation between prompts and images to create synthetic fine-tuning set for improving model alignment. Our analysis indicates that our method is superior to the sfatc-of-the-art yet it only requires the pre-trained denoising network of the T21 model itself to estimate MI, and simple fine- tuning strategy that improves alignment while maintaining image quality. Code available at https: /tt///tttttuuu wtg.tgtuuuu...g.......gg com Caaaa//Chah111111 1/mi tune
INTRODUCTION
Generative models used for Text-to- Image (121) conditional generation (Rombach al. 2022 Ramesh et al.
2022 Saharia al. 2022 Balaji al. 2022b Gafni et al. 2022 Podell et al. (2024) have reached impressive
performance, In particular, diffusion models (Sogg & Ermon 2019b; Ho et 2020 Kingma et al. 202T
Song& & Ermon| 2020 Song et al. 2021 Dhariwal & Nichol 2021) generate extremely high-quality images
by specifying a natural text prompt that acts as a guiding signal (Ho & Salimans 2022, Nichol et al. 2022
Rombach et al. (0222. Yet, accurately translating prompts into images with the intended semantics is still complex (Conwell & Ullman 2022; Feng et al. 2023a Wang et al. (003a)). Issues include catastrophic neglecting (i.e., prompt elements are not generated), incorrect attribute binding (i.e., elements attributes such color, shape, and texture are missing or wrongly assigned), incorrect spatial layout (i.e., elements are not correctly positioned), and a general difficulty in handling complex prompts (Wu et al. 2024).
On the one hand, quantifying model alignment is not trivial. Various works (Hu et al. 2023 Gordon et al. 2023) Grimal et al. 2024) propose different metrics, most of which use complementary Visual Question Answering (VQA) models or Large Language Models (LLMs) to create scores measuring and explaining alignment. Moreover, recent work (Huang et al., (222)) introduces a comprehensive benchmark suite to ease comparison among different metrics and modeling techniques via "categories", i.e., pre-defined set of attribute binding, spatial-related and other tasks.
On the other hand, addressing T21 model alignment we can group the related literature into two main families: inference-tine and fine-tuning methods, For inference-time methods, the key intuition is that the generative process can be optimized by modifying the reverse path the latent variables. Some works (Chefer et al.] [023a] Li et al., 2023b Raasin et al. 2023) mitigate failures by refining the cross-attention units (Tang al. 2023) of the denoising network of
more challenging than measuring it. Broadly,

---

Published as a conference paper at ICLR 2025
on-the-fly, ensuring they attend to all subject tokens in the prompt (typically directly specified as a complementary prompt-specific input for the alignment process) and strengthen their activations. Other inference-tin time methods (Agarwal et al. (2023); Liu et al. (2022);
Stable Diffusion (SD) (Rombach et al., 2022)
Kang et al. al. (2023); Dahary et al. (2024); Meral et al. (2024); Feng al. (2023b); Kim et al. (2023); ); Wu et al.
(2023a); Zhang et al. 2024a b)). focus on individual failure cases. These approaches (i) require a linguistic
analysis of prompts. leading to specialized solutions that rely on auxiliary models for prompt understanding, and (ii) result in considerably longer image generation time due to extra optimization costs during sampling.
Considering fine-tuning methods, some works (Wu et al. 2023d] Lee et al.
2023) require human annotations
to prepare a fine-tuning set, while others (Fan et al. [023] Wallace et al.
2023 Clark et al. 2024) rely on
Reinforcement Learning (RL). Direct Preference Optimization (DPO),
differentiable reward function to
steer model behavior. Recent methods use self-playing (Yuan et al. 2024 Xu et al. 2023 Sun et al. [2023]
Wang al. 2023b Ma et al., 2023), ) auxiliary models such VQA (Li al. 2023a Jiang et al. (024)) or segmentation maps (Kirillov et al. (023) in semi-supervised fine-tuning setting. While these methods do not introduce extra inference time costs, they still require human annotation (which subjective, costly, and
does not scale well) and/or auxiliary models to guide the fine-tuning.
Complementary both families are heuristic-based methods that rely on a variety of "tricks", such as prompt engineering (Witteveen & Andrews| [022, Liu & Chilton 2022; Wang et al. (023a), negative prompting (hin 2022, Mahajan et al. 2023] Ogezi & Shi 2024). prompt rewriting (Mañas et al. 2024) or brute force an
appropriate seed selection (Samuel et (021]]222)])k
et al. 2023). While these methods can be beneficial
specific cases, they fundamentally shift the alignment problem to users.
Overall, current approaches require extra information (human input, auxiliary models, and additional data). To the best of our knowledge, no previous work investigates self-supervised approaches for T21 alignment, i.e., the use of pre-trained model to generate images given specific set of prompts, and select the most aligned ones to prepare a fine-tuning set, without using auxiliary models. In this work, we investigate this strategy from information theoretic perspective, by using MI quantify the non-linear prompt-image relationship. In particular, we focus on the estimation of point-wise MI using neural estimators (Belghazi et al. 2018 Song & Ermon 2019a; Brekelmans et al. [022: Franzese et al., 2024] Kong et al., (2224). and study if and how MI can be used as a meaningful signal to improve T21 alignment, without relying on linguistic analysis of prompts, nor auxiliary models or heuristics. Our method unfolds follows. We build upon the work in (Franzese et al. 2024) and extend it to compute point-wise MI. We then proceed with a self-supervised fine-tuning approach, whereby we use point-wise MI to construct fine-tuning set using synthetic data generated by the T21 model itself. We then use the recent adapter presented in (Liu et al. (0244) fine-tune small fraction of weights injected in the T21 model denoising network. summary, our work presents the following contributions:
We empirically study whether MI between natural prompts and corresponding images considering both qualitative and quantitative approaches. Specifically, we show that MI provides a meaningful indication of alignment with respect to both alignment metrics (BLIP-VQA and HPS) as well as a users study ( 3.1.
(1) We define a point-wise MI estimator suitable for a discrete-time setting ( 2)
(2) We design a self-supervised fine-tuning approach, called MI-TUNE ( 3.2). that uses a small number of fine-tuning samples to align pre-trained T21 model without extra auxiliary models or inference overhead.
(3) We perform an extensive experimental campaign using a recent T21 benchmark suite (Huang et al. 2023) and SD-2.1-base as base model obtaining sizable improvement compared to six alternative meth- ods ( 4). Those benefits hold also when considering more complex tasks (based on DiffusionDB (Wang et al. ))22)) and alternative base models (namely, SDXL (Podell et al.) 2024 Moreover we study the tra between T2I alignment and image quality that has been overlooked in the literature. Specifically, while the well-known FID, FD-DINO and CMMD metrics suggest modest image quality/variety deterioration as a
2

---

Published as a conference paper ICLR 2025
consequence of alignment objectives, optimizing the Classifier Free Guidance (CFG) hyper-parameter of the fine-tuned model at generation time, enables finding "sweet spot" between T21 alignment and image quality.
2 PRELIMINARIES
Diffusion models. Denoising diffusion models (Ho al. 2020 Sohl-Dickstein et al. 2013) are generative models characterized by forward process, that is fixed to a Markov chain that gradually adds Gaussian noise to the data according to a carefully selected variance schedule 3r, and a corresponding process, that has Markov structure well. Intuitively, diffusion models rely on the principle of iterative denoising: starting from a simple distribution Dr ~ (0, I), samples are generated by iterative applications of denoising network co, that removes noise over T denoising steps. A simple way to learn the denoising network Eg is to consider a re-weighted variational lower bound of the marginal likelihood:
sexcrete-time reverse
Csimple (0) (v. (T. P000 [[|E - EØ( volVarao + (VI e, )))
(1)
where = -1- Br ā. 1=1 agg. For sampling, we let of = Br. A similar variational objective can be obtained by switching perspective from discrete continuous time (Song et al. (02T), whereby the denoising
network approximates a score function of the data distribution. For image data, the denoising network is
typically parameterized by a UNET (Ronneberger et al. 2015] Rombach al. 2022).
This simple formulation has been extended to conditional generation (Ho & Salimans 2021), whereby a conditioning signal p injects "external information" in the iterative denoising process. This requires a simple extension to the denoising network such that it can accept the conditioning signal: ca P, (). Then, during training, a randomized approach allows to learn both the conditional and unconditional variants of the denoising network, for example by assigning null value to the conditioning signal. At sampling time, a weighted linear combination of the conditional and unconditional networks, such as EØ (T,, p, co c,, 0, + ((co p, c0 0, t)) can be used.
In this work, we use pre-trained latent diffusion models operating on a learned projection of the input data zo into corresponding latent variable zu which is -wwer-dimensional compared to the original data. Morcover, the conditioning signal is obtained by a text encoder such as CLIP (Radford et al. 2021)
MI estimation. MI central measure study the non-linear dependence between random variables (Shan- non 1948 MacKay 2003), and has been extensively used in machine learning for representation learning (Bell
& Sejnowski] 1995 Stratos 2019 Belghazi et al. 2018 Oord et 2018 Hjelm et al., 2019), and for both
training Alemi et al. |2066 Chen et al. 2016 Zhao et al. 2018) and evaluating generative models Alemii &
Fischer
2018 Huang et al. 2020).
For many problems of interest, precise computation of MI is not trivial (McAllester & Stratos 2020 Paninski 2003). Consequently, a wide range of techniques for MI estimation have flourished. In this work, we focus on realistic and siggh-immnssinall data, which calls for recent advances in MI estimation (Papamakarios et al. Belghazi al., [2188] Oord et al., 2018 (Song& & Ermon 2019a Rhodes et al., (2000) Letizia Tonello
2017
2022
Brekelmans et al.] 2022] Kong et al.) 2024). In particular, ar, we capitalize on recent method (Franzese
ct al.
2024). that relies on the theory behind continuous-time diffusion processes (Song et al., 2021) and
uses the Girsanov Theorem (Øksendal 2003) to show that score functions can be used to compute the Kullback-Leibler (KL) divergence between two distributions. In what follows, we use a simplified notation and gloss over several mathematical details to favor intuition over rigor. Here consider discrete- time diffusion models, which are equivalent to the continuous-time counterpart under the variational formulation, up constants and discretization errors (Song et al. 2021).
We begin by considering the two arbitrary random variables 2 and p which are sampled from the joint distribution proment,prompt where the former corresponds to the distribution of the projections in a latent space
3

---

Published as a conference paper ICLR 2025
Color binding: "A blue car and a red horse"
BLIP-VQ4 VQA 0.95
BLAP-VOA 0.40 HPS =2:500 0,310
BLIP-VQA = -7
BLII--O VOA 0.06 HPS = 0.263
BLIP-VOA VQA 0:05
HPS = 0,319
HPS = 0.312
HPS = 0.258
MI = 36.28
MI 24.56
MI 22.05
M1 = 15.44
MI 14,67
Texture binding: "A fabric dress and glass table"
0012
0.257
HPS 0.213
0.201
RPS 0.231
0.295
MI
26.1
19.86
15.41
9.34
Shape binding: "A round bag and a rectangular wallet"
BLIP-YQA 0.82
BLIP-VQA 0.64
HLIP-VQA = 0.27 HPS 0.262
BLI--VQA = 0.24 HPS 0.216 MI 12.50
BLIP-VOA 000
HPS 0.262
HPS 0.247
HPS 0.160
XII 18.51
MI 17.16
MI 14.84
MI 11.57
Figure 1: Qualitative analysis of MI as an alignment measure (all metrics decrease from left to right). See also Appendix
of the image distribution, and the latter the distribution of prompts used for conditional generation. Then, following the approach in (Franzese et al. 2024), with the necessary adaptation to the discrete domain (see Appendix Aoror details), point-wise MI estimation can be obtained as follows:
3.T
I(2,p) p) = (i.... [se]lco(z..P.)) p. cz,, 0.01 Ke 20x kc)
(2)
Given a pre-trained diffusion model, we compute an expectation (over diffusion times () of the scaled squared norm of the difference between the conditional (g(2,, P, and unconditional networks eg (2 0, Z), which corresponds to an estimate of the point-wise MI between an image and prompt. Intuitively, the difference between these scores quantifies how much extra knowledge of the prompt helps in denoising the perturbed images. This is both a key ingredient and a competitive advantage of our method, as it enables celf-contai approach to alignment based on the T21 model alone without auxiliary models human feedback.
3
OUR METHOD: MI-TUNE
The T21 alignment problem arises when user's intentions, as expressed through natural text prompts, fail materialize in the generated image. Our novel approach aims address alignment using a theoretically grounded MI estimation, that applics across various contexts. To improve model alignment, we introduce a self-supervised fine-tuning method. Leveraging the T2I model itself, we estimate MI and generate an information-thes theoreeti enhanced fine-tuning dataset. While our focus in this work is on T21 alignment, our framework remains extensible to other modalities.
4

---

Published as a conference paper ICLR 2025
3.1 Is MUTUAL INFORMATION MEANINGFUL FOR ALIGNMENT?
To the best of our knowledge, MI has never been evaluated as a meaningful signal for T21 alignment. As such, in this section we perform both qualitative and quantitative analyses to investigate this aspect.
Qualitative analysis. Starting with a qualitative analysis, we select set of simple prompts to probe color, texture, and shape attribute binding from T21-CompBench (Huang et al. 2023) using SD (Rombach et al. 2022) (specifically 3D-2.1-base to generate the corresponding images. We then measure the well-known BLIP-VQA VQA (Huang et al. 2023) and Human Preference Score (HPS) (Wu et al. 2023b) alignment metrics as well as point- MI estimates. BLIP-VQA uses large vision-langue score, by casting questions against an image to verify that the prompt used to generate it is well represented. HPS is an elaborate metric that uses auxiliary pre-trained model, blending alignment with aesthetics according to human perception, which are factors that can sometimes be in conflict. Fiiuree [ccollects some examples and related metric scores revealing a substantial agreement among all measures: all metrics decrease from left to right in the figure, as prompt- -maae alignment deteriorates.
model to compute an alignment
Quantitative analysis. To quantitatively measure the agreement between MI and well-established aligmment metrics, we use all 700 prompts from T2I-CompBench and use SD (again, SD-2. to generate 50 images per prompt. We use point-wise MI to rank such images and select the 1st, 25th, and 50th. For these three representative images, we compute BLIP-VQQ and HPS scores and re-rank them according to both metrics. Last, we measure agreement between the three rankings using Kendall's method (Kendall) 1938), and average results across all prompts. Results indicate good agreement between MI and BLIP-VQA (T= 0.4), and a strong agreement between MI and HPS (r = 0.68).
To strengthen our analysis, we also perform a users study eliciting human preference (see Appendix B. for details). Given a randomly selected prompt from 121 that users can read, we present the top-ranked generated image (among the 50) according to MI, BLIP-VQA and HPS, in randomized order. Users can select one or more images to indicate their preference regarding alignment and aesthetics, for a total of 10 random prompts per user. From the 102 surveys from 46 users, we find that human preference for prompt-image pairs goes to MI for 69.1 %, BLIP-VQA for 73.5% and HPS for 52.2% of the cases, respectively.
Relevant literature. Overall, our analyses support our intuition by which MI meaningful signal for alignment (and possibly aesthetics too), setting the stage for our T21 alignment method. Our intuition is also supported by recent studies investigating the information flow in the generative process of diffusion models. Specifically, Kong et al. (2024) estimates pixel-wise mutual information between natural prompts and the images generated each time-stp of a backward diffusion process. They compare such "information maps" to cross -ross-attention maps (Tang et al. 2023) in an experiment involving prompt manipulation modifications of the initial prompt during reverse diffusion and conclude that MI is much more sensitive to information flow from prompt to images. In a similar vein. Franzese et al. 2024) compute MI between prompt and images at different stages of the reverse process of image generation. Experimental evidence indicates that MI can be used to analyze various reverse diffusion phases: noise, semantic, and denoising stages (Balaji et al. 2022a). While previous studies do not explicitly focus on alignment, they indirectly support our intuition that MI estimated using a diffusion model gauges the amount of information text prompt conveys about image (and vice-versa) which is key for T21 alignment.
3.2 SELF-SUPERVISEI FINE-TUNING WITH MI-TUNE
In summary, given pre- traitrd diffusion model such as SD (Rombach al. 2022) or any variant, such as SDXX. (Podell et al. 2024), we leverage our point-wise MI estimation method to select a small fine-tuning dataset set of information theooretticcc aligned examples.
Our self-supervised alignment method relies on the pre-trained model only to produce a given amount of fine-tuning data, which is then filtered to retain prompt-image pairs with a high degree of alignment, according
5

---

Published as a conference paper ICLR 2025
pair-wise MI estimates obtained using only the pre-trained model. We begin with a set of fine-tuning prompts P. which can be either manually crafted, or borrowed from available prompt collections (Wang et al. 2023a Huang et al., 2023). Ideally, fine-tuning prompts should be conceived to stress the pre-trained model with challenging attribute and spatial bindings, or complex rendering tasks.
Algorithm 1: MI-TUNE
Algorithm Point- wise MI Estimation
Input
:Pre-trained model: ca. Prompt set:
Input :Pre-trained model: cu; Prompt: p
Hyper par:Image pool size: M; Top Mi-aligned samples: in Output :Fine tuned diffusion model eg*
Output : Generated latent: 2; Point-wise MI: 1(z.p)
Function FointWiseM (eg. p);
Fine-tire binettnnnng
17. Initial labent semple
S --
2
~ NN T) for & T, 0
for p in do for E {1.-.. N....MM M) do
11 153 estimation
3
1(z.,p) +=
// Generace and congute MI
[Ke]lea(21, P. ca(=,, 0.4)11"]
4
(() (1)) PPon==oonssee (ea. (())
Boize cample
Append samples
4
40 N 1) iff el w
5
s/p ( Jappend (2 1(260) ))
Samp1 step
s
6
end
Retain only Top-k elenents
v1 c0( p. wow
Sip (Spp
end
end
return Ey- FineTune (eg.
return =. I( p)
As described in Algorithm 1 for each prompt p(O) in the fine-tuning set P, we use the pre-trained model to generate a fixed number M of synthetic images. Given prompt-image paits (p(); ), E [1, M. we estimate wase MI and select the top & pairs, which will be part of the model fine-tuning dataset S. Finally, we augment the pre-trained model with adapters (Hu et al.] 2021 1: Liu et al. 2024). and proceed with fine-tuning. We study the impact of the adapter choice, and whether only the denoising network or both the denoising and text encoder networks should be fine-tuned (Appendix E.2). Moreover, we measure the impact of the number of fine-tuning rounds R to the praitreined model, i.c., we renew the fine-tuning dataset S using the fine-tuned model, and re-fine-tune it using Algorithm [ (S 4.4). Our efficient implementation combines latent generation and point-wise wise computation as shown in Algorithm 2 Since MI estimation involves computing an expectation over diffusion times f, it is easy to combine generation and estimation in the same loop. Moreover, the function is easy to parallelize to significantly speed up the fine-tuning set composition.
4
EXPERIMENTAL EVALUATION
4.1 BENCHMARK AND METRICS
Benchmark. We compare all techniques using T21-CompBench (Huang et al. 2023), a benchmark composed of 700/300 (train/test) prompts across 6 calegories including attribute binding (color, shape, and texture categories), object relationships (2D-spatin and non-spatial associations). and complex composition tasks. These prompts were generated with predefined rules or ChatGPT (OpenAl 2024). We also assess MI- TUNE performance on more realistic prompts by sampling 5,000/1,250 (train/test) prompt-image pairs from DiffusionDB (Wang et al.) (022), large-scale dataset composed of complex human-craftee prompts paired with the corresponding images generated from SD model.
Alignment Metrics. Evaluating T21 alignment is difficult as it requires a detailed understanding of prompt- images pairs, and many metrics have been proposed, e.g., CLIP (Hessel et al., 2021 Radford et al. 2021 ) MINIGPT-4 (Zhu et al., 2024), and human evaluation.
our work we use BLIP-VQL (Huang et al. 2023
HPS (Wu et 2003ch) and UniDet (Zhou et al. 2022). While BLIP-VOR computes answers approach- - a given prompt is decomposed and each part is transformed into a question for an auxiliary VQA model; then, answers are aggregated into single score only based on alignment, HPS includes both alignment and aesthetics this is enabled by an auxiliary model pre-trained using human-annotated data. As in (Huang et al. 2023), the 2D-spatial category is evaluated using the UniDet object detection model.
with questions-
6

---

Published as a conference paper ICLR 2025
We complement these metries with user study. We randomly select 100 prompts per category, and generate 10 pictures per prompt for each method we consider in our evaluation. Then, we run surveys composed of 12 rounds (2 for each category), each showing to the user a randomly selected prompt and randomly selected image for each method, randomly arranged in a grid. At each round, users need to select zero or more images they consider aligned with the prompt. Overall, we collected 42 surveys from users, from which we computed the total percentage of times each method was selected for each category (Appendix B.2p.
Image quality metrics. Assessing performance only considering alignment metrics can hide undesired effects. Intuitively, strong adherence to given prompt reduces the generative process "degrees of freedom" and this trade-off might not be visible even to a trained eye. To investigate these dynamics we compute FID (Heusel al., 2017). FD-DINO (Oquab al. 2024) and CMMD (Jayasumana et al.. 2024) scores FID favors natural colors and textures but struggles to detect objects/shapes distortion, while FD-DINO and CMMD favor image content. Following JImagen-Team et al. 2024), rather than using the 121-CompBench test set, we compute the metrics using 30k samples of the MS MS-COCO-201 2000-2014
(Lin et al. 2015 validation set.
4.2 MI-TUNE FINE-TUNING
Base models. We mainly run our benchmark using 1--ace as base model, but we also report results of the application of MI-TUNE on SDXI. to demonstrate its flexibility.
Fine-tuning sets. 121-CompBench contains 700 training prompts for each category. When using MI-TUNE, we generate M = 50 images for each prompt using the pre-trained model, compute their point-wise MI, and select the top k = (sensitivity M and in Appendix E.I). For the 2D-Spatial category, also
compose fine-tuning sets generating images from SPRIGHT (Chatterjee et al. 2024 model optimized for this (more challenging) category and fine-tuned from SD-2.1 (a higher resolution version of SD-2.1 -bbase. Last, we also contrast MI-TUNE fine-tuning set composition against (i) using HPS rather than MI for image selection (1i) using both Ml-selected and real-pictures and (iii) images from DiffusionDB.
Fine-tuning weights. In our work, fine-tuning corresponds to injecting DoRA (Liu et al. 2024 adapters (rank and scaling factor a are set to 32) only into the attention layers and fully connected layers of the denoising UNET network, whereas other layers are frozen
Other hy hypearrss search. We consider up Re [1, rounds of fine-tuning i.e., using as base model the one obtained from previous round and apply Algorithm [ and Classifier Free Guidance (CFG) E [2.5, 7.5]. For each fine-tuned model we then compute all alignment and image quality metrics. More fine-grained hyperparams details and computational costs considcrations in Appendiix C.
4.3 ALTERNATIVE METHODS
Inference-time methods. Pre-trained model alignment can be improved at inference by optimizing the latent variables 2, throughout the numerical integration used to generate the (latent) image. This process steers model alignment with auxiliary loss based attention maps and fine-grained linguistic analysis of the prompt (e.g., additional input is used to explicitly indicate which words to focus on). In this family, we consider 3 methods: Attend and Excite (A F) (Chefer et al. (023b), Structured Diffusion Guidance (SDG) (Feng et al. 20236) and Semantic-aware Classifier-Free Guidance (SCG) (Shen et al. 2024)
We remark that, albeit in different context, this selection resembles an image retrieval task Krojer et al. 2023
We exclude BLIP-VQA for the fine-tuning set composition avoid biasing the evaluation (Huang et al. 2023).
' 'LoRA adapters (Hu et al. 2021) and fine-tuning also the CLIP- based text encoder do not provide performance improvements (Appendix E.2). Likewise, creating multi-category model by "merging" different per-category models
using a fine-tuming set composed with images from all categories do not provide performance gains (Appendix E.3
7

---

Published as a conference paper at ICLR 2025
Table 1: Alignment results (%). Gray highlighted style when MI-TUNE outperforms all competitors; Grayed text for under-performing methods per-family; Green heatmaps show per-category absolute gains w.r.t the base model.
BQA
RPS
Human (sreer stody)
Method
Culer
Stope
Testue
2D-5p
Sen-Sp. Compl.
(angg)
Slops
Testur
2D-5p.
Non-Sp.
Coupl.
Color
Shuge
Texure
2D-Sp.
Spr-Sp. Cumpl.
80-2.1-beac
49.68
42.71
49,99
15.77
66.23
50.53
(45.81)
27 27.64 24.56
24.99
27.50
26.66
25.0
(26.17)
11,90
40:48
35.71
66.67
29.76 (35.71)
A&BB
61.43
47.39
61.10
16.18
66.21
$1.69
(51,17)
28.44
24,43
25.88
28.12
26.60
(2)
31,95
15.48
52.38
32.14
65.08
30.95
(38.06)
S13G
47:15
45.24
47.13
15:25
66.17
47:41
(44.77 72)
27.2
24.40
24.71
27:10
26. 12
25
125 90)
26.19
15.48
18 10
38.10
61 11.90
29.76
(34 92)
SCG
49.82
43.28
50.16
16.31
66.60
51.07
(46.21)
27.86
24.85
25,57
27.76
26.98
26.03
(26.51)
20.24
11.90
33.33
40.48
69.05
39.29
(35.71)
DPOK
53.28
45.63
52.84
17.19
66.95
51.97
(47.98)
28.20 28.20 24,99
25.44
28.12
26. 80
25 55
(26.57)
23.81
16.67
47.62
34.52
70.24
38.10
(38.49)
GORS
53.59
43.82
$4.47
15.66
28
(47.89)
23.
34.79
25.56
26
26.07
54.52
14.29
48.81
36.90
(3849)
HN-ITM
46.51
15:24
(44.28)
60)
19.05
30.95
(27.58)
MI-TUNB
65.04
50.08
(65.82
118.51
67.77
[54.17
(53.56)
29
19
25.57
26.20
128.50
27.15
(26
(27:21)
66.43
25.01
133.19
145.24
73.81
(46.43
(98.35)
her Infor. Class
11.78
4.68
14.11
0.54
0.37
1.16
[5.44]
0.80
0.29
0.89
0.92
0.32
0.33
(0.59)
2.19
3.58
11.90
4.77
2.38
9.53
(5.72)
2,92
4.48
1.42
1:24
1:75
12.62)
0.56
0.43
0.57
0.62
0.22
0.37
(0.45)
4.76
8.33
3.57
8.34
(5.36)
MB- baae
15.39
17.37
15.83
3.64
17.25)
1.01
1.21
1.00
(1.03)
13:11
12.71
7.14
16.67
(22.64)
MI-TUNE Neati
3.61
7,69
1.72
1.32
0.30
1,89
(1 (1)
0.60
0.58
0.32
0.08
(0:41)
11:91
5.96
0.81
4.76
3.57
7.14
(5.69)
MI-TENE: Arote
5.88
5.68
2,68
7.68
0.44
3.62
(4.33)
2.43
2.52
1.24
0.28
0.63
2.42
(1.55)
34.50
31.29
1.55
11.76
5.08
18.17
(17.06)
A B indicates the absolute difference between A and B: A % corresponds to the percentage difference (A B)/ B: 1: Finc-tuning sot obtained
from SPRIGHT racher than SD-2. I-base; Human scores do not sum to 100 in each
as users can select multiple methods for each question.
Fine-tuning methods. Alternatively, a pre-trained model can be fine-tuned with adapters (Hu et al. 2021 optimized via a variety of RL or supervision methods. Specifically, we consider approaches: Diffusion Policy Optimization with KL. regularization (DPOK) (Fan et al. (203)), Generative mOdel finetuning with Reward-driven Sample selection (GORS) (Huang et al. (003) and Hard-Negatives
(HN-
ITM) (Krojer et al. 2023). Notice that since results in the literature for both families do not necessarily refer to same base models, to guarantee a fair comparison, we adapted and evaluated all methods on SD-2, -base.
4.4 RESULTS
Comparing methods. Table bottom part of the table summarizes (i) the absolute gain with respect to the SD-2.1-base model for each of the best methods in each family and (ti) the percentage gains of MI-TUNE with respect to the alternative method for each category. We also summarize performance as averages across categories for each metric.
reports the alignment results T2I-CompBench. To simplify its reading, the
Despite performance varies, MI-TUNE achieves new state of the art across all categories/metries, often by a sizable margin. While this is more evident for BLIP-VQA and Human, the literature shows that HPS has natural small variations (see Appendix D) hence MI-TUNE gains are significant also for this metric.
Table Tresulss are obtained generating fine-tuning sets from SD-2. bbase for all tasks but 2D-Spatial. For this category, we were able to obtain (at best) BLIP-VQA=15.99 15.93 and HPS=28.13 13. Conversely, generating the fine-tuning images from SPRIGHT resulted beneficial. We can link this result to the self-supervision nature of MI-TUNE. On the one hand, our methodology is not bounded to a specific model. On the other hand, the filtering operated via point-wise MI estimation can benefit from "pre-alignment" MI-TUNE can strengthen existing alignment but might not be sufficient to "induce" it. Notice that all competitors suffer from this trade-off too as no single winner emerges. In particular, despite A&E and GORS are the most frequent best method in their family (winning in 10-out-of-18 scenarios), all competitors show less consistent performance across categories and metrics than MI-TUNE. For instance, for attribute binding (color, shape and texture). fine-tuning methods under-perfone according to BLIP-VQA and Human, but the performance gaps are very close considering HPS. Yet, MI-TUNE achieves consistently higher performance across all categories, outperforming alternative fine-tuning methods by a large margin.
Raw alignment performance apart, it is important to highlight MI-TUNE key differences compared the alternative fine-tuning methods. DPOK uses RL with a reward model (pre-trained with human-labeled real images) to define prompt-image alignment score to guide the fine-tuning, HN-ITM uses a contrastive
8

---

Published as a conference paper ICLR 2025
learning approach based on an ad-hoc dataset with real positive (good alignment) and negative (poor alignment) prompt- pairs, and GORS composes a fine-tuning set generating images from the diffusion model and selecting them based on VLA.-VQA. While GORS is very close in spirit MI-TUNE, its performance explicitly acknowledged by its
"biased" the filtering criteria overlaps with the final evaluation strategy authors (Huang et al., 2023). Overall, while both DPOK and GORS still require external assistance, MI-TUNE generates images and selects them using the target model itself, i.e., it is the first fully self-supervised model for T2I alignment to the best of our knowledge.
Round
Table 2: Comparing image quality/variety scores.
60
1
SUPV04
66
2
MI-TUNE (R=2, CFG=2.5)
DALLE- 31 IMAGEN-3 3 SDXL.
Motric 1D-2.1-base Color Shape Texture Spatial Non--pp Comp. (arg)
CPG
[DDL))
16.8
17.3
18.8
16.8 20.6 (18.7
20.1
13.2
2.5
FD-DINOL)
229.1 279.0 236.9
250.4
251.7
251.9
255 (250.9)
284.4
213.9
185.6
30
CMMD(L)
0.611 0.681 0.634
0.694 0,669
0,709 0.671 (0.680)
0.894
0,854
0.898
FID
5.0
Results from samples of MS-COCO-2011 volidation set; results from Imagen- et
2024
Figure 2: Hyper-params search.
Alignment/image quality-variety trade-off's MI-TUNE results in Table I are obtained from a grid search across multiple fine-tuning rounds and CFG values. In fact, we observe different trade-offs between alignment and image quality across different configurations. We exemplify this in FFguree 2 for the Color category. The figure highlights two opposite dynamics: T2I alignment benefits from multiple fine-tuning rounds (higher BLIP-VQA) but can introduce image artifacts and reduce measured diversity (higher FID). While this trade-off is neither mentioned nor quantified in the literature of the considered methods, it is to be expected strictly abiding to a prompt impacts the "generative pathways" at sampling time. Interestingly, lowering CFG (typically set to 7.5) counterbalances these dynamics and enables a "sweet spot" - as the model better aligns category thanks fine-tuning, one can alleviate the guidance scale dependency at generation. Table 2 complements this analysis by showing FID,, FD-DINO and CMMD scores for all categories, as well for SD-2. 1-base and three state of the art models- while all metries indeed suggest a possible reduction in MI-TUNE scores are comparable with other statc-of-the-ar models (sce Tigure for example images).
image variety considering
Table 3: FT set selection.
Table 4: Alignment (%) using SDXL..
Table 5: DiffusionDR.
BLIP-VQQ HPS
BLIP-YQA
HPS
Model
HPS
Stratogy
Stope
Color Stape
Method
Color Shape Vestare 2D Sp. Now Sp. Coup. Coloe Shape Testure 2D Noa Sp. Comp.
SD-2. 1o--se DiflisionDB
23.99
only
5000
29.13
(ref) SDXL 60.78 49,70
55.78
21.02
68.16 52.68
28.4722.99
25.85 28.50
26.64 25.90
ICS only
59.43-46.37
na. a.
SD-2.1-boe
49.65 42.71 49.09
15.77
66.23
50.53
27.64 24.56
24.09
27.50
26.66 25.70
MI-TUNE
25.33
MI-Reali0.25 25)
48.47
MI-TUNE
69.66
35.86 66.74
22,18
72.17
57.74
29,03
23.90
27.15
29.57
27.56 26.70
MI-TUNE E Base
1,33
MI-Real(0.5)
61.63
NE-TUNE Bidi 8.88 6.16
10.66
1.16
4,01
5.06
0.55
0.91
1,30
L.07
0.92 0.00
0.97
MI MllRcoa).
5983 48.92
28.00 25.60
MI-TEINE%
14.61 12.59 19
5,52
5.88
9.61
1.97
3.64
5,03
3.75
3,45
3.09
Fine-tuning set composition. The strategy to select prompt-image pairs for the fine-tuning set has a large design space beyond the use of MI. In Table 3] we report (for two categories for brevity) aligmment performance using two alternative strategies. Specifically, using HPS rather than MI degrades performance F Results when composing the fine-tuning set by mixing MI-selected and real images selected from the CC2M dataset (Changpinyo et al. 202T) are instead inconsistent (BLIP-VQQ steadily degrades but HPS signals an improvement in some scenarios).
SDXL and DiffusionDB. We complete our evaluation by presenting results obtained applying MI-TUNE on SDXL in Table 4 and considering alternative scenario closer to real user application using DiffussonDB in Table 5 to complement the synthetic nature of C2ICCompeench. As expected, "vanilla" SDXL significantly outperforms SD-2.1-base yet MI-TUNE enables sizable improvements on SDXL alignment (see Figure 4p. For the realistic alignment use case in Table 5] we select prompt-image pairs from DiffusionDB and we contrast alignment when fine-tuning using the images already paired with prompts against MI-selected
"We compute only BLIP-VQA avoid evaluation bias (Huang et al. 2023
9

---

Published as conference paper ICLR 2025
ones. We use 3D-2.1-base base model and report only HPS scores in Table 5 Overall, fine-tuning with DiffusionDB images improves the base model, yet MI-TUNE enables superior performance (see Figure 5
cons
A&R
SDXL.
(Color) "a blue bear and brown boat"
(Coler) "A green apple and brown horse"
jacked
(Shape) "a cubic block and a cylindrical jar of honey"
"A
circular
(Texture) "a metuflic desk and fbuffy blanke!"
Figure 4: Qualitative examples from Table (same seed used for given prompt). More examples in Appendix G
Fine Fine-tuued using images
(2D-Spatial) "a nabbit near bieycle"
(Non-spatial) "A gardener is prining beautiful bonsat tree."
(Human pompl) boaly milkartine jollyfish ss Feadl malinii illaatndion lackll. Satbay, Eigttngg
(Complex) "The white oug on top of the black couster."
Figure Qualitative examples from Table (same seed used for a given prompt). More examples in Appendix F
headphones pommal mucha"
painting Killa
Figure 5: Qualitative examples from Table (same seed used for given prompt). More examples in Appendix HI
5 CONCLUSION
T21 alignment emerged an important endeavor to steer image generation to follow the semantics and user intent expressed through a natural text prompt, as can save considerable manual effort. this work, we presented novel approach to improve model alignment, that uses point-wise MI between prompt-image pairs as a meaningful signal to evaluate the amount of information "flowing" between natural text and images. We demonstrated, both qualitatively and quantitatively, that point-wise MI coherent with existing alignment measures that either use auxiliary VQA models or elicit human intervention.
We presented MI-TUNE, lightweight, sule-supe fine- fine-tuning method that pre-trained T21 model such as SD to estimate MI, and to generate a synthetic set of aligned prompt-image pairs, which then used in parameter-efficient fine-tuning stage, align the T21 model. Our approach does require human annotation, auxiliary VQA models, nor costly inference-time techniques, and achieves a new state-of-the-an across all categories/metries explored in the literature, often by sizable margin. These results carry in more complex tasks, and for various base models, illustrating the flexibility of MI-TUNE.
'The higher prompt complexity does well suit BLIP-VQA text decomposition (see Appendix H.I
10

---

Published as a conference paper ICLR 2025
REFERENCES
Negative prompts. di discussions /857 2022.
Aishwarya Agarwal, Srikrishna Karanam, Joseph, Apoorv Saxena, Koustava Goswami, and Balaji Vasan Srinivasan. synthesis. In Proceedings of the IEEE/CVF October 2023.
A-star: Test-time attention segregation and retention for International Conference on Computer Vision (ICCV), pp.
Alexander A Alemi Ian Fischer. Gilbo: One metric measure them all. Advances in Neural Information Processing Systems, 31, 2018.
Alexander Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. In International Conference on Learning Representations, 2016.
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miik Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, and Ming Yu Liu, ediff-i: eoxttto-image diffusion models with ensemble expert denoisers. arXiv preprint arXiv:2211.0333 2022a.
Yogesh Balaji, Scungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, Ming- Liu. ediff-i: Text-to-image diffusion models with ensemble of expert denoisers. arXiv preprint arXiv:221..01324 2022b.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In Proceedings of the 35th International Conference on Machine Learning, 2018.
Anthony J Bell and Terrence Sejnowski. An information maximization approach blind separation and blind deconvolution. Neural computation, 7(6):1129-113) 1995.
Rob Brekelmans, Sicong Huang, Marzyeh Ghassemi, Greg Ver Steeg, Roger Baker Grosse, and Alireza Makhzani. Improving mutual information estimation with annealed and energy-based bounds. International Conference on Learning Representations, 2022.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image- pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3558-3568, 2021.
Agneet Chatterjee, Gabriela Ben Melech Stan. Estelle Aflalo, Sayak Paul. Dhruba Ghosh, Tejas Gokhale, Ludwig Schmidt, Hannaneh Hajishirzi. Vasudev Lal, Chitta Baral, and Yezhou Yang. Getting it right: Improving spatial consistency in text-to-image models, 2024.
Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or, Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. ACM Trans. Graph., 42(4), 2023a.
Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen Or. aitend-and-exeite Attention basodd semantic guidance for text-to-iman to-i diffusion models, 2023b.
Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Picter Abbecl. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems, 29, 2016.
Kevin Clark, Paul Vicol, Kevin Swersky, and David J. Fleet. Directly fine-tuning diffusion models differentiable rewards. In The Twelfth International Conference on Learning Representations, 2024. URL RRUUtttps://ppeeeeeee het/forume d- SEVL 1
Colin Conwell and Tomer Ullman. Testing relational understanding text-guided image generation, 2022.
Omer Dahary, Or Patashník, Kfir Aberman, and Daniel Cohen-Or. Be yourself: Bounded attention for multi-subject text-to-image generation, 2024.
11

---

Published as a conference paper ICLR 2025
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. In Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 8780-87887 8794. Curran Associates, Inc., 2021.
Ying Fan, Olivia Watkins. Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Chavamzadeh, Kangwook Lee, and Kimin Lee. Reinforcement learning for fine-tuning text-to-image diffu- sion models. Thirty-seventh Conference on Neurul Information Processing Systems, 2023. URL https
//apenrevicw. net/ Ltte//ooommmmdd1Pam00
Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Reddy Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-free free structured diffusion guidance for compositional text-to-image synthesis. In The Eleventh International Conference Learning Representations, 2023a. URL dttpps://pperreee
net/ worumoidriiii............ Dorumiid-PIIq[T4r25
Weixi Feng, Xuchai He, Tsu-Jui Fu, Varun Jampani, Arjun Reddy Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-free structured diffusion guidance for compositional text-to-image synthesis. In The Eleventh International Conference on Learning Representations, 2023b. URL https: https://openreviis openreview
net/ Doruuiidd-UUqq-AAR
Giulio Franzese, Mustapha Bounoua, and Pietro Michiardi. MINDE: Mutual information neural diffusion estimation. In net/
The Twelfth International Conference on Learning Representations, 2024. URL
forum? deammiiddkkwdeae doOkdd3..
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scene: Scene-based text-to-image generation with human priors, 2022.
Brian Gordon, Yonatan Bitton, Yonatan Shafir, Roopal Garg, Xi Chen, Dani Lischinski, Daniel Cohen-Or, and Idan Szpektor. Mismatch quest: Visual and textual feedback for image-text misalignment, 2023.
Paul Grimal, Hervé Le Borgne, Olivier Ferret, and Julien Tourille, Tiam a metric for evaluating alignment text-to- image generation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 2890-2899, January 2024.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A reference-fee free evaluation metric image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: pww.ddiigg........///oo........e neur wttp://ppp.cccppppreeeeeeee/e//p (1lc/Ba.dd4477780ftttt5553 0tt.aooooo..o0.111777799292202
R Devon Hjelm, Alex Fedorov, Samuel Awvoie-arcchlloor Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference Learning Representations, 2019.
Jonathan Ho and Tim Salimans. Classifier-free difffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021. ut//:tttss///eeeerrreeee//e///ts
YbI
Jonathan Ho and Tim Salimans. Classifier- free diffusion guidance. arXiv preprint arXiv:2207.12559: 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems. volume 33, pp. 6840-6851, 2020.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zho Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.
12

---

Published as a conference paper ICLR 2025
Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah Smith. Tifa: Accurate and interpretable text-to-imas to-iiagee faithfulness evaluation with question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 20406-20417, October 2023.
Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: A comprehensive benchmark for open-world compositional texl-to-image generation. In seirth- Conference Neural Information Processing Systems Datasets and Benchnarks Track, 2023. URL httLitttps///peeeeiiitie net/ wwtp://eeeeetoooeeeeeeeee forum poforummid=weee B
Sicong Huang, Alireza Makhzani, Yanshuai Cao, and Roger Grosse, Evaluating lossy compression rates of deep generative models. In International Conference on Machine Learning. PMLR, 2020.
Tmagen-Team et al. Imagen 3, 2024.
Sadeep Jayasumana, Srikumar Ramalingam, Andreas Veit, Daniel Glasner, Ayan Chakrabarti, and Sanjiv Kumar. Rethinking fid: Towards better evaluation metric for image generation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9307 9315, June 2024.
Dongzii Jiang, Guanglu Song. Xiaoshi Wu, Renrui Zhang, Dazhong Shen, Zhuofan Zong Yu Liu, and Hongsheng Li. Comat: Aligning text-to-image diffusion model with image-to-text concept matching, 2024. URI. https
//arxiv.org/abs/220000 03653
Wonjun Kang, Kevin Galim, and Hyung Koo. Counting guidance for high fidelity text-to-image synthesis, 2023.
Shyamgopal Karthik, Karsten Roth, Massimiliano Mancini, and Zeynep Akata. If at first you don't succeed, try, try again: Faithful diffusion- buffusion-bbaed text-to- text-to-image generation by selection. arXiv preprint arXiv:2305. 133000300..330,,,. 2023.
M. G. Kendall. A Ner Measure of Rank Correlation. Biometrika, 30(1-2):81-93, 06 1938.
Yunji Kim, Jiyoung Lee, Jin-IWw Kim, Jung-Wee Woo Ha, and Jun-Yan Zhu. Dense text-to-image generation with attention modulation. ICCV, 2023.
Diederík P Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. httphttss: //opereviee.ee net Forum 21 d- 2LdBqxc11
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4015-4026, October 2023.
Xianghao Kong, Ollie Liu, Han Li, Dani Yogatama, and Greg Ver Steeg. Interpretable diffusion via information decomposition. In The Twelfth International Conference Learning Representations, 2024. URL https://
openreview net/ hww.:/tt//ootrttooe www.rrmmmmd....eeeoo..
Benno Krojer, Flinor Poole-Dayan, Vikram Voleti, Christopher Pal, and Siva Reddy. Are diffusion models vision-and- language reasoners? arXiv preprint arXiv:2305.1633 2023.
Kimin Loc, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback, 2023.
Nunzio Letizia and Andrea M Tonello. Copula density neural estimation. arXiv preprint arXiv:2211. 2022.
Junnan Li, Dongxu Li, Silvio Savarese, Steven Hoi. Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In Proceedings of the 40th International Conference on Machine Learning, 2023a.
Yumeng Li, Margret Keuper, Dan Zhang, and Anna Khoreva. Divide & bind your attention for improved generative semantic nursing. In 34th British Machine Vision Conference 2023, BMVC 2023, 2023b.
13

---

Published as a conference paper ICLR 2025
Tsung Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitiick, and Piotr Dollár. Microsoft coco: Common objects in context, 2015, URL https
/ //axxiv. abs 1405. 0312
Nan Liu, Shuang Li. Yilun Du, Antonio Torralba, Joshua B. Tenenbaum. Compositional visual generation with composable diffusion models. In Computer Vision - EECV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XVII, pp. 423-439, 2022.
Shih- Yang Liu, Chien-Y Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-dccomposed low-rank adaptation, 2024.
Vivian Liu Lydia Chilton, Design guidelines for prompt engineering text-to-image generative models. Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, 2022.
Jian Ma, Junhao Liang, Chen Chen, and Haonan Lu. subjeet-diifussionoooenn domain personalized text-to-image generation without test-time fine-tuning, 2023.
David JC MacKay. Information theory, inference and learning algorithms. Cambridge university press, 2003.
Shweta Mahajan. Tanzila Rahman, Kwang Moo Yi, and Leonid Sigal. Prompting hard or hardly prompting: Prompt inversion for eext-oo-image diffusion models, 2023.
Oscar Mañas, Pietro Astolfi, Melissa Hall, Candace Ross, Jack Urbanek, Adina Williams, Aishwarya Agrawal, Adriana Romero-Soriano, and Michal Drozdzal, Improving ext-to--mage consistency via automatic prompt optimization, 2024.
David McAllester and Karl Stratos. Formal limitations on the measurement of mutual information. In International Conference on Artificial Intelligence and Statistics, 2020.
Tuna Han Salih Meral, Enis Simsar, Federico Tombari, and Pinar Yanardag. Conform: Contrast is all you need for high-fidelity text-to-image diffusion models. In Proceedings of the IEFE/CVF Conference on Computer Vision and Puttern Recognition, 2024.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, llya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text text-guided diffusion models, 2022.
Michael Ogezi and Ning Shi. Optimizing negative prompts for enhanced aesthetics and fidelity text-to-image generation, 2024.
Bernt Øksendal, Stochastic differential equations. Springer, 2003.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. Advances neural information processing systems, 2018.
OpenAl. Chatgpt (gpt-4) [large language model], 2024. attps:t/:://cat http://ca............ooa openal c cott chat
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafranice, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaacldin El-Nouby, Mido Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Po-Yao Huang. Shang-Wee Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL
net. forum? id=a 68SUt 62Ft
Liam Paninski. Estimation of entropy and mutual information. Neural computation, 156):1191-1253, 2003.
George Papamakarios, Theo Pavlakou, and lain Murray. Masked autoregressive flow for density estimation. Advances in neural information processing systems, 30, 2017.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhor,, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for rigo-rcsotution image synthesis. In The Twelfth International Conference Learning Representations, 2024. URLLhtttss:///peereeee net. forum? 2immmid-aaa 52
14

---

Published as a conference paper ICLR 2025
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh. Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, volume 139, 2021.
pp.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents, 2022.
Royi Rassin, Eran Hirsch. Daniel Glickman, Shauli Ravfogel, Yoav Goldberg, and Gal Chechik. Linguistic binding in dif- fusson models: Enhancing attribute correspondence through attention map alignment. In Thirty-seventi Conference on Neural Information Processing Systems, 2023. URL
nct/ f -d-meiid--------
Benjamin Rhodes, Kai Xu, and Michael U Gutmann. Telescoping density-ratio estimation. Advances neural information processing systems, 2020.
Robin Rombach, Andreas Blattmann, Dominik Lorenz. Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent difffusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 10684-10695, June 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. Medical Image Computing and Computer-Assisted Intervention MICCAI 2015, pp. 234-241.22 2015.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photoreal- istic text-to-image diffusion models with deep language understanding. Advances Neural Information Processing Systems, volume 35, pp. 36479 36494, 2022.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models, 2022.
Dvir Samuel, Rami Ben-Ari, Simon Raviv, Nir Darshan, and Gal Chechik. Generating images of rare concepts using pre-trained diffusion models. 2024.
Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3)3379-433, 1948.
Dazhong Shen, Guanglu Song, Zeyue Xue, Yun Wang, and Yu Liu. Rethinking the spatial inconsistency in classifier- free diffusion guidance, 2024.
Jascha Sohl- -iek-Dicsstenn, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequi- librium thermodynamics. In Proceedings of the 32nd International Conference on Machine Learning, volume 37, pp. 2256-2265, 2015.
Jiaming Song and Stefano Ermon. Understanding the limitations of variational mutual information estimators. In International Conference on Learning Representations, 2019a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution, In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Buc, F. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019b.
Yang Song and Stefano Ermon. Improved techniques for training score- score-based generative models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Procesing Systems, volume 33, pp. 12448.
Curran Associates, Inc., 2020.
Yang Song, Jascha Sohl- Dicckstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score- based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
Karl Stratos. Mutual information maximization for simple and accurate part-of-speech induction. In Proceedings of the 2019 Conference of the North American Chapter the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Papers), 2019.
15

---

Published as a conference paper at ICLR 2025
Jiao Sun, Deqing Fu, Yushi Hu, Su Wang, Royi Rassin, Da-Cheng Juan, Dana Alon, Charles Herrmann, Sjoerd van Steenkiste, Ranjay Krishna, and Cyrus Rashtchian. Dreamsync: Aligning text-to-image generation with image understanding feedback, 2023.
Raphael Tang, Linging Liu, Akshat Pandey, Zhiying Jiang, Gefei Yang, Karun Kumar, Pontus Stenetorp. Jimmy Lin, and Ferhan Ture. What the DAAM: Interpreting stable diffusion using cross attention. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pp. 5644-5659, July 2023.
Bram Wallace. Meihua Dang. Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam. Stefano Ermon, Caiming Xiong. Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization, 2023.
Zijie J. Wang, Evan Montoya, David Munechika, Haoyang Yang, Benjamin Hoover, and Duen Horng Chau. DiffusionDB: A large-scale prompt gallery dataset for text-to-image generative models. arXiv:2210. 14896 [es], 2022. URL
https://arxiy org/ abs /awss/221.... 14996
Zijie J. Wang, Evan Montoya, David Munechika, Haoyang Yang, Benjamin Hoover, and Duen Horng Chau. DiffusionDB:
A large-scale prompt gallery dataset for text-to-image generative models. In Proceedings of the 61 Annual Meeting the Association for Computational Linguistics (Volume 1: Long Pupers), 2023a.
Zirui Wang, Zhizhou Sha, Zheng Ding, Yilin Wang, and Zhuowen Tu. Tokencompose: Grounding diffusion with token-level supervision, 2023b.
Sam Witteveen and Martin Andrews. Investigating prompt engineering diffusion models, 2022.
Qiucheng Wu, Yujian Liu, Handong Zhao, Trung Bui, Zhe Lin, Yang Zhang, Shiyu Chang. Harnessing the spatial- text-to-imag synthesis. In Proceedings of the IEEE/CVF
temporal attention of diffusion models for International Conference on Computer Vision (ICCV), pp. 7766-7776, October 2023a.
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis, 2023b.
Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score: Better aligning text to to-image models with human preference, 2023c.
Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score: Better aligning text-to-image models with human preference, 2023d.
Xindi Wu, Dingli Yu, Yangsibo Huang, Olga Russakovsky, and Sanjeev Arora. Conceptmix: A compositional image generation benchmark with controllable difficulty, 2024. URL
14339
Jiazheng Xu, Xiao Liu. Yuchen Wu, Yuxuan Tong. Qinkai Li, Ming Ding, Jie Tang. and Yuxiao Dong Imagereward: Learning and evaluating human preferences for text-to-image generation. In Thirty-sevnnth Conference on Neurul Information Processing Systems, 2023. L
net Seet///ormmmi--deee000a
Huizhuo Yuan, Zixiang Chen, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning of diffusion models for text-to- to-image generation, 2024.
Xinchen Zhang, Ling Yang, Yaqi Cai, Zhaochen Yu, Kai-Ni Wang, Jiake Xie, Ye Tian, Minkai Xu, Yong Tang, Yujiu Yang, and Bin Cui. Realcompo: Balancing realism and compositionality improves text-to-image diffusion models, 2024a. URL
abs abs//402....2 1290E
Yang Zhang, Rui Zhang, Xuecheng Nie, Haochen Li, Jikun Chen, Yifan Hao, Xin Zhang, Luoqi Liu, and Ling Li. Spditfusion: Semantic protection diffusion for multi-concept text-to- -mage generation, 2024b. URL https
7/ farxiv. abs 2409. 01327
Lirui Zhao, Yue Yang, Kaipeng Zhang, Wenqi Shao, Yuxin Zhang, Yu Qiao, Ping Luo, and Rongrong Ji. Diffagent: Fast and accurate ext-to-image api selection with large language model. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 6390-6399, June 2024.
16

---

Published as conference paper ICLR 2025
Shengjia Zhao, Jiaming Song. and Stefano Ermon. lagrangian perspective on latent variable generative models. Proc. 34th Conference on Uncertainty Artificial Intelligence, 2018.
Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl. Simple multi natti-dataset detection. CVPR, 2022.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. MiniGPT-4: Enhancing vision- lsionnlanguage under- standing with advanced large language models. The Twelfih International Conference Learning Representations, 2024.
net/ Forum 1d=1t2bq88127
17

---

Published as a conference paper ICLR 2025
Appendices
Details on MI estimation Proof of Equation (2)
Referenced
19
B Details on user study
20
B.1 Comparing alignment metries Preliminary analysis to understand MI meaningful aligroent signal Referenced
B.2 Comparing alignment methods
Actual benchmark of MI-TUNE against alternative methods Referenced in
Experimental protocol details
List of parameters used and computation considerations. Referenced in
A.J
D HPS scores range
Discussion about the natural small values range provided by HPS Referenced
E Additional results and ablations
24
E.1 Ablation: Fine-tuning set selection strategies
Discussing alternative strategies to MI for composing the fine-tuning set Related to Table
E.2 Ahlation: Fine-tuning model adapters and modalities
Investigatingg LoRA, DoRA and fine-tuming
CLIP Referenced in
E.3 Ablation: Combining categories into single model
Investigating policies to create monolithic model merging multiple categories Reference in 4.2
F
Qualitative examples for T21-CompBench using SD-2.1-base
F.1
Color prompts
F2
Shape prompts
F3
Texture prompts
F.4
2D-Spatial prompts
30
F5
Non-Spatial prompts
F66
Complex prompts
G Qualitative examples for T2I-CompBench using SDXL
33
Fine-tuning with DiffusionDB dataset
Selecting images BLIP-VQA prompts decomposition
Discussing BLIP-VQA limitation when handllngg DiffusionDB prompts Referenced
$4.4
H.2 Qualitative examples for DiffusionDB
35
Qualitative analysis of as an alignment measure
Expanded version Ligure Including all T21 CompBench categories
36
J BLIP-YQA, HPS and MI score distributions
Comparing distributions and MI rank Related to results in
37
18

---

Published as conference paper ICLR 2025
A DETAILS ON MI ESTIMATION
In this Section, we provide the proof for Equation (2) We start by recalling the definition of the forward and backward processes for discrete-time diffusion model. For the forward process, the following Markov chain
T
((ZOT, = (2 p) II (
g(zg 2 = (22 v1 -12
The backward process (with or without a conditioning signal p) evolves according to
T
T
po(2o:r) = p(zr) II po (zt-1 (z)), po p) = p(zr)
Il
po 2 P)
where (a 2 = (2 no (z+), Be1). with pa(z)) = voe (z+ vI ca s0(21:1) Similar expres- sions can be obtained for the conditional version.
Our goal here is to show that the following equality holds
Ep [LL [9(=0| p) {<=))]] = Ex [I(z,
which is the condition that I(z, p) of Equation (2) should satisfy to be valid point point-wise MI estimator. In particular, we will show that
B.T 20x (x.)
Ep |LL [g(z pp) || [(zo)]] = E, p.p.,.. [Ky (oll E Ez(=(,0,0) 010)1
To simplify our proof strategy, consider the ideal case of perfect training, i.e., (zo.T - a( ((=oTT,p). Moreover, since ((x 1;) = q(zz 1t, we can rewrite the KL [g(z0 p) ((z0)] term as follows
KL [0(00 p) || {(z))] = KL [g(= p) ((zorr)) = KL [po (z0:2 p) (o(o))))
/ po po(2a: log
Po( p)) dzor = / po(zu:t (p) Σ log
Po vo(2 2 Zt. P) dzur
po(
po ( 1-11 2t))
/ po (0 (f po(zz-111 ze, log po (= P), dz(-1 d
p0 2x)
/ pz( kI) 1 Ze. Z). po 21)] dz, =
Σ
28, po(zo|p)llH0(z) nz (() P/
1
2,r ar - ā²) / po(zi (o(zz/p)llcooz)) co(zi, p)]
1
88
Σ
1=1
Br,
Kel Kelle0(21, 0, c0(21. 1)]
3,T
Biz,
Ke 0,1) cz p, 1)]l
20
(t.)
which allows to prove that the quantity in Equation (2) is indeed valid wise MI estimator.
19

---

Published as conference paper ICLR 2025
green
Click to remove
+ Click to select
Click remove
Progress:
Figure 6: Weh app screenshot example of the alignment metric comparison survey.
B DETAILS ON USER STUDY
Our user studies are based small focus groups participants only, with (lightly) guided discussions led by a moderator. In those campaigns we elicited feedback from users regarding the comparison of different alignment metries, aiming understand if MI is plausible choice. Although launching large-scale survey campaigns would desirable, this would require a completely different organization and implementation with respect to what what adopted for this work.
The survey web app. Beside punctually comparing alignment metrics 3 and methods S 4.4 we designed a web app collect subjective feedback, in the form of mini surveys, from real users. Each survey is composed multiple tests, each showing a prompt and a set of images generated from Under the hood, the web app corresponds jupyter notebook with ipywi dget s for UI controls, rendered via the voi 1a framework and deployed live via a docker-ized HuggingFac space. Via the web app we run campaigns to compare alignment metrics and to compare alignment methods.
Figure 6 shows an example screenshot of the alignment metric comparison survey As from the example, users are free select from 0 to up images for each prompt. However, stress users subjectivity, we intentionally did not provide guidelines on how to handle "odd" cases (e.g., if the prompt asks for a picture of "an apple", but the picture show more than one apple). Last, each survey saved separate CSV with the timestamp of its creation which also serves as unique identifier of the survey, i.c., neither user identifier nor cookies are required by the web app logic, so users' privacy and anonymity is preserved.
B.1 COMPARING ALIGNMENT METRICS
the first surveys campaign aimed to understand how users perceive images pre-selected by BLIP-VQA. HPS and MI. Specifically, we run surveys composed of 10 tests, each showing prompt and the related best image among 50 generations (using 1--se)- as ranked according to each metric separately ( 3). Each of the 10 prompts is randomly selected from pool of 700 prompts for the 121-Combench color category, and at each test the order in which the 3 pictures is shown is also randomized.
html
20

---

Published as conference paper ICLR 2025
Table User study about comparing alignment metrics.
Metric
Campaign answers (%)
MI BLIP-VQA
UPS
Academic users
Random users
Students
avg
0
14.7
16.9
25.0
78.9
1.8
2.7
6.2
10.4
22.0
4.1
12.2
4.0
7.4
3.6
5.0
4.0
10.6
0.9
5.2
6.0
2.5
2.3
3.6
18.7
10.6
15.4
40.4
16.0
44.6
33.7
69.1
39.7
64.6
57.8
73.5
56.0
69.1
66.2
$2.2
39.9
53.2
48.2
(sclected)
(not selected)
(indifferent to the sclection)
We run surveys across three user groups: Academic users (5 members) are representative of highly informed and tech savvy users, who are familiar with how generative models work; Random users (25 members) are representative of illiterate users who are not familiar with computer-based image generation; Students (16 members) are representative of masters' level students who are familiar with image generation tools, and who have attended introductory-lever machine learning classes.
Overall, we collected 102 surveys (45, 35 and 22 surveys across days for Academics, Random users and Students respectively) which we detail in Table 6) The top part of the table breaks down all possible answers combinations. The results, although with some differences between user groups, clearly highlight that the three alignment metrics we consider in this work are roughly equivalent, with MI and BI.IP-VAA being preferred over HPS. For the Academics and Students groups, all the three images are considered sufficiently aligned with the prompt in almost half of the cases (40.4% and 44.6% respectively). Interestingly. random users select only one of the three images about 10x much more frequently than the other two groups (on average 14.2% for real users while 5.4% and 3% for Academics and Students respectively). We hypothesise that being previously exposed (or not) to the technical problems of image generation from the alignment perspective, or simply being literate (or not) about machine learning can influence the selection among the three pictures.
The bottom part of the table summarizes the answers for each individual metric. Despite the general preference for BLIP-VQA, the results corroborate once more that MI provides a meaningful alignment signal (possibly compatible with aesthetics too).
Finally, we recall that our goal in this section is to study whether MI is plausible alignment measure, rather than electing the "best" aligument metric. Indeed, this analysis does not indicate the final performance of alignment methods, which instead we report in Table
B.2 COMPARING ALIGNMENT METHODS
In this second survey campaign aimed to understand how users perceive images generated by the 8 methods we considered in our study, i... "vanilla" SD, A&E (Chefer et al. (003b)), SDG (Feng et al. 2022bb and SCG (Salimans & Ho 2022) DPOK (Fan et al. 2023), GORS (Huang et al. 2023). HN-ITM (Krojer let al. 2023) and our method Mutual Information Fine Tuning (MI-TUNE) (when used with a single round of fine- -une-nu
To do so, we run surveys composed of 2 tests for each 121-Combench category (12 rounds in total). Each test shows prompt and the 8 pictures generated using different method. For each category, randomly
21

---

Published as conference paper ICLR 2025
Table Users study comparing alignment methods. Bold shows best performance; oshows the best method per-family.
Alignment
Category (%)
Methodology
Model
Color
Shape
Texture
2D-Spatial
Non-spatial
Complex
none
SD-2. -hase
29.76
11.90
40.48
35.71
66.67
29.76
A&E
631.95
615.48
652.38
32.14
65.48
30.95
Inference-time
SDG
26.19
+15.48
38. 38.10
38.11
61.90
29.76
20.24
11.90
33.33
o40.48
o69.05
639.29
23.81
16.67
647.62
34.52
+70.24
638.10
Fine-tuning
GORS
034.52
14.29
48.81
636.90
65.48
30.95
HN-ITM
23.81
619.05
30.95
20.24
47.62
23.81
MI MU-TUNE
46.43
25.01
53.19
45.24
73.81
46.43
selected 100 prompts from T2I-Combench test set to pre-generate the pictures. run time, the web app randomly selects prompts for each category. and also randomly selects images from the related pool. Last, it randomly arranges both the tests (so that categories are shuffled) and the methods (so that pictures of a method are not visualized in the same position in the visualized grid).
Table collects
results of a campaign with 42 surveys. Specifically, the table shows the percentage of answers where the picture of given method was selected (no matter if other methods were also selected) theses results are integrated in right side of Table I and are duplicated here for completeness.
C EXPERIMENTAL PROTOCOL DETAILS
We report Table 8 all the hyper-parameters we used for our experiments.
Table Training hyperparameters.
Name
Value
Trainable model
Trainable timesteps
(5500, 1000)
PEFT
DoRA (Liu at
Rank
32
Learning rate (LR) Gradient norm chipping
le
1.0
T.R scheduler
Constant
LR warmup steps
Optimizer
Adam
Adam
-g
0.9
.gg
AdaaW weight decay
AdamW
1n
Resolution
512 x 512
Classifie--fee guidance scale
7.5
Denoising steps
Batch size
400
Training iterations
GPUs Training
NVIDIA A100
Next, we provide additional details on the computational cost of MI-TUNE, In our approach, there are two distinct phases that require computational effort:
22

---

Published as a conference paper ICLR 2025
The first is the construction of the fine-tuning set based on point-wise MI. As a reminder, for this phase, we use pre-trained SD model (namely SD-2.1-base at a resolution 512 x 512) and, given a prompt, conditionally generate 50 images, while at the same time computing point-wise MI between the prompt and each image. This is done for all the prompts in the set P. Specifically for T21-Combench, each category training set has 700 prompts, and for each prompt we generate 50 images from which we select the one with highest MI. The generation of the x0 fine-tuning set requires roughly 24 hours, i.e., about 2min per-prompt on a single A A100-80GB GPU - the 50 images are generated together (as they roughly require 50GB of the 80GB available VRAM), while each prompt is processed sequentially.
Generation:
Fine-Tuning: The second is the parameter efficient fine-tuning of the pre-trained model. Using the configura- tion discussed above, MI-TUNE requires 8 hours when using single A100-80GB GPU.
Note that (i) there is no overhead at image generation time: once a pre-trained model has been fine-tuned with MI-TUNE, conditional sampling takes the same amount of time of "vanilla" SD and (ii) while we report computational costs considering a single GPU, this is extreme scenario and the time to process the workloads scales down (almost linearly) with the number of GPUs used according observations.
D HPS SCORES RANGE
Wu et al. (2023a) report detailed benchmark of their metries across 20+ models in the HPS-v2 GitHub repository https: //ttt//gittbbb..
These details are hidden by default when loading the repository home page and need to be explicitly "opened" expanding collapsed menus (e.g.. v2 benchmark). To ease discussion, in [Tall 9 we report an extract of these benchmarks focusing on StableDiffusion as other models are out scope for our study.
Table 9: HPS benchmark across multiple Stable Diffusion models extracted for HPSSv2 GitHub repo.
Benchmark Model
Animation
Concept-Art
Painting
Photo
(avg)
SDXL Refiner (0.9)
28.45
27.66
27.67
27.46
(27.80)
SDXL. Base (0.9)
28.42
27.63
27.60
27.29
(27.73)
v2
SD (2.0)
27.48
26.89
26.86
27.46
(27.17)
SD (1.4)
27.26
26.61
26.66
27.27
(26.95)
SDXL Refiner (0.9)
33.26
32.07
31.63
28.38
(31.34)
SDXL Base (0.9)
32.84
31.36
30.86
27.48
(30.63)
v2.1
SD (2.0)
27.09
26.02
25.68
26.73
(26.38)
(1.4)
26.03
24.87
24.80
25.70
(25.35)
Results refer to two benchmark and are visually split between SD and SDXL. The columns Animation, Concept-Art, Painting and Photo are different images style, while (avg) reflects average by row.
Both versions of the benchmark present similar takeaways which we can summarize in two main observations. Specifically, (i) different versions of the same model present < 0.5 differences and (it) SDXL outperforms SD of about +1 point- - the variation of HPS scores is extremely contained even if these models are different generations apart.
Our HPS scores in Table present similar properties, but other literature (e.g., Table 2 in Zhao et al. (0044) present similar evidence.
23

---

Published as a conference paper ICLR 2025
E ADDITIONAL RESULTS AND ABLATIONS
E.I ABLATION: FINE-TUNING SET SELECTION STRATEGIES
Fine-tuning set selection strategy. It important to stress that creating fine-tuning dataset using the very same metric used for the final evaluation can artificially introduce bias stated in (Huang et al. 2023): "calculating the rewards for GORS with the automatic evaluation metrics can lead biased results".
The selection strategy to compose the fine-tuning dataset is directly related to alignment scores and different fine-tuning methods opt for different choices. Specifically: HN-ITM uses an ad-hoc dataset with real positive and negative pairs; GORS uses synthetic dataset with selection, but the fine-tuning loss of each sample is weighted by BLIP-VQA DPOK synthesizes new images at each training iteration since it is an online RL fine-tuning approach, and uses pre-trained human preference model for reward. (Tbll 1] in the main paper, shows alternative fine-tuning strategies based on synthetic generated data using variety of selection scores: GORS and DPOK are the closest methods to MI-TUNE from this point of view, yet generally underperforming compared to it.
Table 10: FT set selection.
For completeness, perform experiment where fine- tune based on a dataset selected via HPS scores, Results in Table 10 (same Table 3] but duplicated here for simplicity) show that selecting fine-tuning samples based on MI outper- forms such an alternative strategy, using BLIP-VQA.
BLIP-VQA
HPS
Strategy
Color Shape
Color Shape
MI only
65.04 50.08 19.1322.55
HPS only
59.43 46.87
n.a. n.a.
Next, another natural question to ask is whether the self- supervised fine-tuning method we suggest this work is a valid strategy. Indeed, instead of using synthetic image data for fine-tuning the base model, it is also possible to use real-life,
MI+Real(0.22 25)
61.34
48.47
29.16 25.87
MI+Real(O.S)
61.63
49.50 29.38 25.92
MI+Real(0.9)
59.83 48.92 28.60 25.60
captioned image data. Then, we present an ablation on the use of real samples, along with synthetic images, in the fine-tuning procedure. In Table (ooottom) we report the experimental results obtained by composing the fine-tuning dataset by imposing the ratio of images generated by the SD model to z, and the ratio of real images taken from the CC12M dataset (Changpinyo et al. 2021 to (1 z), where in both cases we select the candidate images to be used in the fine-tuning set S using MI. So, for example, M1+Real(0.25) indicates that we use 25% of real images. Interestingly, we observe the following trend. Complementing the synthetically generated samples with few real ones does not benefit alignment (lower BLIP-VQA) but might have positive effect for aesthetics (higher HPS).
Fine-tuning set size. We continue by reporting an ablation on the fine-tuning set S size.
Specifically, based on Algorithm | two parameters determine both the quality and the associated computa- tional cost related to the fine-tuning set the number candidate images M, and how many k are selected to be included in S.
Table 11: BLIP-VQA alignment results T21-CompBench's Color and Shape categories varying size and composition fine-tuning set. Results obtained using R=1.
Hyper- Hyper-params
Category
M
k
Color
Shape
30
58.12
47.48
50
7
59.31
47.26
61,57
48.40
100
60.12
47.80
500
59.28
46.79
24

---

Published as a conference paper ICLR 2025
Table 11 shows that the best performance is obtained selecting 2% images image out of 50). We repeated the finetuning experiments on the categories Color and Shape by varying the selection ratio in the ranges {7/50, 1/30, 1/100, 1/500). Results indicate that the best selection ratio is the middle-range corresponding to the baseline MI-TUNE. We hypothesise that higher selection ratios pollute the fine-tuning set with lower quality images, while more selective threshold favours images which have the highest alignment but possibly lower realism. Additionally, we remark that the number M of candidate images has negligible impact, above M = 50, whereas fewer candidate images induce degraded performance. Hence, the value M = 50 is, in experiments, a sweet-spot that produces a valid candidate set, while not imposing large computational burden.
E.2 ABLATION: FINE-TUNING MODEL ADAPTERS AND MODALITIES
In this Section, we provide additional results Table 12) on MI-TUNE, concerning which part of the pre- trained SD model to fine-tune. In particular, we tried fine-tune the denoising UNET network alone and both the denoising and the text encoding (CLIP) networks. The baseline results are obtained, as described in the main paper, with Do-RA (Liu al. 2024) adapters. Switching Lo-RA layers Hu et al. (2021) incurs in performance degradation, a trend observed also for other tasks in the literature (Liu et al. 2024). Interestingly, joint fine-tuning of the UNET backbone together with the text encoder layers degrades performance well, which has also been observed in the literature Huang et al. (2023). Even if. in principle, a joint fine-tuning strategy should provide better results, as the amount of information transferred from the prompt to the image is neckle by the text encoder architecture, we observed empirically more unstable training dynamics than the variant where only the score network backbone is fine-tuned, resulting in degraded performance.
Table 12: BLIP-VQA alignment results 121-CompBench's 's Color and Shape categories finetuning different portions of the model.
Category
Model
Color
Shape
MI-TUNE DoRA
61.57
48.40
MI-TUNE LoRA
58.25
48.27
MI-TUNE UNet+Eext(ooint)
57.88
47.79
E.3 ABLATION: COMBINING CATEGORIES INTO A SINGLE MODEL
The design space for T2I alignment improvement has many options and this should call not only to investigate aligmment performance but also operational and computational costs. For instance, fine-tuning methods require to create ad-hoc models while one can argue that a Lingle/multi-pureer model might be more lean and general solution.
This calls for investigating if/how different task-specific fine-tuned models can be combined into single model to address the different tasks at once. For the T2I-Combench, we considered two design options:
1. Weights merging: the DoRA weights of the 6 distinct per-category models are "merged" doing their arithmethic means forming a new "meta" model.
2. Joint optimization: we create a new "meta" model by running a single fine-tuning process but using the union of the category-specine fine-tuning set.
25

---

Published as conference paper ICLR 2025
61.5
48.5
61.0
Color
48.2
60.5
48.1
60.0
48.0
59.5
0.0
0.2
0.4
0.6
1.0
0.0
0.2
0.4
0.6
0.8
1.0
A
A
Figure 7: Weights merging: Ax Color (1.0 A) Shape.
To start from a reference example, Fig. 7 reports the BLIP-VQA obtained when testing on the color and shape test sets the merged model obtained of the two task-specific models. The hyper-parameter À is used balance merging. For instance, at À = O, performance on color (left plot) are obtained using the shape-only model. Overall, the results show that these two categories (partially) conflicting across all X values. Yet, a performance trade off might be sufficient in some scenarios.
Table 13: Benchmarking strategies for combining models.
MI-TUNE
Color (BLIP-VQA) (BLIP-VQA)
Shape
Texture (BLIP-VQA) (UNIDET)
2D-Spatial
Non-spatial Complex (BLIP-VQA)
variants
(BLIP-VQA)
from Table
61.57
48.40
58.27
18,51
67.77
53.54
Model weighting
58.50
48,23
58.22
16.72
68.28
54.35
Joint optimization
60.35
47.73
57.96
18.44
69.68
54.88
We then extended the analysis across all categorics using a simple arithmetic mean for model merging, i.c., all models have the same weight. Results reported in Table 13 using MI-TUNE as reference. Overall, for most categories, the single "meta" model has degraded performance and neither weights merging nor joint optimization the best alternative across all categories.
26

---

Published as conference paper ICLR 2025
F QUALITATIVE EXAMPLES FOR 121-COMPBENCH USING SD-2 -BASE
F.1 COLOR PROMPTS
SD-2.1-base
DPOK
GORS
HN-ITM
A&E
SDG
SCG
MI-TUNE
black apple and green backpack
red apple and green train
blue bear brown boat
green banana and brown horse
green bench blue bowl
green bench and a blue bowl
green banana red suitcase
green banana and red suitcase
Figure 8: Qualitative examples for Color from Table (same seed used for given prompt).
27

---

Published as conference paper ICLR 2025
F.2
SHAPE PROMPTS
SD-2.1-bas.
DPOK
GORS
HN-ITM
A&E
SDG
SCG
MI-TUNE
big hippopotamus a small mouse
a big hippopotamus and small rabbit
big and small mouse
cubic block and cylindrical jar of honey
O
round bagel rectangular toaster
an oval picture frame and rectangular painting
an oval picture frame and square painting
big whale and small dolphin
Figure 9: Qualitative examples of the Shape category from Table (same seed used for given prompt).
28

---

Published as conference paper ICLR 2025
F.3
TEXTURE PROMPTS
SD-2.1-base
DPOK
GORS
HN-ITM
A&E
SDG
SCG
MI-TUNE
fabric bag and leather chair
a fabric dress and glass vase
metallic lamp a fluffy blanket metallic earring and leather sofa
a plastic bottle and leather wallet
rubber tire and fabric pillow
rubber tire and metallic fork
a rubber tire and plastic bottle
Figure 10: Qualitative examples of the Texture category from Table seed used for given prompt).
29

---

Published as conference paper ICLR 2025
F.4 2D-SPATIAL PROMPTS
SD-2.1-base
DPOK
GORS
HN-ITM
A&E
SDG
SCG
MI-TUNE
bicycle top of turtle
bird on the top of person
chicken on the girl
clock on the top of a sheep
a cup right a dog
A
horse on side of bicycle
man on the right lamp
sheep on the left of lamp
Figure 11: Qualitative examples of the 2D-Spatial category from Table
seed used for given prompt).
30

---

Published as conference paper ICLR 2025
F.5
NON-SPATIAL PROMPTS
SD-2.1-base
DPOK
GORS
HN-ITM
A&E
SDG
SCG
MI-TUNE
child is jumping on trampoline their backyard.
A gardener is pruning a beautiful bonsai tree.
man holding soldering iron and repairing broken electronic device.
A person is holding a pencil and sketching portrait.
person practicing their juggling skills at circus.
A woman is practicing her violin in her music room
The paper crafter is making paper flower bouquet.
A child is playing with toy construction set and building tower.
Figure 12: Qualitative examples of the Non-spatial category from Table (same seed used for given prompt).
31

---

Published as conference paper ICLR 2025
F.6
COMPLEX PROMPTS
SD-2.1-base
DPOK
GORS
HN-ITM
A&E
SDG
SCG
MI-TUNE
The red hat was on top of the brown coat.
TI
The hat was on top of brown coat.
The red hat was on top of the hrown coat.
The white mug on top the black coaster.
The striped rug was on top of the tiled floor.
The striped rug was on top of the tiled floor.
The sleck black laptop sat on the clean white desk.
The black chair was on the left the white table.
Figure 13: Qualitative examples of the Complex category from Table (same seed used for a given prompt).
32

---

Published as conference paper ICLR 2025
G
QUALITATIVE EXAMPLES FOR T2I-COMPBENCH USING SDXL
SDXL
MI-TUNE
SDXL
MI-TUNE
SDXL
MI-TUNE
SDXL
MI-TUNE
Color prompis.
black cat and gray red backpack and blue chair
red bowl and blue train
blue bench and bowl
Shape
prompts.
hig elephant and small
big lian and small
circular mirror and triangular shelf
tall skyseraper and short cottage
prompts.
bag glass
glass jacket glass plate
jacket glass
2D-Spatial
prompts.
bicycle girl
top balkuon
top chicken
Non-Spatial prompis.
A dog is chasing after a wagging its tail
gardener pruning beautiful bonsai tree
person is looking at display A person looking at sculpture vintuge clothing and admiring the fashion
garden and uppreciating the artwork
Complex prompts.
The black chair is the blue rug
The black peneiil was next the green notebook
The blue mug on top of the
The bright yellow banana contrasted with the dull brown apple
Figure 14: Qualitative examples from
Table
(same seed used for given prompt).
33

---

Published as a conference paper ICLR 2025
H FINE-TUNING WITH DIFFUSIONDB DATASET
H.1 SELECTING IMAGES AND BLIP-VQA PROMPTS DECOMPOSITION
this section, we provide additional details about using prompts created by real users, i.e., DiffusionDB.
Dataset properties. DiffusionDB was collected scraping the StableDiffusion discord channels "[...] We download chat messages from the Stable Diffusion Discord channels with DiscordCha saving them FTMM..fles, We focus channels where users can command a bot to run Stable Diffuusion Version to generate images by ryping a prompt, hyperparaneters, and the number of images [...]" (Wang et al. (022). The scraped data is then packaged into parquet files (containing metadata such prompt, image filenames and hyperparams) and zip files (containing the actual images WebP format) and made available on HuggingFace.
Fine-tuning with DiffusionDB. We fine-tune 1--aee on 1 1,250 prompts randomly sampled and compare two different scenarios. A first dataset is composed using images provided by DiffusionDB itself. As each prompt in DiffusionDB is paired to (about) 4 generated images we obtain 5, 5,000 prompt-image pairs reference dataset. For the second dataset, we use the 1,250 prompts to generate M = 50 images for each prompt and selecting the k image with the highest MI. We repeat this procedure times construct a complementary fine-tuning dataset with prompt-image 5,000 pairs. We fine-tune SD-2. -base on each of the two datasets with pre-trained loss, then on 500 DiffusionDB prompts (again, randomly selected and disjoint from the training set prompt-image pairs) generating 10 images for each test prompt.
Table 14: DiffusionDB.
Model
HPS
Table 14 (which is duplicating here Table 5 for simplicity) shows the results.
SD-2.1-bac
23.99
Fine-tuning either using the DiffusionDB images or MI-TUNE can improve HPS score alignment with respect the SD-2.1 b-ase baseline. Yet, MI- TUNE improves upon using directly DiffusionDB images, i.e., using MI is
DiffusionDB
24.35
MI-TUNE
25.32
MI-TUNE bare
1.33
very competitive compared (expensive) manual labeling.
MI-TUNEL: Di-TNNNDDD Dipiision88
0.97
shows the abs, difference between A and
BLIP-VQA prompts decomposition. Our evaluation considers only HPS as we find that the higher prompt complexity does not well suit the BLIP-VQA prompt decomposition. Recall that BLIP-VQA requires to split the prompt into "noun phrases", each used to create VQA for the BLIP model. Specifically, BLIP-VQA uses spaCy's 's English pipeline complex when the prompt is complex. Below we report some examples related to extracting first three noun phrases extracted from human prompts.
core_web web_ sm extract noun phrases from the prompt which result
Examples of good/easy segmentations:
concept of a silent hill monster. painted by edward hopper.
anthropomorphic shark, digital art, concept art
geodesic landscape, john chamberlain, christopher balaskas, tadao ando, 4k
Examples of segmentations with missing/broad subjects:
realistic architectural visualization of sustainable mixed use post mordern post growth walkable people oriented urban development.
realistic wide angle painting of a vintage cathode ray tube, in park, in and advanced state of decay, psychedelic mushrooms all around, in a post apocalyptic city, ghibli, daytime, dynamic lighting
render of dreamy beautiful landscape, dreamy, by herbaceous plants, artger, large scale, detailed vintage photo hyper realistic ultra realistic photo realistic photography, unreal engine, high detailed, k
34

---

Published as conference paper ICLR 2025
H.2 QUALITATIVE EXAMPLES FOR DIFFUSIONDE
SD-2.1-base
Fine-tuned using DiffusionDB images
MI-TUNE
SD-2.1-base
Fine-tuned using DiffusionD8 images
MI-TUNE
(( (( cate bluc hodgehog with big gold ring and blue lightning in green grassland. 1) ), big gold ring!. blue für, clear sky, extremely detailed, fantasy painting, by haptiste monge!!!!
closcup photorcalistic photograph of mark zuckerberg cating money. film still, vibrant colors. this k hd image is trending on artstation, featured behance, well renderad, extra crisp, features intricate detail, epic compo
concept art, pirl sitting street, brnids blue and green, singu- junichi higushi, imakake, intricate, halance, ulrra derailed, masterpicce
the cifel tower gets hit by asteroid, multiple asteroids are in the the backgmund burning, apocalyptic, highly detailed, digital paintin, sharp focus, tending
portrait, volumetric lighting, lighting
lovecrall giger fractal random antibero portrail. pixur style, by stanley artgerm hagshaw.
dog dress during the renaissance
unhropomophicc owl, serious looking wearing mechanical san- glasses and grey suit, kawary, trending pixiv, anime, furry trending luralfinity,
unthropamorphic wolf with glasses wearing lab coat, trending on aristation, trending furaflinity, digital art. kawacy, anime, furry warm light. bucklighting. cartson, cuacept
crystul big peur in nest, transpurent, with light glares, reflec- tions, photo realistic, photography, photorealism, ultra realistie, intricale, detail, light, depth field, engine, dalr, rix, style swarovski, dior, faberge.
rami madek an angel in golden gray background, alphonse mucha, rhads, ross tran. artstation, arigerm, octane render, 6 k.
Figure 15: Qualitative examples from Table (same seed used for given prompt).
35

---

Published as conference paper ICLR 2025
I QUALITATIVE ANALYSIS OF MI AS AN ALIGNMENT MEASURE
Figure 16 is expanding Figure I to include qualitative examples for all categories C21-CohmBench.
Color binding:
blue car and red horse"
BLIP-VOA 0.93 HPS 0.519
BLIP-VOD VQA 0.40
BLIP-VOA HPS 0.312 MI = 22.05
BLIP-VOA 0,06 HPS 01263
BLIP-VOA 0:05
HPPS 0,310
HPS 0.258
MI 36.28
24.56
MI 15.44
MI 14.67
Texture binding: fabric dress and glass table"
0213
19.86
Shape binding: round bag and rectangular waller"
BLIP-VQA QAA- 0.82
BLIP-VQA 0.64 HPS = 0.247
BLIP-VOA 0.27 HPS 0.262 MI 14.84
BLIP-VOA 01.24 HPS 00:50:3.1 0.216
BLIP-VOA 0.01 HPS = 0.160
HPS 0.262
MI = 1861
MI 17.16
MI 12.50
MI 11.57
Spatial relation:
"a man
the top of turtle"
UNIDET 1.00 HPS
UNIDET HPS
UNIDET 0.273
UNIDEY =
UNIDET =
HPS
HPS 0.180
Non-spatial relation: "A dog chasing after a ball and wagging its tail"
BLIP-VOA 0.97
VLI
BLIP-VQA 0.92 HPS 0.251
BLIP-VOA 0.88 HPS 0 0.236 7.88
BLIP-VOA 0.09
0.290
0.258 MI 11.39
0.152
M1 15.82
MI $.99
MI 5.02
Complex prompt: "The red hat was on top of the brown coat"
5.27
4.06
Figure 16: Qualitative analysis of MI alignment measure (all metrics decrease from left to right).
36

---

Published as conference paper ICLR 2025
J BLIP-VQA, HPS AND MI SCORE DISTRIBUTIONS
MI image rank
10
45
1.0
1.0
2.0
0.8
0.8
0.8
0.30
VOA
Figure 17: CDF of alignment scores. Color reflect images rank based MI.
The analysis presented in $ 3.1 shows that BLIP-VQA, HPS and MI relate to each other. However, two aspects not discussed in $ are (i) the support of each metric and (ii) how the distribution of the scores compare between well and poor aligned images. In this ablation we address both aspects using the following protocol.
We considered all 700 training prompts for the color category (the consideration presented in this ablation extends to the other T21-Compi 221-CompBench categories too), we generated 50 images for each prompt, and computed the 3 metrics for each of the 50 images. Last, for each prompt, we rank the images based on (1:highest, 50:lowest) overall we obtained a 700 prompts x 50 images x 4 (3 metrics + rank) tensor.
We then investigated if/how the MI rank affects the distribution the scores for BLIP-VQA and HPS. Intuitively, given the highest-ranked (viz lowest-ranked) images based on MI, also BLIP-VQA and HPS should show very high values (viz low values). practice, we first reordered the scores of the three metrics for each prompt based the MI rank and then we derived 50 distributions for each metric, one for each column the tensor collecting the scores of each metric. Figure 17 shows the obtained distributions color coded based on the MI rank.
Considering the metries support, we can notice a few differences among the three metrics. Specifically, BLIP-VQA is in the [0.1] range and for all rank values, the whole support is always used. Conversely, despite HPS also the [0, 1] range the actual support is more skewed this corroborates the discussion presented in Amppendix D Last, while MI is unbounded, the scores mostly contained in the [0-40] range.
Considering the relationship between the rank and the scores, all metrics show very similar patterns. Specif- ically, all distributions very smooth matter the rank. Moreover, expected, for all metrics the distributions smoothly shift horizontally with respect to their rank - the color gradient separates very well red/high rank, yellow/middle rank, blue/low rank.
The kendal r analysis reported in S 3.1 considers the 1s, 25th 50th image for a prompt, selected by ranking the images based their MI score. This consistent with the analysis presented in Figure 17 and based on the figure we argue that our selection of 3 pictures (having the highest, mid, lowest scores for each prompt) is a reasonable choice for the results reported in 3.1 as they are representative of the spectrum of values observed by the metrics.
*IPSS is defined as the cosine similarity between image and text embeddings, similarly to CLIP. As such, theoretically, the score is in [-1, range. However, in practice, and for the T21-CompBench dataset, the score is effectively only in the [0, range.
37


Reviews:
[{"review_id": "GY0OKwH9eS", "reviewer": "Reviewer_A9zb", "summary": "This work proposes a self-supervised approach using Mutual Information (MI) for model alignment, requiring only the pre-trained T2I model and a simple fine-tuning process, outperforming current state-of-the-art methods.", "questions": "Referring Weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method does not require additional image datasets for training.\n2. The idea is relatively novel.", "weaknesses": "1. Comparison methods, \"Attend and Excite (A&E) (Chefer et al., 2023b), Structured Diffusion Guidance(SDG) (Feng et al., 2023b) and Semantic-aware Classifier-Free Guidance (SCG) (Shen et al., 2024)\"  mentioned in line 329, is implemented on SD1.4, but the proposed work is implemented on SD 2.1, which is powerful than SD1.4 and may introducing evaluation bias.\n2. The experiments, the train set, and the test set are split from the same dataset, but this may exist some correlations, what's the results on out-of-distribution prompts?\n3. Although mutual information provides a theoretical explanation, it essentially uses classifier guidance as a loss for fine-tuning. Given that this approach is so simple, has there been any similar attempt in previous work?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n6crnqVhja", "reviewer": "Reviewer_8jmm", "summary": "This paper introduces Mutual Information (MI) to guide model alignment, which uses self-supervised fine-tuning manner. It relies on a point-wise MI estimation between prompts and images to create a synthetic fine-tuning set for improving model alignment.", "questions": "Please see the waeakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The idea of introducing  self-supervised fine-tuning manner is interesting.\n\n- Mutual Information in the pipeline is simple and effective.\n\n- It seems is a plug-and-play module, which is useful for most T2I models.", "weaknesses": "- More detailed ablations are needed. The authors employ MI as the metric to select fine-tuning samples, which eliminates the extra usage of other models. However, what if we use SOTA VQA models as the metric? Intuitively, SOTA VQA models are more precise than the MI metric.\n\n- An inherent drawback of MI is that it can measure how much help comes from the prompt but cannot guide in the right direction. For example, in cases of color misalignment, how should we deal with this issue?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AsrHxsr1eh", "reviewer": "Reviewer_LaKZ", "summary": "The proposed MI-TUNE introduces a self-supervised fine-tuning approach to enhance text-to-image alignment in diffusion models. At its core, the method leverages Mutual Information (MI) between text prompts and their corresponding generated images to improve model alignment, eliminating the need for human annotation. The method generates multiple images per prompt, selects the top-K aligned ones based on MI estimation, and uses these for fine-tuning.", "questions": "1. How were the 700 prompts selected for the MI quantitative analysis?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The main problem \"Is mutual information meaningful for alignment?\" is compelling and necessary, showing the potential of MI as a new direction for text-image alignment\n2. The paper is well-organized and easy to follow.\n3. The proposed fine-tuning approach is intersting and seems to have predictive power.", "weaknesses": "1. MI scores are missing from comparison tables and images despite being central to the method.\n2. Figure 1 only demonstrates MI effectiveness on simple category prompts (color, texture, shape), lacking validation on more challenging cases like spatial relationships or complex compositions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iO2avsDwjX", "reviewer": "Reviewer_8y4V", "summary": "This paper proposes MI-TUNE, a finetuning strategy for T2I models. This strategy introduces mutual information calculation into the T2I model tuning process to guide model alignment. In brief, the tuning method uses self-supervised fine-tuning and relies on a point-wise MI estimation between prompts and images to create a synthetic fine-tuning set for improving model alignment. This paper claims to be the first to introduce mutual-information calculation into T2I model training, leading to effective guidance for model alignment. The MI index is also measured and compared with well-established metrics and user study to prove the rationality. The experiments on MI are performed and evaluated on T2I-CompBench and user studies.", "questions": "Please see the weakness part. How about the performance of MI Tuning on DiT based models or Flow-based model, since the flow-based model reveals stronger generation capability comparing to traditional DDPM training. If MI tuning strategy fails to have good performance on flow-based training, the impact may be weaker.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThis paper claims to be the first to introduce the mutual information (MI) for T2I model training alignment. And the MI is proven to be effective on model alignment and strongly agrees with well-established metrics.\n2.\tMI-TUNE results on several measurements (BLIP-VQA, HPS and Human) on image features reveals state-of-the-art performance.\n3.\tThe paper also provides the proof for MI index to have a valid point-wise manner. Introducing mutual information into T2I model tuning is interesting, it may benefit the generative model training.\n4.\tThe paper is easy to follow with algorithm and pseudo code provided.", "weaknesses": "1.\tNote that the paper mainly focuses on SD-based (SD 2.1, SDXL) models. These models are mostly the same styles, e.g., similar network structures and traditional denoising training strategies. Is there any possibility that the MI tuning incorporated with flow-based models like DiT-based models (SD3, Pixart series or so). And it is interesting to see if the proposed MI tuning behaves different with different types of models.\n2.\tThe evaluations on MI mainly focus on only simple semantic concepts like color, shape and texture. Is MI-tuning sensitive to object numbers or so?\n3.\tThe paper fixes the denoising steps to 50 when inferencing an image, are there any differences in performance of MI-tuning when using different steps except 50?\n4.\tIn quantitative analysis of Sect. 3.1, the paper mentions that the point-wise MI ranks images and select 1st, 25th and 50th as the representative images. Why the three images are representative? This needs more detailed explanations. Also, the reason of the selection needs quantitative analysis.\n5.  Some of the ablations mentioned in previous sections are hard to locate in the following contents, the writing can be improved in this part.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)