Information Theoretic Text-to-Image Alignment

## Abstract

Abstract Diffusion models for Text-to-Image ( \oldtextsc t2i ) conditional generation have seen tremendous success recently. Despite their success, accurately capturing user intentions with these models still requires a laborious trial and error process. This challenge is commonly identified as a model alignment problem, an issue that has attracted considerable attention by the research community.
Instead of relying on fine-grained linguistic analyses of prompts, human annotation, or auxiliary vision-language models to steer image generation, in this work we present a novel method that relies on an information-theoretic alignment measure. In a nutshell, our method uses self-supervised fine-tuning and relies on point-wise mutual information between prompts and images to define a synthetic training set to induce model alignment.
Our comparative analysis shows that our method is on-par or superior to the state-of-the-art, yet requires nothing but a pre-trained denoising network to estimate \oldtextsc mi and a lightweight fine-tuning strategy.

## 1 Introduction

Generative models used for Text-to-Image (\oldtextsct2i)conditional generation [, , , , , ] have reached impressive performance. In particular, diffusion models [, , , , , ] produce images obtained by specifying a natural text prompt, that acts as a guiding signal [, , ] for the generative process. Such synthetic images are of extremely high-quality, diversity and realism.
However, a faithful translation of the prompt into an image that contains the intended semantics has been shown to be problematic [, , ]. Issues include catastrophic neglecting — one or more elements of the prompt are not generated, incorrect attribute binding — attributes such as color, shape, and texture describing elements are missing or wrongly assigned, incorrect spatial layout — generated elements do not follow a specified positioning, and more generally, difficulty in handling complex prompts.
Such model alignment problems have attracted a lot of attention recently. Addressing \oldtextsct2i alignment is challenging, and a wide range of methods have been studied, including heuristics for prompt engineering [, , ] and negative prompting [, , ], which shift the alignment problem to users. Alternatively, other approaches advance that \oldtextsct2i alignment can be improved by an appropriate seed selection [, ], or prompt rewriting []. More generally, the bulk of \oldtextsct2i alignment research develops along two main lines: inference-time and fine-tuning methods.

For inference-time methods, the key intuition is that the generative process can be optimized by modifying the reverse path of the latent variables. Some works [, , ] mitigate failure cases by refining on-the-fly the cross-attention units [] of the denoising network of Stable Diffusion (\oldtextscsd) [], to attend to all subject tokens in the prompt and strengthen their activations. Many other inference-time methods, which focus on individual failure cases, have been proposed [, , , , , , , ]. The common trait of inference-time approaches is that they require a linguistic analysis of prompts, leading to specialized solutions that rely on auxiliary models for prompt understanding. Due to the extra optimization costs incurred during sampling, inference-time methods result in longer image generation time.

Alternatively, fine-tuning can also be used for \oldtextsct2i alignment. Some works [, ] require human annotation, which is used to prepare a fine-tuning set. Other methods [, , ] rely on reinforcement learning, Direct Preference Optimization (\oldtextscdpo) or a differentiable reward function to steer model behavior. Additionally, recent methods [, , , , ] use self-playing, auxiliary models such as Visual Question Answering (\oldtextscvqa) [] or segmentation maps [] in a semi-supervised fine-tuning setting. Most fine-tuning methods either require human annotation, which is subjective, costly, and does not scale well, or auxiliary models that must be instructed to provide alignment scores. However, fine-tuning methods do not incur in additional costs at image generation time.

Finally, various works [, , ] define \oldtextsct2i alignment metrics that rely on \oldtextscvqa and Large Language Models (\oldtextscllms) to measure and explain misalignment problems. In particular, a recent work [] introduces a comprehensive benchmark suite, which can be used to assess alignment w.r.t. several categories, such as attribute binding, spatial attributes and prompt complexity.

In this paper we take an information theoretic perspective on \oldtextsct2i alignment by using Mutual Information (\oldtextscmi) to quantify the non-linear relationship between prompts and the corresponding images. In particular, we focus on the estimation of point-wise \oldtextscmi using neural estimators [, , , , ], and show that \oldtextscmi can be interpreted as an extremely general measure of \oldtextsct2i alignment, which does not require linguistic analysis of prompts, nor does it rely on auxiliary models or heuristics.
Our method, which we call \oldtextscmi-tune, unfolds as follows. We build upon the work in [] and extend it to compute point-wise \oldtextscmi.
We then proceed with a self-supervised fine-tuning approach, whereby we use point-wise \oldtextscmi to construct an information-theoretic aligned fine-tuning set using synthetic data generated by the \oldtextsct2i model itself. We then use the recent adapter presented in [] to fine-tune a small fraction of weights of the \oldtextsct2i model denoising network.
In summary, our work presents the following contributions:

- •
We propose a new alignment metric that relies on \oldtextscmi between natural prompts and corresponding images (§ 2). We show that \oldtextscmi-based alignment is compatible with the most advanced techniques available today, which are based on auxiliary \oldtextscvqa or human preference models (§ 3.1). Pairwise \oldtextscmi is an extremely simple and general measure of similarity between arbitrary distributions, and we leverage it to propose a new \oldtextsct2i alignment method.
- •
We design a self-supervised fine-tuning method that uses a small number of fine-tuning samples to align a pre-trained \oldtextsct2i model with no inference-time overhead (§ 3.2).
- •
We perform an extensive experimental campaign (§ 4) using a recent \oldtextsct2i benchmark suite [], and obtain extremely competitive results for a variety of alignment categories. Previous work require an accurate analysis of which misalignment problem to address, either through linguistic analysis and prompt decomposition, or through costly human-annotation campaigns. Instead, our approach relies on a theoretically sound measure of alignment, and it is lightweight, as it does not require large amounts of fine-tuning data.

## 2 Preliminaries

### 2.1 Diffusion models

Denoising diffusion models [, ] are generative models which are characterized by a forward process, that is fixed to a Markov chain that gradually adds Gaussian noise to the data according to a carefully selected variance schedule $\beta_{t}$, and a corresponding discrete-time reverse process, that has a Markov structure as well. Intuitively, diffusion models rely on the principle of iterative denoising: starting from a simple distribution ${{\bm{{x}}}}_{T}\sim\mathcal{N}(\mathbf{0},{{\bm{{I}}}})$, samples are generated by iterative applications of a denoising network ${{\bm{{\epsilon}}}}_{{\bm{{\theta}}}}$, that removes noise over $T$ denoising steps.
A simple way to learn the denoising network ${{\bm{{\epsilon}}}}_{{\bm{{\theta}}}}$ is to consider a re-weighted variational lower bound of the marginal likelihood:

$$ $\mathcal{L}_{\text{simple}}({{\bm{{\theta}}}})=\mathbb{E}_{t\sim U(0,T),{{\bm{ {x}}}}_{0}\sim p_{\text{data}},{{\bm{{\epsilon}}}}\sim\mathcal{N}(\mathbf{0},{ {\bm{{I}}}})}\left[||{{\bm{{\epsilon}}}}-{{\bm{{\epsilon}}}}_{{\bm{{\theta}}}} (\sqrt{\bar{\alpha}_{t}}{{\bm{{x}}}}_{0}+(\sqrt{1-\bar{\alpha}_{t}}){{\bm{{ \epsilon}}}},t)||^{2}\right],$ (1) $$

where $\alpha_{t}=1-\beta_{t}$, $\bar{\alpha}_{t}=\prod_{s=1}^{t}\alpha_{s}$. For sampling, we let $\sigma_{t}^{2}=\beta_{t}$. A similar variational objective can be obtained by switching perspective from discrete to continuous time [], whereby the denoising network approximates a score function of the data distribution. For image data, the denoising network is typically parametrized by a \oldtextscunet [, ].

This simple formulation has been extended to conditional generation [], whereby a conditioning signal ${{\bm{{p}}}}$ injects “external information” in the iterative denoising process. This, requires a simple extension to the denoising network such that it can accept the conditioning signal: ${{\bm{{\epsilon}}}}_{{\bm{{\theta}}}}({{\bm{{x}}}}_{t},{{\bm{{p}}}},t)$. Then, during training, a randomized approach allows to learn both the conditional and unconditional variant of the denoising network, for example by assigning a null value to the conditioning signal. At sampling time, a weighted linear combination of the conditional and unconditional networks, such as $\tilde{{{\bm{{\epsilon}}}}}_{{\bm{{\theta}}}}({{\bm{{x}}}}_{t},{{\bm{{p}}}},t)
={{\bm{{\epsilon}}}}_{{\bm{{\theta}}}}({{\bm{{x}}}}_{t},\emptyset,t)+\gamma({{
\bm{{\epsilon}}}}_{{\bm{{\theta}}}}({{\bm{{x}}}}_{t},{{\bm{{p}}}},t)-{{\bm{{
\epsilon}}}}_{{\bm{{\theta}}}}({{\bm{{x}}}}_{t},\emptyset,t))$ can be used.

In this work, we use pre-trained diffusion models [], which operate on a learned projection of the input data ${{\bm{{x}}}}_{0}$ into a corresponding latent variable ${{\bm{{z}}}}_{0}$, which is lower dimensional compared to the original data. Moreover, the conditioning signal ${{\bm{{p}}}}$ is obtained by a text encoder such as \oldtextscclip [].

### 2.2 MI estimation

\oldtextscmi is a central measure to study the non-linear dependence between random variables [, ], and has been extensively used in machine learning for representation learning [, , , , ], and for both training [, , ] and evaluating generative models [, ].

For many problems of interest, precise computation of \oldtextscmi is not an easy task [, ]. Consequently, a wide range of techniques for \oldtextscmi estimation have flourished. In this work, we focus on realistic and high-dimensional data, which calls for recent advances in \oldtextscmi estimation [, , , , , , , ].
In particular, we capitalize on a recent method [], that relies on the theory behind continuous-time diffusion processes [] and uses the Girsanov Theorem [] to show that score functions can be used to compute the Kullback-Leibler (\oldtextsckl) divergence between two distributions, as well as the (conditional) entropy of a random variable.
In what follows, we use a simplified notation, and gloss over several mathematical details to favor intuition over rigor.
Here we consider discrete-time diffusion models, which are equivalent to the continuous-time counterpart under the variational formulation, up to constants and discretization errors [].

We begin by considering the two arbitrary random variables ${{\bm{{z}}}},{{\bm{{p}}}}$ which are sampled from the joint distribution $p_{\text{latent},\text{prompt}}$, where the former corresponds to the distribution of the projections in a latent space of the image distribution, and the latter to the distribution of prompts used for conditional generation.
Then, following the approach in [], with the necessary adaptation to the discrete domain (see Appendix A for details), point-wise \oldtextscmi estimation can be obtained as follows

$$ $\oldtextsc{I}({{\bm{{z}}}},{{\bm{{p}}}})=\mathbb{E}_{t,{{\bm{{\epsilon}}}}\sim \mathcal{N}(\mathbf{0},{{\bm{{I}}}})}\left[\kappa_{t}||{{\bm{{\epsilon}}}}_{{ \bm{{\theta}}}}({{\bm{{z}}}}_{t},{{\bm{{p}}}},t)-{{\bm{{\epsilon}}}}_{{\bm{{ \theta}}}}({{\bm{{z}}}}_{t},\emptyset,t)||^{2}\right],\,\,\,\kappa_{t}=\frac{ \beta_{t}T}{2\alpha_{t}(1-\bar{\alpha}_{t})}.$ (2) $$

Given a pre-trained diffusion model, we compute an expectation (over diffusion times $t$) of the scaled squared norm of the difference between the conditional and unconditional networks, which corresponds to an estimate of the point-wise \oldtextscmi between an image and a prompt. This is not only a key ingredient of our information-theoretic alignment method, but it also represents a competitive advantage, as it enables a self-contained approach to alignment, which relies on the \oldtextsct2i model alone and does not need auxiliary models, nor human feedback.

## 3 Our method: \oldtextsc mi-tune

The \oldtextsct2i alignment problem arises when user intentions, as expressed through natural text prompts, fail to materialize in the generated image.
Our novel approach aims to address alignment using a theoretically grounded \oldtextscmi estimation, that applies across various contexts. While our focus in this work is on \oldtextsct2i alignment, our framework remains extensible to other modalities.
To improve model alignment, we introduce a self-supervised fine-tuning method. Leveraging the same T2I model, we estimate \oldtextscmi and generate an information-theoretic aligned fine-tuning dataset.

### 3.1 Information-theoretic alignment

Information theoretic measures, such as \oldtextscmi, have recently emerged in the literature to study the generative process of diffusion models. For example, the work in [] estimates pixel-wise mutual information between natural prompts and the images generated at each time-step of a backward diffusion process. They compare such “information maps” to cross-attention maps [] in an experiment involving prompt manipulation — modifications of the initial prompt during reverse diffusion – and conclude that \oldtextscmi is much more sensitive to information flow from prompt to images. In a similar vein, the work in [] computes \oldtextscmi between prompt and images at different stages of the reverse process of image generation. Experimental evidence indicates that \oldtextscmi can be used to analyze various reverse diffusion phases: noise, semantic, and denoising stages [].
In summary, \oldtextscmi can be used to gauge the amount of information a text prompt conveys about an image, and vice-versa.

In the following, we first focus on a qualitative analysis to support our claim. We begin with a set of simple prompts to probe color, texture and shape attribute binding, using \oldtextscsd [] to generate the corresponding images. For each prompt-image pair, we compute the \oldtextscblip-vqa score, Human Preference Score (\oldtextschps) [], point-wise \oldtextscmi estimates, and show some examples. Figure 1 indicates substantial agreement among all these measures in identifying prompts and images alignment.

Figure: \oldtextscblip-vqa =0.93 \oldtextschps =0.3198 \oldtextscmi =36.28
Refer to caption: extracted/5634501/images/a_blue_car_and_a_red_horse/a_blue_car_and_a_red_horse_0-onlyimg.png

Next, our goal is to measure the agreement between \oldtextscmi and well-established alignment metrics, to substantiate why \oldtextscmi is a valid measure. We use \oldtextscblip-vqa and \oldtextschps, which is an elaborate metric, blending alignment with aesthetics, according to human perception, which are factors that can sometimes be in conflict. We select 700 prompts from [] and use \oldtextscsd to generate 50 images per prompt. We use point-wise \oldtextscmi to rank such images, and select the 1st, the 25th, and the 50th. For these three representative images, we compute both \oldtextscblip-vqa and \oldtextschps scores, and re-rank them according to such alignment metrics.
We measure agreement between the three rankings using Kendall’s $\tau$ method [], averaged on all prompts. Our results indicate a positive agreement between \oldtextscmi and \oldtextscblip-vqa, with $\tau=0.4$, and a strong agreement between \oldtextscmi and \oldtextschps, with $\tau=0.68$.

To strengthen our analysis, we also design an on-line experiment to elicit human preference (see § B.1 for details). Given a randomly selected prompt from [] that users can read, we present the top-ranked generated image (among the 50) according to \oldtextscmi, \oldtextscblip-vqa and \oldtextschps, in a randomized order. Users can select one or more images to indicate their preference regarding alignment and aesthetics, for a total of 10 random prompts per user. Our results indicate that human preference for prompt-image pairs goes to \oldtextscmi for 69.1%, \oldtextscblip-vqa for 73.5% and \oldtextschps for 52.2% of the cases, respectively. These results corroborate once more that \oldtextscmi is a valid alignment measure, and that it is also compatible with aesthetics, setting the stage for our \oldtextsct2i alignment method.

### 3.2 Self-supervised fine-tuning with \oldtextsc mi-tune

In a nutshell, given a pre-trained diffusion model such as \oldtextscsd [] (or any variant thereof), we leverage our point-wise \oldtextscmi estimation method to select a small fine-tuning dataset set of information-theoretic aligned examples. Our method unfolds as follows.

Our self-supervised objective is to rely solely on the pre-trained model to produce a given amount of fine-tuning data, which is then filtered to retain only prompt-image pairs with a high degree of alignment, according to pair-wise \oldtextscmi estimates. We begin with a set of fine-tuning prompts $\mathcal{P}$, which can be either manually crafted, or borrowed from available prompt collections [, ]. Ideally, fine-tuning prompts should be conceived to stress the pre-trained model with challenging attribute and spatial binding, or complex rendering tasks.

Figure: Algorithm 1 \oldtextscmi-tune

As described in Algorithm 1, for each prompt $p^{(i)}$ in the fine-tuning set $\mathcal{P}$, we use the pre-trained model to generate a fixed number $M$ of synthetic, paired images. Given prompt-image pairs $(p^{(i)};z^{(j)}),\,j\in[1,M]$, we estimate pair-wise \oldtextscmi and select the top $k$ pairs, which will be part of the model fine-tuning dataset $\mathcal{S}$.
Finally, we augment the pre-trained model with adapters [, ], and proceed with fine-tuning, according to several variants. We study the impact of the adapter choice, and whether only the denoising network, or both the denoising and text encoder networks should be fine-tuned. Moreover, we measure the impact of the number of fine-tuning rounds $R$ to apply to the pre-trained model, i.e., we renew the fine-tuning dataset $\mathcal{S}$, and fine-tune the last model generation.

Figure: Algorithm 2 Point-wise \oldtextscmi Estimation

Our efficient implementation combines latent generation, and point-wise \oldtextscmi computation, as shown in Algorithm 2. Indeed, since \oldtextscmi estimation involves computing an expectation over diffusion times $t$, it is easy to combine generation and estimation in the same loop. Moreover, this function can be easily parallelized, to gain significant performance boost in the construction of the fine-tuning set $\mathcal{S}$.

## 4 Experimental evaluation

We now delve into the details of our experimental protocol, to assess the performance of our proposed method \oldtextscmi-tune in comparison to existing approaches.

### 4.1 Baselines

We begin with an overview of the recent \oldtextsct2i alignment methods we use in our comparative analysis.

Base model. We use a pre-trained \oldtextsct2i model called \oldtextscsd-2.1-base, which works at $512\times 512$ resolution. For fairness in our comparative analysis, we adapt all competitors to the same base model.

Inference-time methods. These methods intervene at sampling time, and optimize the latent variables ${{\bm{{z}}}}_{t}$ throughout the numerical integration used to generate the (latent) image. This is done through an auxiliary loss function that uses attention maps to steer model alignment, based on a fine-grained linguistic analysis of the prompt. In this category, we consider: Attend and Excite (\oldtextsca&e) [], Structured Diffusion Guidance (\oldtextscsdg) [] and Semantic-aware Classifier-Free Guidance (\oldtextscscg) [].

Fine-tuning methods. Given a pre-trained model, these methods fine-tune it using adapters []. Several variants exist, such as those based on reinforcement learning, and those based on supervision. In this category, we consider: Diffusion Policy Optimization with KL regularization (\oldtextscdpok) [], Generative mOdel finetuning with Reward-driven Sample selection (\oldtextscgors) [] and Hard-Negatives Image-Text-Matching (\oldtextschn-itm) [].

### 4.2 Benchmark and metrics

We evaluate the capability of \oldtextsct2i models to integrate and compose various concepts into a detailed and consistent scene based on textual prompts, which are generated with either predefined rules or ChatGPT [] using \oldtextsct2i-CompBench []. This benchmark suite involves combining attributes with objects, assembling different objects with defined interactions and spatial relationships, and creating intricate compositions.
In this work we use the 6 categories defined in \oldtextsct2i-CompBench, which are: attribute binding (including color, shape, and texture categories), object relationships (including spatial and non-spatial relationship), and complex compositions.

Evaluating compositional properties of \oldtextsct2i is difficult as it requires a detailed cross-modal understanding for a given prompt-image pair. Despite several metrics have been used in the past, including \oldtextscclip [, ], \oldtextscminiGPT-4 [], and of course human evaluation, in our work we use \oldtextscblip-vqa and \oldtextschps. \oldtextscblip-vqa has been consistently used across the literature, and allows for an additional comparison for approaches that rely on a different base model. In addition, \oldtextschps is another useful metric that results from a pre-trained model on human annotated data, that contributes to an evaluation of competing methods based on alignment and aesthetics. Finally, note that the score for the 2D-Spatial category is based on UniDet [], as done in \oldtextsct2i-CompBench [].

### 4.3 Details on our method \oldtextsc mi-tune

Next, we provide additional details on \oldtextscmi-tune, including fine-tuning prompts and training details.

Fine-tuning prompts. For the fine-tuning prompts set $\mathcal{P}$, we use text prompts defined in []. We select prompts from each of the 6 available categories. For each fine-tuning prompt ${{\bm{{p}}}}^{(i)}$, we generate $M$ images using the pre-trained \oldtextscsd-2.1-base model, compute their point-wise \oldtextscmi, and select the top $k$. We remark that, albeit in a different context, this task resembles that of image retrieval, as discussed in [].
We also study the benefits of several rounds $R$ of model fine-tuning. Furthermore, we explore the impact of fine-tuning prompt variety, as well as different values for $M$ and $k$.
The default values are $M=50$, $k=1$, and $R=1$, which result in a fine-tuning set $\mathcal{S}$ containing 700 prompt-image pairs. Additional configurations and results are available in Appendix D.

Fine-tuning Details. With \oldtextscmi-tune, we inject DoRA [] layers into the denoising \oldtextscunet network only, whereas other layers, and the \oldtextscclip text encoder, are frozen. DoRA layers’ rank and scaling factor $\alpha$ are set to 32. Full setup details are available in Appendix C. We experimented with different configurations, including LoRA layers [], as well as fine tuning both the denoising network and the \oldtextscclip-based text encoder, but measured no performance improvements, as shown in Appendix D.

### 4.4 Results

Table 1 reports an overview of the performance of all competing methods and \oldtextscmi-tune. In particular, for \oldtextscmi-tune we use the default configuration, and study the impact of multiple fine-tuning rounds $R$. Moreover, we also show results on the “spatial” category for a variant, called \oldtextscmi-tune $\star$, in which we fine-tune \oldtextscSpRight [] according to point-wise \oldtextscmi, a \oldtextscsd-2.1-base model that was already fine-tuned for spatial alignment. Also, recall that \oldtextscmi-tune does not incur in extra overhead at generation time, nor require auxiliary models, human annotation or very large fine-tuning sets.

**Table 1: Alignment results (%) on \oldtextsct2i-CompBench. Bold indicates best performance; Underscore indicates second-best performance; Highlighted text indicates higher performance than the best method when performing multiple rounds of fine-tuning; The symbol $\diamond$ indicates the best competing method in each category.**
| Alignment<br>Method |  | Category |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | Color | Shape | Texture | 2D-Spatial | Non-spatial | Complex |
|  | Model | (\oldtextscblip-vqa) | (\oldtextscblip-vqa) | (\oldtextscblip-vqa) | (\oldtextscunidet) | (\oldtextscblip-vqa) | (\oldtextscblip-vqa) |
| none | \oldtextscsd-2.1-base | 49.65 | 42.71 | 49.99 | 15.77 | 66.23 | 50.53 |
| Inference-time | \oldtextsca&e | $\diamond$ 61.43 | $\diamond$ 47.39 | $\diamond$ 64.10 | 16.18 | 66.21 | $\diamond$ 51.69 |
|  | \oldtextscsdg | 47.15 | 45.24 | 47.13 | 15.25 | 66.17 | 47.41 |
|  | \oldtextscscg | 49.82 | 43.28 | 50.16 | $\diamond$ 16.31 | $\diamond$ 66.60 | 51.07 |
| Fine-tuning | \oldtextscdpok | 53.28 | $\diamond$ 45.63 | 52.84 | $\diamond$ 17.19 | 66.95 | 51.97 |
|  | \oldtextscgors | $\diamond$ 53.59 | 43.82 | $\diamond$ 54.47 | 15.66 | $\diamond$ 67.47 | $\diamond$ 52.28 |
|  | \oldtextschn-itm | 46.51 | 39.99 | 48.78 | 15.24 | 65.31 | 49.84 |
|  | \oldtextscmi-tune | 61.57 | 48.40 | 58.27 | 15.93 | 67.77 | 53.54 |
|  | \oldtextscmi-tune $\star$ | - | - | - | 18.51 | - | - |
|  | \oldtextscmi-tune-$R=2$ | 66.42 | 52.13 | 62.74 | 14.17 | 66.46 | 53.97 |
|  | \oldtextscmi-tune-$R=3$ | 69.67 | 51.62 | 65.69 | 14.16 | 66.31 | 52.95 |

In general, we remark that the best competitor for the attribute binding category (color, shape and texture) is \oldtextsca&e in the inference-time family, which also achieves good results for the spatial category. Instead, fine-tuning methods perform better in the other categories. Overall, \oldtextscmi-tune obtains excellent results in its default configuration, which are comparable or better than \oldtextsca&e in the attribute binding cases, and better than other fine-tuning methods except for the spatial category.
For the spatial category, on which most methods struggle, we demonstrate that the methodology we propose in this work can be applied to any existing model. In particular, \oldtextscmi-tune $\star$ obtains the best results, paving the way for \oldtextscmi to be used as an extremely general alignment measure. Additional results, including an ablation study on \oldtextscmi-tune is available in Appendix D.
Furthermore, we notice the benefits of additional fine-tuning rounds, especially for the attribute binding categories: highlights indicate when additional fine-tuning obtains better \oldtextscblip-vqa scores than the best performing method.

**Table 2: \oldtextschps alignment/aesthetics results across six categories from \oldtextsct2i-CompBench.**
| Model | Category |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | Color | Shape | Texture | 2D-Spatial | Non-spatial | Complex |
| \oldtextscsd-2.1-base | 27.64 | 24.56 | 24.99 | 27.5 | 26.66 | 25.7 |
| Inference-time best | 28.44 | 24.43 | 25.88 | 27.76 | 26.98 | 25.6 |
| Fine-tuning best | 28.15 | 24.99 | 25.56 | 28.12 | 26.88 | 26.07 |
| \oldtextscmi-tune | 29.13 | 25.56 | 26.2 | 28.5 | 27.15 | 26.7 |
| \oldtextscmi-tune $\star$ | - | - | - | 29.7 | - | - |

In Table 2, we present results according to \oldtextschps []. We compare the \oldtextscsd baseline and \oldtextscmi-tune to the best (for each category) inference-time and fine-tuning methods, as indicated with marker $\diamond$ in Table 1. According to human preference as predicted by the \oldtextschps model, our method stands out both in terms of alignment and aesthetics, indicating that \oldtextscmi captures and reconcile factors that can sometimes be in conflict.

Finally, in § B.2, we measure human preference through a user study, and compare \oldtextscmi-tune generated images to those obtained from all the alternatives we study in our evaluation. Results indicate that \oldtextscmi-tune is the preferred method for the majority of categories we consider.

In Figure 2, we show a selection of qualitative samples on some challenging prompts (see also Appendix E). We visualize images generated according to the base model, the six competitors, and \oldtextscmi-tune, and use a fixed seed across methods. In our experiments, \oldtextscmi-tune demonstrates that \oldtextsct2i alignment can be greatly improved when compared to “vanilla” \oldtextscsd and existing alignment methods.

Figure: \oldtextscsd
Refer to caption: extracted/5634501/images/benchmark/color/a_blue_bear_and_a_brown_boat_53.png

## 5 Discussion

The self-supervised nature of our approach to fine-tuning, which solely relies on the \oldtextsct2i model for the generation of a synthetic candidate set of prompt-image pairs, is limited by the capabilities of the pre-trained model. Indeed, an underlying assumption we make is to expect the pre-trained model to eventually produce images that are aligned to the prompt, as done in previous works [, ]. Then, our information-theoretic alignment measure can be used to produce a useful fine-tuning set $\mathcal{S}$. However, in some cases (e.g., spatial or complex prompts) the pre-trained model can systematically fail in producing aligned samples. In this scenario, our method can be applied to existing \oldtextsct2i models that have been fine-tuned with complementary techniques, and substantially boost alignment performance, as we demonstrate in our experiments, as shown in Table 1.

In our work, we considered a large design space. In the context of fine-tuning approaches, we designed an alternative method to exploit pair-wise \oldtextscmi. Indeed, it is simple to rewrite the loss from Equation 1 to include a point-wise \oldtextscmi loss term, and use a small fraction of the \oldtextsct2i model training set to fine-tune it according to an “embedded” information-theoretic alignment measure. While this approach is in principle valid, we noticed that throughout the parameter optimization phase, \oldtextscmi estimates are affected by high variance, due to the poor quality of the unconditional denoising network of typical pre-trained \oldtextsct2i models []. Training the \oldtextsct2i model from scratch according to [] can mitigate this issue, but defeats the purpose of using a pre-trained model.

Alternatively, we designed a variant of our method that operates at inference time, in a similar vein to the work in []. To do that, we blend the works [, ], to replace attention maps with pixel-level point-wise \oldtextscmi maps. Conditional sampling can then be amended as in []: we can define a new inference-time loss term that we use to update ${{\bm{{z}}}}_{t}$ on-the-fly, using information maps instead of cross-attention maps to steer model alignment. This is another valid approach that however bears the same problems discussed above: information maps quality suffers from a poorly trained unconditional denoising network, leading to poor results. Since the work in [] does not explicitly focus on model alignment, the authors build information maps that use the unconditional denoising network with an edited prompt, instead of a null token, which is not an option for our study.

## 6 Conclusion

\oldtextsct2i alignment emerged as an important endeavor to steer image generation to follow the semantics and user intent expressed through a natural text prompt, as it can save considerable manual effort.

In this work, we presented a novel approach to measure alignment, that uses point-wise \oldtextscmi between prompt-image pairs to quantify the amount of information “flowing” between natural text and images. We demonstrated, both qualitatively and quantitatively, that pair-wise \oldtextscmi is coherent with existing alignment measures that either use external \oldtextscvqa models or human preference.

Armed with an information-theoretic measure of alignment, we presented \oldtextscmi-tune, a lightweight, self-supervised fine-tuning method that uses a pre-trained \oldtextsct2i model such as \oldtextscsd to estimate \oldtextscmi, and generates a synthetic set of aligned prompt-image pairs, which is then used in a parameter-efficient fine-tuning stage, to align the \oldtextsct2i model.
Our approach does not require human annotation, auxiliary \oldtextscvqa models, nor costly inference-time techniques, and outperforms state-of-the-art approaches on most alignment categories examined in the literature.

Limitations. Accurate \oldtextscmi estimation is a difficult task, especially for high-dimensional, real-world data. For our method, \oldtextscmi estimation accuracy depends on the quality of the pre-trained denoising network of the base \oldtextsct2i model which, in the case of \oldtextscsd, is problematic for the unconditional version. As such, it is possible that \oldtextscmi-based alignment can suffer. Furthermore, despite \oldtextscmi appears to be more aligned to human preference than existing metrics, it is less interpretable when compared to methods that perform linguistic prompt analyses, or use auxiliary \oldtextscvqa models for explainability.

Finally, our information-theoretic alignment measure can be applied to arbitrary distributions, beyond text and images. Further work is necessary to demonstrate its application to other pre-trained models that operate on other modalities, such as e.g., the audio domain.

Broader impact. A valid concern to point out regarding our method \oldtextscmi-tune is that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. Nevertheless, we believe that the positive impact of our work outweighs such problems.

## Appendix A Details on \oldtextsc mi estimation

In this Section, we provide the proof for Equation 2.We start by recalling the definition of the forward and backward processes for a discrete-time diffusion model. For the forward process, we use the following Markov chain

$$ $q({{\bm{{z}}}}_{0:T},{{\bm{{p}}}})=q({{\bm{{z}}}}_{0},{{\bm{{p}}}})\prod \limits_{t=1}^{T}q({{\bm{{z}}}}_{t}\,|\,{{\bm{{z}}}}_{t-1}),\quad q({{\bm{{z}} }}_{t}\,|\,{{\bm{{z}}}}_{t-1})=\mathcal{N}({{\bm{{z}}}}_{t};\sqrt{1-\beta_{t}} {{\bm{{z}}}}_{t-1},\beta_{t}I)$ $$

The backward process (with or without a conditioning signal ${{\bm{{p}}}}$) evolves according to

$$ $p_{{{\bm{{\theta}}}}}({{\bm{{z}}}}_{0:T})=p({{\bm{{z}}}}_{T})\prod\limits_{t=1 }^{T}p_{{{\bm{{\theta}}}}}({{\bm{{z}}}}_{t-1}\,|\,{{\bm{{z}}}}_{t}),\quad p_{{ {\bm{{\theta}}}}}({{\bm{{z}}}}_{0:T}\,|\,{{\bm{{p}}}})=p({{\bm{{z}}}}_{T}) \prod\limits_{t=1}^{T}p_{{{\bm{{\theta}}}}}({{\bm{{z}}}}_{t-1}\,|\,{{\bm{{z}}} }_{t},{{\bm{{p}}}})$ $$

where $p_{{{\bm{{\theta}}}}}({{\bm{{z}}}}_{t-1}\,|\,{{\bm{{z}}}}_{t})=\mathcal{N}({{
\bm{{z}}}}_{t-1};\mu_{{\bm{{\theta}}}}({{\bm{{z}}}}_{t}),\beta_{t}I)$, with $\mu_{{\bm{{\theta}}}}({{\bm{{z}}}}_{t})=\frac{1}{\sqrt{\alpha_{t}}}\left({{\bm
{{z}}}}_{t}-\frac{\beta_{t}}{\sqrt{1-\bar{\alpha}_{t}}}{{{\bm{{\epsilon}}}}}_{
{\bm{{\theta}}}}({{\bm{{z}}}}_{t},t)\right)$. Similar expressions can be obtained for the conditional version.

Our goal here is to show that the following equality holds

$$ $\mathbb{E}_{{{\bm{{p}}}}}[\oldtextsc{kl}\left[q({{\bm{{z}}}}_{0}\,|\,{{\bm{{p} }}})\;\|\;q({{\bm{{z}}}}_{0})\right]]=\mathbb{E}_{{{\bm{{z}}}},{{\bm{{p}}}}}[ \oldtextsc{I}({{\bm{{z}}}},{{\bm{{p}}}})],$ $$

which is the condition that $\oldtextsc{I}({{\bm{{z}}}},{{\bm{{p}}}})$ of Equation 2 should satisfy to be a valid point-wise \oldtextscmi estimator. In particular, we will show that

$$ $\mathbb{E}_{{{\bm{{p}}}}}[\oldtextsc{kl}\left[q({{\bm{{z}}}}_{0}\,|\,{{\bm{{p} }}})\;\|\;q({{\bm{{z}}}}_{0})\right]]=\mathbb{E}_{t,{{\bm{{p}}}},{{\bm{{z}}}}, {{\bm{{\epsilon}}}}}\left[\kappa_{t}||{{\bm{{\epsilon}}}}_{{\bm{{\theta}}}}({{ \bm{{z}}}}_{t},{{\bm{{p}}}},t)-{{\bm{{\epsilon}}}}_{{\bm{{\theta}}}}({{\bm{{z} }}}_{t},\emptyset,t)||^{2}\right],\,\,\,\kappa_{t}=\frac{\beta_{t}T}{2\alpha_{ t}(1-\bar{\alpha}_{t})}.$ $$

To simplify our proof strategy, we consider the ideal case of perfect training, i.e., $p_{{{\bm{{\theta}}}}}({{\bm{{z}}}}_{0:T},{{\bm{{p}}}})=q({{\bm{{z}}}}_{0:T},{{
\bm{{p}}}})$. Moreover, since $q({{\bm{{z}}}}_{t}\,|\,{{\bm{{z}}}}_{t-1},{{\bm{{p}}}})=q({{\bm{{z}}}}_{t}\,|
\,{{\bm{{z}}}}_{t-1})$, we can rewrite the $\oldtextsc{kl}\left[q({{\bm{{z}}}}_{0}\,|\,{{\bm{{p}}}})\;\|\;q({{\bm{{z}}}}_{
0})\right]$ term as follows

$$ $\displaystyle\oldtextsc{kl}\left[q({{\bm{{z}}}}_{0}\,|\,{{\bm{{p}}}})\;\|\;q({ {\bm{{z}}}}_{0})\right]=\oldtextsc{kl}\left[q({{\bm{{z}}}}_{0:T}\,|\,{{\bm{{p} }}})\;\|\;q({{\bm{{z}}}}_{0:T})\right]=\oldtextsc{kl}\left[p_{{\bm{{\theta}}}} ({{\bm{{z}}}}_{0:T}\,|\,{{\bm{{p}}}})\;\|\;p_{{{\bm{{\theta}}}}}({{\bm{{z}}}}_ {0:T})\right]=$ $\displaystyle\int p_{{\bm{{\theta}}}}({{\bm{{z}}}}_{0:T}\,|\,{{\bm{{p}}}})\log \frac{p_{{\bm{{\theta}}}}({{\bm{{z}}}}_{0:T}\,|\,{{\bm{{p}}}})}{p_{{{\bm{{ \theta}}}}}({{\bm{{z}}}}_{0:T})}\differential{{\bm{{z}}}}_{0:T}=\int p_{{\bm{{ \theta}}}}({{\bm{{z}}}}_{0:T}\,|\,{{\bm{{p}}}})\sum\limits_{t=1}^{T}\log\frac{ p_{{\bm{{\theta}}}}({{\bm{{z}}}}_{t-1}\,|\,{{\bm{{z}}}}_{t},{{\bm{{p}}}})}{p_{ {{\bm{{\theta}}}}}({{\bm{{z}}}}_{t-1}\,|\,{{\bm{{z}}}}_{t})}\differential{{\bm {{z}}}}_{0:T}=$ $\displaystyle\sum\limits_{t=1}^{T}\int p_{{\bm{{\theta}}}}({{\bm{{z}}}}_{0:t-2 ,t:T}\,|\,{{\bm{{p}}}})\left(\int p_{{\bm{{\theta}}}}({{\bm{{z}}}}_{t-1}\,|\,{ {\bm{{z}}}}_{t},{{\bm{{p}}}})\log\frac{p_{{\bm{{\theta}}}}({{\bm{{z}}}}_{t-1} \,|\,{{\bm{{z}}}}_{t},{{\bm{{p}}}})}{p_{{{\bm{{\theta}}}}}({{\bm{{z}}}}_{t-1} \,|\,{{\bm{{z}}}}_{t})}\differential{{\bm{{z}}}}_{t-1}\right)\differential{{ \bm{{z}}}}_{0:t-2,t:T}=$ $\displaystyle\sum\limits_{t=1}^{T}\int p_{{\bm{{\theta}}}}({{\bm{{z}}}}_{t}\,| \,{{\bm{{p}}}})\oldtextsc{kl}\left[p_{{\bm{{\theta}}}}({{\bm{{z}}}}_{t-1}\,|\, {{\bm{{z}}}}_{t},{{\bm{{p}}}})\;\|\;p_{{\bm{{\theta}}}}({{\bm{{z}}}}_{t-1}\,| \,{{\bm{{z}}}}_{t})\right]\differential{{\bm{{z}}}}_{t}=$ $\displaystyle\sum\limits_{t=1}^{T}\frac{1}{2\beta_{t}}\int p_{{\bm{{\theta}}}} ({{\bm{{z}}}}_{t}\,|\,{{\bm{{p}}}})\norm{\mu_{{{\bm{{\theta}}}}}({{\bm{{z}}}}_ {t})-\mu_{{{\bm{{\theta}}}}}({{\bm{{z}}}}_{t},{{\bm{{p}}}})}^{2}\differential{ {\bm{{z}}}}_{t}=$ $\displaystyle\sum\limits_{t=1}^{T}\frac{1}{2\beta_{t}}\frac{\beta_{t}^{2}}{ \alpha_{t}(1-\bar{\alpha}_{t})}\int p_{{\bm{{\theta}}}}({{\bm{{z}}}}_{t}\,|\,{ {\bm{{p}}}})\norm{\epsilon_{{{\bm{{\theta}}}}}({{\bm{{z}}}}_{t})-\epsilon_{{{ \bm{{\theta}}}}}({{\bm{{z}}}}_{t},{{\bm{{p}}}})}^{2}\differential{{\bm{{z}}}}_ {t}=$ $\displaystyle\mathbb{E}_{t,{{\bm{{z}}}}_{t}}\left[\kappa_{t}\norm{\epsilon_{{{ \bm{{\theta}}}}}({{\bm{{z}}}}_{t},\emptyset,t)-\epsilon_{{{\bm{{\theta}}}}}({{ \bm{{z}}}}_{t},{{\bm{{p}}}},t)}^{2}\right]=$ $\displaystyle\mathbb{E}_{t,{{\bm{{z}}}},\epsilon}\left[\kappa_{t}\norm{ \epsilon_{{{\bm{{\theta}}}}}({{\bm{{z}}}}_{t},\emptyset,t)-\epsilon_{{{\bm{{ \theta}}}}}({{\bm{{z}}}}_{t},{{\bm{{p}}}},t)}^{2}\right],\quad\kappa_{t}=\frac {\beta_{t}T}{2\alpha_{t}(1-\bar{\alpha}_{t})}$ $$

which allows to prove that the quantity in Equation 2 is indeed a valid point-wise \oldtextscmi estimator.

## Appendix B Details on user study

We now present our preliminary user study, which is based on focus groups only, with (lightly) guided discussions led by a moderator. Although it is easy to use the methodology described below to launch a large-scale survey campaign, this would require a completely different organization and implementation, which we did not put in place for this work.

The survey web app.
Beside *punctually* comparing alignment metrics § 3 and methods § 4.4, we designed a web app to collect *subjective* feedback, in the form of mini surveys, from real users. Each survey is composed of multiple rounds, each showing
a prompt and a set of images generated from it.
Under the hood, the web app corresponds to a jupyter notebook with ipywidgets(^1^11[https://ipywidgets.readthedocs.io/en/stable/](https://ipywidgets.readthedocs.io/en/stable/)) for UI controls, rendered via the voila(^2^22[https://voila.readthedocs.io/en/stable/using.html](https://voila.readthedocs.io/en/stable/using.html)) framework and deployed live via a docker-ized HuggingFace space. Via the web app we run campaigns to *compare alignment metrics* and to *compare alignment methods*.

Figure: Figure 3: Web app screenshot example of the alignment metric comparison survey (URL blackened out for anonymization).
Refer to caption: extracted/5634501/images/user-survey/survey_screenshot.png

Fig. 3 shows an example screenshot of the alignment metric comparison survey. As from the example, users are free to select from 0 to up to 3 images for each prompt in the alignment metric comparison survey (0 to up to 8 for the alignment comparison survey). However, to stress users subjectivity, we intentionally did not provide guidelines on how to handle “odd” cases (e.g., if the prompt asks for a picture of “an apple”, but the picture show more than one apple). Last, each survey is saved as a separate CSV with the timestamp of its creation which also serves as unique identifier of the survey, i.e., neither user identification is required, nor cookies are set as part of the web app logic, so users’ privacy and anonymity is preserved.

**Table 3: User study about comparing alignment metrics.**
| Metric | Campaign answers (%) |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| \oldtextscmi | \oldtextscblip-vqa | \oldtextschps | Academic users | Random users | Students | avg |
| ○ | ○ | ○ | 14.7 | 16.9 | 25.0 | 18.9 |
| ○ | ○ | ● | 1.8 | 14.0 | 2.7 | 6.2 |
| ○ | ● | ○ | 10.4 | 22.0 | 4.1 | 12.2 |
| ○ | ● | ● | 4.0 | 7.4 | 3.6 | 5.0 |
| ● | ○ | ○ | 4.0 | 10.6 | 0.9 | 5.2 |
| ● | ○ | ● | 6.0 | 2.5 | 2.3 | 3.6 |
| ● | ● | ○ | 18.7 | 10.6 | 16.8 | 15.4 |
| ● | ● | ● | 40.4 | 16.0 | 44.6 | 33.7 |
| ● | ◑ | ◑ | 69.1 | 39.7 | 64.6 | 57.8 |
| ◑ | ● | ◑ | 73.5 | 56.0 | 69.1 | 66.2 |
| ◑ | ◑ | ● | 52.2 | 39.9 | 53.2 | 48.2 |

### B.1 Comparing alignment metrics.

In this analysis we aimed to understand how users perceive images selected via \oldtextscblip-vqa, \oldtextschps and \oldtextscmi.
To do so, we run surveys composed of 10 rounds, each showing a prompt and the related best image among 50 generations
according to each metric. Each of the 10 prompts is randomly selected from a pool of 700 prompts
for the color category, and at each round the order in which the 3 pictures is shown is also randomized.

We run surveys across three user groups: *Academic users*, counting 5 members, are representative of highly informed and tech savvy users, who are familiar with how generative models work; *Random users*, counting 25 members, are representative of illiterate users who are not familiar with computer-based image generation; *Students*, counting 16 members, are representative of masters’ level students who are familiar with image generation tools, and who have attended introductory-level machine learning classes.

Overall, we collected 45, 35 and 22 surveys across 3 days for Academics, Random users and Students respectively which we detail in
Table 3. The top part of the table breaks down all possible answers combinations. The results, although with some differences between user groups, clearly highlight that the three alignment metrics we consider in this work are roughly equivalent, with \oldtextscmi and \oldtextscblip-vqa being preferred over \oldtextschps. For the Academics and Students groups, all the three images are considered sufficiently aligned with the prompt in almost half of the cases ($40.4\%$ and $44.6\%$ respectively). Interestingly, random users select only one of the three images about 10$\times$ much more frequently than the other two groups (on average 14.2% for real users while 5.4% and 3% for Academics and Students respectively). We hypothesise that being previously exposed (or not) to the technical problems of image generation from the alignment perspective, or simply being literate about machine learning can influence the selection among the three pictures.

The bottom part of the table summarizes the answers for each individual metric.
Despite the general preference for \oldtextscblip-vqa, the results corroborate once more that MI is a valid alignment measure, and that it is
also compatible with aesthetics.

Finally, we recall that our goal in this section is to study whether \oldtextscmi is a *plausible* alignment measure, rather than electing the “best” alignment metric. Indeed, this analysis does not indicate the final performance of alignment methods, which instead we report in Table 1.

**Table 4: Users study about comparing alignment methods. Bold indicates best performance; Underline indicates second-best performance; the symbol $\diamond$ indicates the best competing method in each category.**
| Alignment |  | Category (%) |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Methodology | Model | Color | Shape | Texture | 2D-Spatial | Non-spatial | Complex |
| none | \oldtextscsd-2.1-base | 29.8 | 11.9 | 40.5 | 35.7 | 66.7 | 29.8 |
| Inference-time | \oldtextsca&e | $\diamond$31.0 | $\diamond$15.5 | $\diamond$52.4 | 32.1 | 65.5 | 31.0 |
|  | \oldtextscsdg | 26.2 | 15.4 | 38.1 | 38.1 | 61.9 | 29.8 |
|  | \oldtextscscg | 20.2 | 11.9 | 33.3 | $\diamond$40.5 | $\diamond$69.0 | $\diamond$39.3 |
| Fine-tuning | \oldtextscdpok | 23.8 | 16.7 | $\diamond$47.6 | 34.5 | $\diamond$70.2 | $\diamond$38.1 |
|  | \oldtextscgors | $\diamond$34.5 | 14.3 | 48.8 | $\diamond$36.9 | 65.5 | 31.0 |
|  | \oldtextschn-itm | 23.8 | $\diamond$19.0 | 31.0 | 20.2 | 47.6 | 23.8 |
|  | \oldtextscmi-tune | 46.4 | 25.0 | 51.2 | 45.2 | 73.8 | 46.4 |

### B.2 Comparing alignment methods.

In this analysis we aimed to understand how users perceive images generated by the 8 methods we considered
in our study, i.e., “vanilla” \oldtextscsd, \oldtextsca&e [], \oldtextscsdg [] and \oldtextscscg [] \oldtextscdpok [], \oldtextscgors [], \oldtextschn-itm [] and
our method \oldtextscmi-tune (when used with a single round of fine-tuning).

To do so, we run surveys composed of 12 rounds, i.e., 2 rounds for each category.
Each round shows a prompt and the 8 pictures generated using a different method.
For each category, we randomly selected 100 prompts from T2I-Combench test set to pre-generate the pictures.
At run time, the web app randomly select 2 prompts for each category and randomly pics
images from the related pool. Then it randomly arranges the rounds (so that categories
are shuffled) and also the methods (so that pictures of a method are not visualized in the same order).

Table 4 collects the results of a campaign with 42 surveys.
Specifically, the table shows the percentage of answers where the picture of a given method was selected (no matter if other methods were
also selected). Overall, the results confirm what measured via BLIP-VQA in Table 1 – MI is a solid choice for model alignment.

## Appendix C Experimental protocol details

We report in Table 5 all the hyper-parameters we used for our experiments.

**Table 5: Training hyperparameters.**
| Name | Value |
| --- | --- |
| Trainable model | \oldtextscunet |
| Trainable timesteps | $t\sim U(500,1000)$ |
| PEFT | DoRA [] |
| Rank | $32$ |
| $\alpha$ | $32$ |
| Learning rate (LR) | $1e-4$ |
| Gradient norm clipping | $1.0$ |
| LR scheduler | Constant |
| LR warmup steps | 0 0 |
| Optimizer | AdamW |
| AdamW - $\beta_{1}$ | $0.9$ |
| AdamW - $\beta_{2}$ | $0.999$ |
| AdamW - weight decay | $1e-2$ |
| AdamW - $\epsilon$ | $1e-8$ |
| Resolution | $512\times 512$ |
| Classifier-free guidance scale | 7.5 |
| Denoising steps | $50$ |
| Batch size | 400 |
| Training iterations | 300 |
| GPUs for Training | 1 $\times$ NVIDIA A100 |

Next, we provide additional details on the computational cost of \oldtextscmi-tune. In our approach, there are two distinct phases that require computational effort:

- •
The first is the construction of the fine-tuning set $\mathcal{S}$ based on point-wise \oldtextscmi. As a reminder, for this phase, we use a pre-trained \oldtextscsd model (at a resolution $512\times 512$) and, given a prompt, conditionally generate 50 images, while at the same time computing point-wise \oldtextscmi between the prompt and each image. This is done for all the prompts in the set $\mathcal{P}$, which counts 700 entries.
In our experiments, on a single A100-80GB GPU, the generation of the fine-tuning set $\mathcal{S}$ requires roughly 24 hours.
- •
The second is the parameter efficient fine-tuning of the pre-trained model. Using the configuration discussed above, \oldtextscmi-tune requires 8 hours for the fine-tuning, on a single A100-80GB GPU.

Note that there is no overhead at image generation time: once a pre-trained model has been fine-tuned with \oldtextscmi-tune, conditional sampling takes the same amount of time of “vanilla” \oldtextscsd.

## Appendix D Additional results and ablations

### D.1 Ablation: Fine-tuning set size

In this Section, we provide additional details and ablations on the size and composition of the fine-tuning set $\mathcal{S}$.

There are two factors that determine both the quality and the associated computational cost related to the fine-tuning set $\mathcal{S}$: the number of candidate images $M$, out of which a given fraction $k/M$ is selected to be included in $\mathcal{S}$. Next, we analyze such trade-off.
The baseline model \oldtextscmi-tune in Table 6 corresponds to result obtained by selecting the top $2\%$ images (1 out of 50) according to the \oldtextscmi metric. We repeated the finetuning experiments on the categories Color and Shape by varying the selection ratio in the ranges $\{7/50,1/30,1/100,1/500\}$. Results indicate that the best selection ratio is the middle-range corresponding to the baseline \oldtextscmi-tune. We hypothesise that higher selection ratios pollute the fine-tuning set with lower quality images, while a more selective threshold favours images which have the highest alignment but possibly lower realism. Additionally, we remark that the number $M$ of candidate images has a negligible impact, above $M=50$, whereas fewer candidate images induce degraded performance. Hence, the value $M=50$ is, in our experiments, a sweet-spot that produces a valid candidate set, while not imposing a large computational burden.

**Table 6: \oldtextscblip-vqa-\oldtextscvqa alignment results on \oldtextsct2i-CompBench’s Color and Shape categories varying size and composition of fine-tuning set.**
| Model | Category |  |
| --- | --- | --- |
|  | Color | Shape |
| \oldtextscmi-tune-7out50 | 59.31 | 47.26 |
| \oldtextscmi-tune-1out30 | 58.12 | 47.48 |
| \oldtextscmi-tune-1out50 | 61.57 | 48.40 |
| \oldtextscmi-tune-1out100 | 60.12 | 47.80 |
| \oldtextscmi-tune-1out500 | 59.28 | 46.79 |

Next, another natural question to ask is whether the self-supervised fine-tuning method we suggest in this work is a valid strategy. Indeed, instead of using synthetic image data for fine-tuning the base model, it is also possible to use real-life, captioned image data. Then, we present an ablation on the use of real samples, along with synthetic images, in the fine-tuning procedure. In Table 7 we report the experimental results obtained by composing the fine-tuning dataset by $x\%$ generated images and $(1-x)\%$ of real images taken from the CC12M dataset [], where in both cases we select, using \oldtextscmi, the candidate images to be used in the fine-tuning set $\mathcal{S}$. So, for example, \oldtextscmi-tune-Real0.25, indicates that we use 25% of real images.
Interestingly, we observe the following trend. Complementing the synthetically generated samples with few, real ones, improves performance: this is akin to a regularization term which prevents the quality of the generated images to deviate too much from the quality of the real ones. However, increasing the fraction of real images can be detrimental.

**Table 7: \oldtextscblip-vqa-\oldtextscvqa alignment results on \oldtextsct2i-CompBench’s Color and Shape categories finetuning using a mixture of generated and real images.**
| Model | Category |  |
| --- | --- | --- |
|  | Color | Shape |
| \oldtextscmi-tune | 61.57 | 48.40 |
| \oldtextscmi-tune-Real0.25 | 61.34 | 48.47 |
| \oldtextscmi-tune-Real0.5 | 61.63 | 49.50 |
| \oldtextscmi-tune-Real0.9 | 59.83 | 48.92 |

### D.2 Ablation: Model fine-tuning

In this Section, we provide additional results (Table 8) on \oldtextscmi-tune, concerning which part of the pre-trained \oldtextscsd model to fine-tune. In particular, we have tried to fine-tune the denoising \oldtextscunet network alone and both the denoising and the text encoding (\oldtextscclip) networks. The baseline results are obtained, as described in the main paper, with Do-RA [] adapters. Switching to Lo-RA layers [] incurs in a performance degradation, a trend observed also for other tasks in the literature []. Interestingly, joint fine-tuning of the \oldtextscunet backbone together with the text encoder layers degrades performance as well, which has also been observed in the literature []. Even if, in principle, a joint fine-tuning strategy should provide better results, as the amount of information transferred from the prompt to the image is bottle-necked by the text encoder architecture, we observed empirically more unstable training dynamics than the variant where only the score network backbone is fine-tuned, resulting in degraded performance.

**Table 8: \oldtextscblip-vqa-\oldtextscvqa alignment results on \oldtextsct2i-CompBench’s Color and Shape categories finetuning different portions of the model.**
| Model | Category |  |
| --- | --- | --- |
|  | Color | Shape |
| \oldtextscmi-tune-DoRA | 61.57 | 48.40 |
| \oldtextscmi-tune-LoRA | 58.25 | 48.27 |
| \oldtextscmi-tune-U+T(joint) | 57.88 | 47.79 |

## Appendix E Additional qualitative results

### E.1 Color Binding Samples

Figure: \oldtextscsd
Refer to caption: extracted/5634501/images/benchmark/color/a_black_apple_and_a_green_backpack_34.png

### E.2 Shape binding samples

Figure: \oldtextscsd
Refer to caption: extracted/5634501/images/benchmark/shape/a_big_hippopotamus_and_a_small_mouse_115.png

### E.3 Texture binding samples

Figure: \oldtextscsd
Refer to caption: extracted/5634501/images/benchmark/texture/a_fabric_bag_and_a_leather_chair_153.png

### E.4 Spatial prompts

Figure: \oldtextscsd
Refer to caption: extracted/5634501/images/benchmark/spatial/a_bicycle_on_the_top_of_a_turtle_1531.png

### E.5 Non-Spatial prompts

Figure: \oldtextscsd
Refer to caption: extracted/5634501/images/benchmark/non_spatial/A_child_is_jumping_on_a_trampoline_in_their_backyard._3052.png

### E.6 Complex prompts

Figure: \oldtextscsd
Refer to caption: extracted/5634501/images/benchmark/complex/The_red_hat_was_on_top_of_the_brown_coat._610.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Referring Weakness
soundness: 3
strengths: 1. The proposed method does not require additional image datasets for training.
2. The idea is relatively novel.
summary: This work proposes a self-supervised approach using Mutual Information (MI) for model alignment, requiring only the pre-trained T2I model and a simple fine-tuning process, outperforming current state-of-the-art methods.
weaknesses: 1. Comparison methods, "Attend and Excite (A&E) (Chefer et al., 2023b), Structured Diffusion Guidance(SDG) (Feng et al., 2023b) and Semantic-aware Classifier-Free Guidance (SCG) (Shen et al., 2024)"  mentioned in line 329, is implemented on SD1.4, but the proposed work is implemented on SD 2.1, which is powerful than SD1.4 and may introducing evaluation bias.
2. The experiments, the train set, and the test set are split from the same dataset, but this may exist some correlations, what's the results on out-of-distribution prompts?
3. Although mutual information provides a theoretical explanation, it essentially uses classifier guidance as a loss for fine-tuning. Given that this approach is so simple, has there been any similar attempt in previous work?

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 2
questions: Please see the waeakness.
soundness: 2
strengths: - The idea of introducing  self-supervised fine-tuning manner is interesting.

- Mutual Information in the pipeline is simple and effective.

- It seems is a plug-and-play module, which is useful for most T2I models.
summary: This paper introduces Mutual Information (MI) to guide model alignment, which uses self-supervised fine-tuning manner. It relies on a point-wise MI estimation between prompts and images to create a synthetic fine-tuning set for improving model alignment.
weaknesses: - More detailed ablations are needed. The authors employ MI as the metric to select fine-tuning samples, which eliminates the extra usage of other models. However, what if we use SOTA VQA models as the metric? Intuitively, SOTA VQA models are more precise than the MI metric.

- An inherent drawback of MI is that it can measure how much help comes from the prompt but cannot guide in the right direction. For example, in cases of color misalignment, how should we deal with this issue?

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. How were the 700 prompts selected for the MI quantitative analysis?
soundness: 2
strengths: 1. The main problem "Is mutual information meaningful for alignment?" is compelling and necessary, showing the potential of MI as a new direction for text-image alignment
2. The paper is well-organized and easy to follow.
3. The proposed fine-tuning approach is intersting and seems to have predictive power.
summary: The proposed MI-TUNE introduces a self-supervised fine-tuning approach to enhance text-to-image alignment in diffusion models. At its core, the method leverages Mutual Information (MI) between text prompts and their corresponding generated images to improve model alignment, eliminating the need for human annotation. The method generates multiple images per prompt, selects the top-K aligned ones based on MI estimation, and uses these for fine-tuning.
weaknesses: 1. MI scores are missing from comparison tables and images despite being central to the method.
2. Figure 1 only demonstrates MI effectiveness on simple category prompts (color, texture, shape), lacking validation on more challenging cases like spatial relationships or complex compositions

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Please see the weakness part. How about the performance of MI Tuning on DiT based models or Flow-based model, since the flow-based model reveals stronger generation capability comparing to traditional DDPM training. If MI tuning strategy fails to have good performance on flow-based training, the impact may be weaker.
soundness: 3
strengths: 1.	This paper claims to be the first to introduce the mutual information (MI) for T2I model training alignment. And the MI is proven to be effective on model alignment and strongly agrees with well-established metrics.
2.	MI-TUNE results on several measurements (BLIP-VQA, HPS and Human) on image features reveals state-of-the-art performance.
3.	The paper also provides the proof for MI index to have a valid point-wise manner. Introducing mutual information into T2I model tuning is interesting, it may benefit the generative model training.
4.	The paper is easy to follow with algorithm and pseudo code provided.
summary: This paper proposes MI-TUNE, a finetuning strategy for T2I models. This strategy introduces mutual information calculation into the T2I model tuning process to guide model alignment. In brief, the tuning method uses self-supervised fine-tuning and relies on a point-wise MI estimation between prompts and images to create a synthetic fine-tuning set for improving model alignment. This paper claims to be the first to introduce mutual-information calculation into T2I model training, leading to effective guidance for model alignment. The MI index is also measured and compared with well-established metrics and user study to prove the rationality. The experiments on MI are performed and evaluated on T2I-CompBench and user studies.
weaknesses: 1.	Note that the paper mainly focuses on SD-based (SD 2.1, SDXL) models. These models are mostly the same styles, e.g., similar network structures and traditional denoising training strategies. Is there any possibility that the MI tuning incorporated with flow-based models like DiT-based models (SD3, Pixart series or so). And it is interesting to see if the proposed MI tuning behaves different with different types of models.
2.	The evaluations on MI mainly focus on only simple semantic concepts like color, shape and texture. Is MI-tuning sensitive to object numbers or so?
3.	The paper fixes the denoising steps to 50 when inferencing an image, are there any differences in performance of MI-tuning when using different steps except 50?
4.	In quantitative analysis of Sect. 3.1, the paper mentions that the point-wise MI ranks images and select 1st, 25th and 50th as the representative images. Why the three images are representative? This needs more detailed explanations. Also, the reason of the selection needs quantitative analysis.
5.  Some of the ablations mentioned in previous sections are hard to locate in the following contents, the writing can be improved in this part.

Meta-review: "This paper proposes a method base on mutual information to improve text-to-image alignment in generative text-to-image generative models.
The strengths of the paper indicated in the reviews include: well written, simple finetuning process that does not require additional data or models to improve performance, novel idea & interesting direction, and the proposed method can be combined with many T2I models.
Weaknesses: lacks evaluation on out-of-distribution prompts, misses some comparisons, MI scores not reported while central part of method, validation limited to relatively simple prompts,  study limited to Unet diffusion models and missing evaluation of transformer-based models with flow training, missing ablations on nr of inference steps.

Decision: Accept (Poster)