Information Theoretic Text-to-Image Alignment

Authors: CHAO WANG, Giulio Franzese, Alessandro Finamore, Massimo Gallo, Pietro Michiardi
Primary area: generative models
Keywords: Diffusion model, Text-image alignment, Mutual information

Abstract:
Diffusion models for Text-to-Image (T2I) conditional generation have recently achieved
tremendous success. Yet, aligning these models with user’s intentions still involves a
laborious trial-and-error process, and this challenging alignment problem has attracted
considerable attention from the research community. In this work, instead of relying on
fine-grained linguistic analyses of prompts, human annotation, or auxiliary vision-language
models, we use Mutual Information (MI) to guide model alignment. In brief, our method
uses self-supervised fine-tuning and relies on a point-wise MI estimation between prompts
and images to create a synthetic fine-tuning set for improving model alignment. Our
analysis indicates that our method is superior to the state-of-the-art, yet it only requires
the pre-trained denoising network of the T2I model itself to estimate MI, and a simple
fine-tuning strategy that improves alignment while maintaining image quality. Code available at https://github.com/Chao0511/mitune.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730026132776

thread_type:
official_review

<thread_object>
creator: reviewer_8y4v

<content>
summary:
This paper proposes MI-TUNE, a finetuning strategy for T2I models. This strategy introduces mutual information calculation into the T2I model tuning process to guide model alignment. In brief, the tuning method uses self-supervised fine-tuning and relies on a point-wise MI estimation between prompts and images to create a synthetic fine-tuning set for improving model alignment. This paper claims to be the first to introduce mutual-information calculation into T2I model training, leading to effective guidance for model alignment. The MI index is also measured and compared with well-established metrics and user study to prove the rationality. The experiments on MI are performed and evaluated on T2I-CompBench and user studies.

strengths:
1.	This paper claims to be the first to introduce the mutual information (MI) for T2I model training alignment. And the MI is proven to be effective on model alignment and strongly agrees with well-established metrics.
2.	MI-TUNE results on several measurements (BLIP-VQA, HPS and Human) on image features reveals state-of-the-art performance.
3.	The paper also provides the proof for MI index to have a valid point-wise manner. Introducing mutual information into T2I model tuning is interesting, it may benefit the generative model training.
4.	The paper is easy to follow with algorithm and pseudo code provided.

weaknesses:
1.	Note that the paper mainly focuses on SD-based (SD 2.1, SDXL) models. These models are mostly the same styles, e.g., similar network structures and traditional denoising training strategies. Is there any possibility that the MI tuning incorporated with flow-based models like DiT-based models (SD3, Pixart series or so). And it is interesting to see if the proposed MI tuning behaves different with different types of models.
2.	The evaluations on MI mainly focus on only simple semantic concepts like color, shape and texture. Is MI-tuning sensitive to object numbers or so?
3.	The paper fixes the denoising steps to 50 when inferencing an image, are there any differences in performance of MI-tuning when using different steps except 50?
4.	In quantitative analysis of Sect. 3.1, the paper mentions that the point-wise MI ranks images and select 1st, 25th and 50th as the representative images. Why the three images are representative? This needs more detailed explanations. Also, the reason of the selection needs quantitative analysis.
5.  Some of the ablations mentioned in previous sections are hard to locate in the following contents, the writing can be improved in this part.

questions:
Please see the weakness part. How about the performance of MI Tuning on DiT based models or Flow-based model, since the flow-based model reveals stronger generation capability comparing to traditional DDPM training. If MI tuning strategy fails to have good performance on flow-based training, the impact may be weaker.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730515511911

thread_type:
official_review

<thread_object>
creator: reviewer_lakz

<content>
summary:
The proposed MI-TUNE introduces a self-supervised fine-tuning approach to enhance text-to-image alignment in diffusion models. At its core, the method leverages Mutual Information (MI) between text prompts and their corresponding generated images to improve model alignment, eliminating the need for human annotation. The method generates multiple images per prompt, selects the top-K aligned ones based on MI estimation, and uses these for fine-tuning.

strengths:
1. The main problem "Is mutual information meaningful for alignment?" is compelling and necessary, showing the potential of MI as a new direction for text-image alignment
2. The paper is well-organized and easy to follow.
3. The proposed fine-tuning approach is intersting and seems to have predictive power.

weaknesses:
1. MI scores are missing from comparison tables and images despite being central to the method.
2. Figure 1 only demonstrates MI effectiveness on simple category prompts (color, texture, shape), lacking validation on more challenging cases like spatial relationships or complex compositions

questions:
1. How were the 700 prompts selected for the MI quantitative analysis?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730774104149

thread_type:
official_review

<thread_object>
creator: reviewer_8jmm

<content>
summary:
This paper introduces Mutual Information (MI) to guide model alignment, which uses self-supervised fine-tuning manner. It relies on a point-wise MI estimation between prompts and images to create a synthetic fine-tuning set for improving model alignment.

strengths:
- The idea of introducing  self-supervised fine-tuning manner is interesting.

- Mutual Information in the pipeline is simple and effective.

- It seems is a plug-and-play module, which is useful for most T2I models.

weaknesses:
- More detailed ablations are needed. The authors employ MI as the metric to select fine-tuning samples, which eliminates the extra usage of other models. However, what if we use SOTA VQA models as the metric? Intuitively, SOTA VQA models are more precise than the MI metric.

- An inherent drawback of MI is that it can measure how much help comes from the prompt but cannot guide in the right direction. For example, in cases of color misalignment, how should we deal with this issue?

questions:
Please see the waeakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730798259568

thread_type:
official_review

<thread_object>
creator: reviewer_a9zb

<content>
summary:
This work proposes a self-supervised approach using Mutual Information (MI) for model alignment, requiring only the pre-trained T2I model and a simple fine-tuning process, outperforming current state-of-the-art methods.

strengths:
1. The proposed method does not require additional image datasets for training.
2. The idea is relatively novel.

weaknesses:
1. Comparison methods, "Attend and Excite (A&E) (Chefer et al., 2023b), Structured Diffusion Guidance(SDG) (Feng et al., 2023b) and Semantic-aware Classifier-Free Guidance (SCG) (Shen et al., 2024)"  mentioned in line 329, is implemented on SD1.4, but the proposed work is implemented on SD 2.1, which is powerful than SD1.4 and may introducing evaluation bias.
2. The experiments, the train set, and the test set are split from the same dataset, but this may exist some correlations, what's the results on out-of-distribution prompts?
3. Although mutual information provides a theoretical explanation, it essentially uses classifier guidance as a loss for fine-tuning. Given that this approach is so simple, has there been any similar attempt in previous work?

questions:
Referring Weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1732118627359

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank the reviewers for their work in assessing our paper, and for the helpful comments and questions. Thank you for noting the originality of our approach, that blends self-supervision and pointwise mutual information estimation, and for recognizing that our method is “plug-and-play”, as it does not need auxiliary procedures for text-to-image alignment. Reviewers also duly noted the new state-of-the-art results we obtained on both standard benchmarks and on difficult real-world data.

We have prepared a detailed answer to all comments and questions, hoping that our clarifications will lead to an even more positive assessment of our work. We also uploaded a revised version to include the following:

* Clarified in Sec.3 that we use all prompts from T2I-Compbench
* Added Appendix I to extend Fig.1 by including examples from all T2I-CompBench categories, and updated Fig.1 caption to refer to the new appendix. 
* Clarified in Sec.4.3  that results in Table 1 refer to evaluating all methods on SD2.1
* Added Appendix J to discuss the relationship between metrics distributions and rank
* Reworked the appendix table of content to ease the understanding of the scope of each appendix and reference them to the relevant parts of the main paper. 
* We will integrate additional results of MI-TUNE applied to a DiT-based architecture, specifically PixArt, as suggested by one reviewer. Experiments are in progress, and we will provide an update message as soon as they finish.
</content>
</thread_object>

</thread>


Metareview:
metareview:
"This paper proposes a method base on mutual information to improve text-to-image alignment in generative text-to-image generative models.
The strengths of the paper indicated in the reviews include: well written, simple finetuning process that does not require additional data or models to improve performance, novel idea & interesting direction, and the proposed method can be combined with many T2I models.
Weaknesses: lacks evaluation on out-of-distribution prompts, misses some comparisons, MI scores not reported while central part of method, validation limited to relatively simple prompts,  study limited to Unet diffusion models and missing evaluation of transformer-based models with flow training, missing ablations on nr of inference steps.