IMG: Calibrating Diffusion Models via Implicit Multimodal Guidance
Authors: name: Jiayi Guo, name: Chuanhao Yan, name: Xingqian Xu, name: Yulin Wang, name: Kai Wang, name: Gao Huang, name: Humphrey Shi
Organization: SHI Labs

Ensuring precise multimodal alignment between diffusion-generated images and
input prompts has been a long-standing challenge. Earlier works finetune
diffusion weight using high-quality preference data, which tends to be limited
and difficult to scale up. Recent editing-based methods further refine local
regions of generated images but may compromise overall image quality. In this
work, we propose Implicit Multimodal Guidance (IMG), a novel
re-generation-based multimodal alignment framework that requires no extra data
or editing operations. Specifically, given a generated image and its prompt,
IMG a) utilizes a multimodal large language model (MLLM) to identify
misalignments; b) introduces an Implicit Aligner that manipulates diffusion
conditioning features to reduce misalignments and enable re-generation; and c)
formulates the re-alignment goal into a trainable objective, namely Iteratively
Updated Preference Objective. Extensive qualitative and quantitative
evaluations on SDXL, SDXL-DPO, and FLUX show that IMG outperforms existing
alignment methods. Furthermore, IMG acts as a flexible plug-and-play adapter,
seamlessly enhancing prior finetuning-based alignment methods. Our code will be
available at https://github.com/SHI-Labs/IMG-Multimodal-Diffusion-Alignment.