D-Fusion: Direct Preference Optimization for Aligning Diffusion Models with Visually Consistent Samples
Authors: Zijing Hu
Fengda Zhang
Kun Kuang
Venue: ICML 2025 poster (2025)
Keywords: Diffusion Models
Alignment
Reinforcement Learning
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: This paper introduces D-Fusion, a compatible approach that constructs DPO-trainable visually consistent samples, to further enhance the prompt-image alignment of diffusion models with reinforcement learning.

Abstract: The practical applications of diffusion models have been limited by the misalignment between generated images and corresponding text prompts. Recent studies have introduced direct preference optimization (DPO) to enhance the alignment of these models. However, the effectiveness of DPO is constrained by the issue of visual inconsistency, where the significant visual disparity between well-aligned and poorly-aligned images prevents diffusion models from identifying which factors contribute positively to alignment during fine-tuning. To address this issue, this paper introduces D-Fusion, a method to construct DPO-trainable visually consistent samples. On one hand, by performing mask-guided self-attention fusion, the resulting images are not only well-aligned, but also visually consistent with given poorly-aligned images. On the other hand, D-Fusion can retain the denoising trajectories of the resulting images, which are essential for DPO training. Extensive experiments demonstrate the effectiveness of D-Fusion in improving prompt-image alignment when applied to different reinforcement learning algorithms.

D-Fusion: Direct Preference Optimization for Aligning Diffusion Models with Visually Consistent Samples
Zijing Hu" 1
Fengda Zhang" 2
Kun Kuang'
Abstract
DPO
DPO-D-Fusim
SD
DPO
DPO-D-Fusin Esion
The practical applications of diffusion models have been limited by the misalignment between generated images and corresponding text prompts. Recent studies have introduced direct preference optimization (DPO) to enhance the alignment of these models. However, the effectiveness of DPO is constrained by the issue of visual inconsis- tency, where the significant visual disparity be- tween well-aligned and poorly-aligned images prevents diffusion models from identifying which factors contribute positively to alignment during fine-tuning. To address this issue, this paper in- troduces D-Fusion, a method to construct DPO- trainable visually consistent samples. On one hand, by performing mask-guided self-attention fusion, the resulting images are not only well- aligned, but also visually consistent with given poorly-aligned images. On the other hand, D- Fusion can retain the denoising trajectories of the resulting images, which are essential for DPO training. Extensive experiments demonstrate the effectiveness of D-Fusion in improving prompt- image alignment when applied to different rein- forcement learning algorithms.
moolay cnting
Rorse playing baskettall"
"a rabbit driving a car
Figure I. (Misalignment) Diffusion models (e.g. Stable Diffusion (SD) (Rombach et al., 2022)) often encounter the issue that the generated images do accurately match the given prompts. Ex- isting RL-based fine-tuning methods (e.g.. DPO (Wallace et al., 2023)) have limited effectiveness in improving the alignment. For each set of images above, we use the same seed for sampling.
models transform random noises to corresponding images via a sequential denoising process. However, as shown in Figure 1, diffusion models often encounter the issue of prompt-image misalignment (Jiang et al., 2024; Mrini et al., 2024). Prompt-image misalignment refers to the problem that the generated images do not accurately match the given text prompts, which limits the real-world applications of diffusion models.
1. Introduction
To address this issue, recent studies have explored incorpo- rating reinforcement learning (RL.) algorithms to fine-tune pre-trained diffusion models (Black et al., 2024; Clark et al., 2024; Fan et al., 2023; Wallace et al., 2023; Xu et al., 2023; Yang et al., 2024a;b; Hu et al., 2025). In the paradigm of RL, the step-by-step denoising process of diffusion models is reinterpreted as sequential decision-making problem. In this formulation, the intermediate noisy image each timestep regarded as state, while each denoising opera- tion corresponds to action. Among these RL algorithms, direct preference optimization (DPO) stands out for its ad- vantage of climinating the need for an explicit reward model, making it a widely adopted approach (Wallace et al., 2023; Yang et al., 2024a). As illustrated in Figure 2(a), researchers first sample images from the diffusion model with given
Diffusion models have made remarkable success in various domains, such as medicine (Xu et al., 2022), roboties (Chi al., 2024), and 3D synthesis (Poole et al., 2022). Recently, the application of diffusion models in the field of text-to- image generation has gained widespread attention (Ho et al., 2020; Dhariwal & Nichol, 2021). Under the guidance of the given text descriptions, usually called prompts, these
'Equal contribution 'College of Computer Science and Technol- ogy, Zhejiang University, Hangzhou, China *College of Computing and Data Science, Nanyang Technological University, Singapore. Correspondence to: Kun Kuang <kunkuang @ wwwtaaanngzjuuuu..u..... edu.e..
Proceedings of the nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

D-Fusion: DPO for Aligning Diffusion Models with Visually Consistent Samples
For Language Models:
For Diffusion Models:
deg
eating
sandwich
Ry
Xe-1
Ag
correct
Step 3: Training
High Preference
Low Preference
monkey
eating
sandwich
complete derssion-mekingg sequence
missed seriision-wkklig seguesee (b) Editing Samples in Language Models and Diffusion Models.
Prompts egg. a monkey eating a sandwich
...
xF
x
x
Reference Image
Step 2:
Evaluation
High Preference
Diffusion Model
Basc Image Preference
shaged goise
shared random naise
Step Sampling
x2
Target Image
Images
High Preference
(a) Pipeline Fine-tuning Diffusion Models with DPO.
(c) Self-Attention Based Denoising Trajectories Fusion.
Figure 2. (Visual Inconsistency) When people train diffusion models with direct preference optimization (DPO), the visual disparity between well-aligned and poorly-aligned images are enormous. This visual inconsistency limits the success of DPO in enhancing diffusion models. Meanwhile, the visually consistent samples obtained through manual editing lack denoising trajectories and are not suitable for training. To this end, we introduce D-Fusion, which constructs RL-trainable visually consistent samples.
prompts, and then evaluate the alignment between images and prompts via human preference or model prediction. These sampled images, along with their preference orders and denoising trajectories, enhance the alignment of diffusion model.
tions (Huang et al., 2024; Bai et al., 2024) present in the text output the language model, thereby obtaining factual training data (i.e., high-preference data) with consistent lin- guistic style. Since the editing or annotations to text output are at the token-level, and the decision-making sequence in language model is constructed token-by-token (Rafailov et al., 2024), RL training can still proceed. By performing DPO with the pairs consisting of hallucinated text and cor- responding factual text, the language model receives dense reward signals and achieves fine-grained alignment.
be further used in DPO to
However, DPO has so far achieved limited success in im- proving prompt-image alignment, primarily due to the issue of visual inconsistency in the training data. Visual incon- sistency refers to the disparity between images in terms of structure, style or appearance, which commonly observed in the images denoised from different noises. As shown in Figure 2(a), high-preference images (i.e., well-aligned im- ages) differ from low- www-prfereene images (i.e., moorly-aligned images) not only in the alignment-related factors, but also in unrelated factors (e.g., background). Interfered by unre- lated factors, it is difficult for the model to identify which factors contribute positively to alignment. Meanwhile, with great differences, it is difficult to tell which image aligns better sometimes (e.g., an image with only monkey and image with only sandwich). As a result, learning effective denoising policies from those samples becomes challenging.
However, these methods on language models fail when ap- plied to the text-to-image diffusion models. As illustrated in Figure 2(b), the decision- model is constructed timestep-by-timeste ing methods, such as manual editing, Imagen Editor (Wang et al., 2023) and Imagic (Kawar et al., 2023), are capable of both aligning images and maintaining visual consistency. Yet, these methods perform editing at the pixel-level, caus- ing the loss of the timestep-be seiiie--yyttimee timestep decision-making making se- quences. Once edited to better align with the prompts, these images lack corresponding denoising trajectories, making them unsuitable for RL. fine-tuning. This motivates us to ask: How can we generate earanlltr visually consistent image pairs to fine-tune diffusion models?
sequence of the diffusion Existing edit-
We believe that performing DPO with visually consistent image pairs can help diffusion models learn effective poli- cies. Recent studies have corroborated similar perspectives on RL training of large language models (LLMs) and multi- modal large language models (MLLMs) (Kong et al., 2025; Yu et al., 2024). As illustrated in Figure 2(b), these studies make fine-grained editing or annotations to the hallucina-
In this paper, we address this challenge by introducing D- Fusion: self-attention based Denoising trajectory Fusion, method to construct RL-trainable visually consistent images. Our method offers key innovations in two phases. (1) In the
2

---

D-Fusion: DPO for Aligning Diffusion Models with Visually Consistent Samples
sampling phase, we propose apply self-attention fusion between high-preferrnce sample (called reference image) and low-preference sample (called base image) under the guidance of an auto-extracted mask to obtain new sample (called target image), as illustrated in Figure 2(c). The mask, which is derived from the denoising process of the reference image, can reveal the position of alignment-related area. By area,
the increasing demand for more interactive and user-driven generation, researchers begin exploring methods to incor- porate controllability into these models (Cao et al., 2024; Tong et al., 2023). A variety of studies aim to control the generation process of diffusion models with specific con- ditions, such as class labels (Dhariwal & Nichol, 2021; Ho & Salimans, 2022), layouts (Zheng et al., 2024), im- ages (Preechakul et al., 2022) and audios (Yang et al., 2023). With the introduction of text encoders, diffusion models gain the ability to generate images from text (Rombach et al., 2022). Subsequent studies therefore focus on fine-tuning the pre-trained text-to-image diffission models to improving alignment (Jiang et al., 2024; Lee et al., 2023). Among them, RL has been widely employed to enhance the controllability of diffusion models (Black et al., 2024: Clark al., 2024; Fan al., 2023; Wallace et al., 2023; Xu et al., 2023; Yang et al., 2024a:b; Hu et al., 2025). In this paper, by mitigating the issue of visual inconsistency, we further improve the performance of RL in training diffusion models.
applying self-attention fusion in the alignment-relate the target image becomes as well-aligned as the reference image. Simultaneously, with shared random noise, the target image exhibits a high degree of visual consistency with the base image. (2) In the training phase, since the self-attention fusion is applied step-by-ster along the denoising process, we collect the intermediate states to form the trajectories, which are the necessity of RL training. By performing DPO between the base images and corresponding target images, the diffusion models can achieve better prompt- image alignment than those fine-tuned with naive samples.
We conduct comprehensive experiments with three lists of prompts on Stable Diffusion (Rombach et al., 2022). The three prompt lists respectively consider the behavior of the object, the attribute of the object, and the positional relation- ship between the objects, which we believe can encompass broad spectrum of commonly used prompt types in image generation. Furthermore, we apply D-Fusion to variety of RL algorithms for fine-tuning diffusion models, including DPO (Wallace et al., 2023), DDPO (Black et al., 2024) and DPOK (Fan et al., 2023). Experimental results show that D-Fusion can effectively enhance the alignment of diffusion models across different prompts, and is compatible with different RL algorithms.
2.2. Reinforcement Learning with Fine-grained Data
Reinforcement learning is training paradigm that has played an important role in improving aligmment of both diffusion models and language models. In the area of trust- worthy LLMs/MLLMs, alignment to human preference has attracted widespread attention (Liu et al., 2024; Tu et al., 2023; Zhu et al., 2025; Yang et al., 2025), where reinforce- ment learning from human feedback (RLHF) has been em- ployed accordingly (Bai et al., 2022; Rafailov et al., 2024; Ouyang et al., 2022). Current language models generate text in an auto-regressive manner (Vaswani et al., 2023), thus the loken-by-token generation process can be regarded as a Markov decision process. Recently, researchers perform fine- graingr corrections or assign fine-grained human feed- back to the textual training data (Kong et al., 2025; Yu et al., 2024; Wu et al., 2023). Fine-grained data can provide dense reward signals to RL, thus achieving impressive fine-tuning results. In this paper, by employing denoising trajectory fu- sion, we provide visually consistent samples for RL training of diffusion models, which have similar fine-grained effects.
The main contribution of this work can be summarized as ! (1) We for the first time highlight the necessity of fine- tuning diffusion models with visually consistent image pairs when applying DPO, and discuss the challenge in obtaining RL-trainable visually consistent images. (2) We introduce D-Fusion, a compatible approach construct visually con- sistent samples and corresponding denoising trajectories, where the latter is curial for RI. training, to address the above challenge. (3) Comprehensive experimental results demonstrate the effectiveness of D-Fusion in improving prompt-image alignment when applied to different prompts and different RL algorithms.
2.3. Attention Control for Diffusion Models
The attention mechanism has garnered considrralle interest and sparked a wealth of research (Vaswani et al., 2023; Doso- vitskiy et al., 2021: Wang et al., 2024). In diffusion models, some studies have demonstrated how cross-attention maps in the denoising process determine the layouts of generated images (Hertz et al., 2022; Brooks et al., 2023; Mokady et al., 2022). Additionally, other studies have explored the role of self-attention layers in these models (Cao et al., 2023; Tumanyan et al., 2022; Shi et al., 2024). These studies can edit images by controlling the attention layers, and some of them have the potential to preserve the denoising tra-
2. Related Work
2.1. Controllable Generation with Diffusion Models
Diffusion models have demonstrated impressive ability in generating high-quality and high-fidelity images (Ho et al., 2020; Song& & Ermon, 2020; Peebles& & Xie, 2023). With
'The code for this work is available at this repository: htnp://gituhh.om//uuzjjjgg/////
3

---

D-Fusion: DPO for Aligning Diffusion Models with Visually Consistent Samples
howw
Kt
X
k/2xw/22
N//ww//4
self-otention crossociention XX transforiter otcention block
h/8xww88
(a) Overview of the Denosing Process in U-Net Based Diffusion Models.
prompt: monkey eating sandwich
merge> resize flatten
binarize item- related maps
8/4
monkey
kyXw
a monkey eating a sandwich
sheep eating a sandwich
cross-ottn maps (reference image)
sandwiin
mask
Ny XWI
tiger playing basketball
cow playing basketball
self-ottn keys
seif-ottn keys
aiffusion model
shared
a raccoon driving car
rabbit driving a car
self-ortn values (reference image)
self-attn volues (target image)
noise
(torget image) (base image)
attention fusion
shared noise
(b) Denoising Trajectory Fusion.
(c) Visually Consistent Samples.
Figure 3. (Method Overview) We propose D-Fusion construct RL-trainable visually consistent samples. Each layer of the U-Net based diffusion models contains several transformer attention blocks, and each block contains a self-attention module and cross-attention module. (b) D-Fusion constructs visually consistent samples through two steps: cross-attentio mask extraction and self attention fusion. (c) Examples of visually consistent samples. Each set consists of three images: the reference image, the base image, and the target image. The target images are not only well-aligned as the reference images but also maintain visual consistency with the base images.
jectories. However, these methods generally transfer an image from one prompt to another, which does not corre- spond our task of aligning an image with corresponding prompt. Nevertheless, these methods inspire to explore iitention-based techniques in this paper. We present some observations on these methods in Appendix B.
2022), each iteration is performed by applying the following denoising formula:
Px Xr, c) -x (a c), o,12), (,)
(I)
where denotes current timestep, M4 represents the predic- tion made by a diffusion model parameterized by 0, and a; is the fixed timestep-dependent variance. The reverse pro- produces a denoising trajectory {x7, {xT,XT .., ending with sample X
3. Method
In this section, we start by formulating the problem, fol- lowed by detailed introduction to D-Fusion, covering both the sampling and training phases.
A Atention Mechanism in Diffusion Models. Transformer attention blocks (Vaswani et al., 2023) have been applicd in each layer of the U-Net (Ronneberger al., 2015) based diffusion models. As shown in Figure 3(a), the U-Net based diffusion models contain several down-sampling layers, middle layer, and corresponding up-sampling layers. Fur- thermore, each layer contains several transformer attention blocks, and each attention block in diffusion models con- tains self-attention module and a cross-attention module. The attention mechanism can be formulated as follows:
3.1. Problem Formulation
Text-to-Image Diffusion Models. In this work, we consider pre-trained text-to-image diffusion models p(xo c). which generate sample Xo conditioned on a textual prompt Be- ginning with random noise Xy ~ NN I), diffusion models iteratively transform the noise through T steps into a clear image Xo that matches the given prompt (Sohl-Dickstein al., 2015; Dhariwal & Nichol, 2021). Building upon the samplers of DDPM (Ho al., 2020) and DDIM (Song et al.,
QKT
Attention(Q2 (Q, V) softmax
-)V,
(2)
V dkey

---

D-Fusion: DPO for Aligning Diffusion Models with Visually Consistent Samples
where Q Rmxddeey are queries projected from image fea- tures, and K RRuxdee,, V E TRxxdeaauee are keys and values projected from image features (in self-attention mod- ule) or prompt embeddings (in cross-attention module). In this formula, softmax(
Diffusion), and by x wy represent the image resolution at layer of the U-Net. Inspired by previous work (Hertz et al., 2022; Cao et al., 2023), the cross-attention maps contain sufficient information about shapes and structures of the generated images, among which the first up-sampling layer (with resolution 4 x w = 16 x 16 in Stable Diffusion) performs the best. Therefore, we average the cross-attention maps across all heads and all attention blocks in the first up-sampling layer, and extract mask from them.
QK") ) is commonly referred to as Vdncy attention maps, represented by A.
4
Denoising as a Decision-Making Problem. The denoising process in diffusion models can be formulated as a sequen- rial cecisionnmakinng problem. The process can be defined by tuple (S, (S,A,F. P. R, Tq). where S is the state space, AA is the action space, P is the transition function, R is the reward function, and Th is the decision-making policy. At each timestep t, the state $1 E S is represented by (c, t, xt). Le., the text prompt, the current timestep, and the noisy image the current timestep. The action 0 E A refers to the denoising operation that generates the next noisy image Xt-1- The transition P(s++1 $t, ae) specifies the distri- bution over the next state St+1 given the current state St and action ae, and is provided by corresponding samplers in DDPM and DDIM. The reward R(c, Xo) corresponds to the prompt-image alignment score in our settings, which can be given by human preference or model evaluation. And the policy is defined as [a 8t) = po (x,-1 Xt, c), which deseribes how to select the current action based current state. By adopting this formulation, we can enhance the prompt-image alignment in diffusion models by maximiz- ing the following objective:
Formally, after averaging and reshaping, the cross-attention maps are denoted as Apross E xgixxxle where |c] is the number of tokens in prompt c. The i-th attention map Aprou i] indicates the extent to which each pixel in the image should pay attention to the i-th token in the prompt. Let O. = {ol,..., oo represents index list of the item- related tokens in prompt c, then the mask My can be ex- tracted with the following formula:
Mr + [ [. ol 2 thr,)) )11 P1:0: 0), on
(4)
where thr, is a hyperparameter that defines the mask thresh- old for the o-th token, 1 is the all-one matrix, and 0 is the all-zero matrix. In this formula, we first binarize the atten- tion maps to generate the masks for corresponding items, as shown in the Figure 3().. Afterwards, we merge them into one mask through XOR operation. The resulting mask Mt E B x where IB {0. 1), reveals the position of the items mentioned by the prompt in the reference image.
JRL( = xe--pc)).xo-pp(x00) le) [R(xo c)],
(3)
Self-Attention Fusion. To generate ideal target image 1e our approach is based on the idea of having the prompt- related area imitate the reference image I", while the other area retain the features of base image lb. Inspired by previ- ous work (Cao et al., 2023; Tumanyan et al., 2022), the (i.j) entry in the self-attention maps A E IR (he x (hexx)) indicates the extent to which the i-th pixel in the image should pay attention to the j-th pixel at timestep t. There- fore, we design the mechanism named self-attention fusion, to control the attention allocation in 7" Starting with the same random noise as 1o, we progressively denoise it with diffusion model, and apply self-attention fusion at each timestep t as follows. Firstly, we resize the mask M, to match the image resolution of the current layer, resulting in new mask M. E Bhixwi, Afterwards, we manipulate the keys and values in self-attention as Eq.(
where p(c) is a uniform distribution over the candidate prompts.
3.2. Sampling: Denoising Trajectory Fusion
When employing reinforcement learning, people first sam- ple a set of images 4I.... In. and reserve their denoising trajectories. These images contain both well-aligned and poorly-aligned ones, and can be further used as training data for RL. We refer to these well-aligned images as ref- erence images and poorly aligned-images The goal of our method is to generate target image Io that is both as well-aligned as a given reference image I", and visually consistent with given base image 1", where I" and I are generated with the same textual prompt c. D-Fusion reaches this goal through the following two steps: cross-attention mask extraction and self-attention fusion. Formally, the denoising trajectories of I" and 1 are rep- resented as {x,, xf-.......................... x and {x/, x/ x x respectively.
base images.
Ka K" Flatten (M,) + K" (1 Flatten (M))), Va new = V a Flatten M + V o (1 - Fllttenn( M.)):
(5)
where the signal o is Hadamard product 2, the K R x(xxxxi) yr E (he Xdmatuu are keys and val-
Cross-Attention Mask Extraction. Firstly, we extract a mask M, from reference image at each timestep t, i.e., at the denoising process from x to x"_1- 1. Let h x w represent the image resolution of x (h x w = 64 x 64 in Stable
Foo two matrices A and B, the Hadamard product is A B [an] o [byj] = (a.bu]].
5

---

D-Fusion: DPO for Aligning Diffusion Models with Visually Consistent Samples
ues of I", the K Va are keys and values of 1a generated at current denoising step, and Flatten() reshapes Mr into ((hxx)))x1, enabling it to compute the Hadamard product with the keys and values after auto-broadcasting.
4.1. Experimental Setup
Diffusion Models. We use Stable Diffusion (SD) 2.1- base (Rombach et al., 2022), one of the most advanced diffusion models, the base model for the experiments. We employ DDIM (Song et al., 2022) as the sampler. The weight of noise in DDIM sampler is set to 1.0, which de- cides the degree of randommess at each denoising step. We apply Low-Rank Adaptation (LoRA) (Hu et al., 2021) for efficient fine-tuning. Following the previous work (Black et al., 2024), the total denoising timesteps 7 is set to 20. Each experiment is conducted with three different seeds.
By applying self-attention fusion, the diffusion model can generate the target image I" that is not only as well-aligned as 1", but also visually consistent with 1 On one hand, by injecting the fused keys Keww the self-attention maps allocate attention to the prompt-related area with reference to I". Subsequently, by injecting the fused values View the final image features in the prompt-related area also take into account the features of I" Thus the prompt-related area in To becomes well-aligned, as the reference image I" goes. On the other hand, by sharing the same random noise with 1h retaining the original queries, and retaining the keys and values in prompt-unrelated area, the target image /a also achieves visual consistency with 7o.
Prompt Templates. We construct the prompt lists based on three templates. The three templates consider the behavior of the object, the attribute of the object, and the positional re- lationship between the objects respectively.
(1) Template I: " "a(n) [animal] [activity|". The animal is chosen from the the list of 45 common animals given by previous work (Black et al., 2024), and the activity is chosen from the list: "ceating a sandwich", "driving a car" and "playing basketball". (2) Template 2: "a(n) [color] and [material] [object]". We select six common colors (e.g., red) and nine common ma- terials (e.g., wooden) for this template. The object list chosen from the Visual Relation Dataset (VRD) (Lu et al., 2016). We randomly combine colors, materials, and ob- jects to form the prompts. (3) Template 3: "the [object 1] [predicate] ] the [object 2/".. We select four position-related predicates: "above", "below", "on the left of"" and "on the right of We construct the prompts based on the annota- tions of VRD to ensure their rationality. The prompt list for each template contains 40 prompts for training, and 40 prompts for generalization test. We present the full prompt lists in Appendix G.
3.3. Training: DPO with Visually Consistent Samples
By applying denoising trajectory fusion based on Ib and with reference to I", we can get the well-aligned target image 1a along with its denoising trajectory xx ....,........................... xii- The direct preference optimization can therefore be employed with the image pair consisting of high-preference image 1" and low-preference image 1". Following traditional DPO (Wallace et al., 2023; Yang et al., 2024a), the diffusion model with parameters can be opti- mized with following objective:
T Σ log o
Do Po(x;- x², c)
-B Po (x xt, c)
x4, c) 1).
po po. (x-_1 x(,c) c) pa (xl
1
(6)
where is the sigmoid function, Gold is the parameters of diffusion model prior to update, and / is hyperparameter controlling the deviation from Pe to Pe... By introducing CLIP (Radford et al., 2021) to replace human in evaluating prompt-image alignment, it becomes possible to conduct multiple rounds of online learning, allowing the model to progressively adapt to new image distribution that aligns well with corresponding prompts. Beyond DPO, we further employ DDPO (Black et al., 2024) and DPOK (Fan et al., 2023) to fine-tune diffusion models with visually consistent samples. For a comprehensive description of implementa- tion details, we refer the readers to Appendix C.
Rewards and Evaluation Metries. We evaluate the prompt- image alignment by CLIPScore (Hessel et al., 2022). and also use it as the reward function (if needed). A higher CLIPScore represents better alignment. In terms of imple- mentation, we use Vit-H-14 CLIP model (Radford et al., 2021; Ilharco et al., 2021).
4.2. Qualitative Evaluation
We first evaluate the performance of D-Fusion when applied to DPO (Wallace al., 2023; Yang al., 2024a). After em- ploying DPO with or without D D-Fusion for the same training rounds, we sample a series of images from original model and fine-tuned models with same random seeds, as shown in Figure 4(top). The results qualitatively show that training diffusion models with visually consistent samples yields better performance in improving prompt-image alignment than training without them when employing DPO. We also conduct a human preference test with 22 independent hu- man raters (ranging from undergraduates to Ph.D.), who are asked to select the image that best aligns with corresponding
4. Experiments
In this section, we demonstrate the effectiveness of D-Fusion both qualitatively and quantitatively. Afterwards, we focus on ablation studies on denoising trajectories and RL algo- rithms, as well as demonstrating the generalization ability. For simplicity, we refer to DPO+D-Fusion (i.e., employing DPO with D-Fusion) as our method some places.
6

---

D-Fusion: DPO for Aligning Diffusion Models with Visually Consistent Samples
SD
DPO
Ours
SD
DPO
Ours
SD
DPO
Ours
SD
DPO
Ours
Template
"a rabbit driving car"
"a zebra playing basketball"
"a sheep eating sandwich"
"a deer driving car'
Template 2
yellow and marble lamp"
"a green and metallic table"
"a black and glass dog"
"a yellow and metallic dog "
Template 3
"the table below the laptop" "M
"the umbrella above the table"
"the dog the right of the vase" "the person the left bear"
Template
"a sheep playing basketball"
"a horse driving
"a lizard eating sandwich"
tiger riding bike"
Template
"a green and wooden dog"
"a white and wooten bow!"
"a yellow and fabric table"
"a white and marble car"
Template 3
"the jacket above the pants"
"the bag sink
"the basket on right the person" " "the on left of the chair"
Figure 4. (Qualitative Results) Examples of images generated by original model and fine-tuned models on three templates. For each set of images, we use the same random seed. both training prompts (top) and test prompts (bottom), models fine-tuned by DPO+D-Fusion achieves better prompt-image alignment compared to the original model and the models fine-tuned by naive DPO.
prompt from set of three images generated by different models. We report the average preference rates in Figure 6. The results indicate that the preference rates of the images generated by the models fine-tuned with our method con- sistently outperform those by original model (SD) and by the models fine-tuned with naive DPO on the three prompt templates. We present more samples in Appendix F.
the models fine-tuned by method almost always achieve higher alignment scores than the models fine-tuned by naive DPO. These results indicate that training with visually con- sistent samples can enhance diffusion models to a greater extent than training with naive DPO.
4.4. Ablation Study
We conduct ablation studies on denoising trajectories and RL algorithms. For the former, we investigate the effective- ness of training with the denoising trajectories generated through DDIM inversion (Song al., 2022; Mokady et al., 2022). For the latter, apply D-Fusion across different RL algorithms to assess its compatibility and performance.
4.3. Quantitative Evaluation
We also quantitatively demonstrate the alignment of the models fine-tuned by DPO with or without D-Fusion. As shown in Figure 5, conduct multiple rounds of fine- tuning the diffusion models with different methods. At each round, we use the same seed to sample the fine-tuned different models, and test the alignment scores of the gener- ated images. The results illustrate the alignment scores as the training progresses on the three prompt templates, where the x-axis represents the amount of image data used fine- tune the models, and y-axis represents the CLIPScore. It can be seen that after training with the same amount of data,
Comparison with DDIM Inversion. The goal of DDIM inversion is to estimate the initial random noise x from clear image Xe step by step, thus constructing a denois- ing trajectory (Xo. x ...., xine] in reverse order. this ablation study, we apply DDIM inversion visually con- sistent samples, and utilize the generated denoising trajec-

---

D-Fusion: DPO for Aligning Diffusion Models with Visually Consistent Samples
33A
94.70%
8:30
$4.47%
94.0%
2340
0.00
p312
0.04
832
0.26%
4.5%
8.71%
18%4
11.36%
Traning Dun
Dtuts
Lamplos
Templase2
Tanpbir 3
DPO---itasoo IPU
DPU+D
DDO-DD Pissin
SD
DPO
Templatz [:a]j) [enimsl]] [activity]
Templeic ags) [aske] [musicial] (ibjezt]
Tanplic (eiject 13 [praliond) the [1
Figure 6. (Human Evaluation) Hu- man preference rates for prompt-image alignment of the images generated by SD, DPO and our method.
Figure 5. (Alignment) Alignment curves of the diffusion models fine-tuned with or without D-Fusion on three prompt templates. Results show that training with D-Fusion can enhance the alignment of diffusion models to greater extent.
1.32
6.02
0.3
0.29
034
0371
Table 1. (Generalizationn Prompt-image alignment (measured by CLIPScore 1) of the images generated by the SD, DPO, and our method on three templates. The prompts for generalization test. not used during training.
6379
ClPPSwee
CUPSer
0:31
0.23
4224
0.5
000 DDIM Invercian - -DOOOD--Fooo D-Flscio
DDPO
DDIM Lnvesiion
DPOK DDBK Inveriioz
-POK-D D---D--aaaa Fasias
Methods
Temp.
Temp.2
Temp.3
(4) and D-Fusion
DDPO and D-Fasion
DPOK and D-Fusion
SD
0.3725
0.3396
0.3213
Figure 7. (Ablation Study) The ablation studies on denoising trajectories and RL algorithms with template 1. Results indicate that (1) Constructing denoising trajectories by DDIM inversion is not a practical way; (2) Integrating D-Fusion can enhance the effect of different RL algorithms.
DPO
0.3733
0.3407
0.3230
Ours
0.3758
0.3446
0.3276
tories to replace those provided by D-Fusion, which are then used to train the models. As shown in Figure 7, the results reveal that after applying DDIM inversion, the mod- els do not receive any noticeable improvement in alignment. This indicates that DDIM inversion fails to provide accurate noise estimations. Previous work (Mokady et al., 2022) has noted that, if denoising from x to reconstruct the image, the reconstructed one exhibited visible difference from the original one, which is a consistent phenomenon with our observation here. The reason DDIM inversion relies on a rough assumption that (x f) = < (x,, t), leading to inaccurate estimations. Therefore, compared to our method, DDIM inversion is not a practical approach construct denoising trajectories for RL training.
4.5. Generalization Ability
The models fine-tuned with our method exhibit generaliza- tion capabilities, further enhancing the potential for real- world applications. As shown in Table 1, for each prompt template, we use different models to separately sample 1,280 images with the same random seeds. The prompts used here are not optimized with RL fine-tuning, but accord with corresponding template. The results indicate that the images generated by the models fine-tuned by our method achieve higher alignment scores compared to those gener- ated by original model (SD) and DPO. Figure 4(bottom) presents the image examples generated with these prompts, qualitatively showing generalization ability of the models fine-tuned with our method. For more image examples, we refer the readers to Appendix F.
Compatibility with Different RI Algorithms. D-Fusion demonstrates compatibility with a variety of RL algorithms. In this ablation study, we further apply D-Fusion to the widely used RL-based diffusion fine-tuning methods, in- cluding DDPO (Black et al., 2024) and DPOK (Fan al., 2023). The implementation details are presented in Ap- pendix C. As shown in Figure 7, among these methods, the integration of D-Fusion enhances the alignment of diffusion models to a greater extent. More experimental results on template 2 and are shown in Appendix E. The results demonstrate that training with visually consistent samples effective across different RL algorithms.
5. Conclusion
In this work, we mitigate the issue of prompt-image mis- alignment diffusion models by employing direct pref- erence optimization with visually consistent samples. We highlight the challenge of obtaining RL-trainable visually consistent samples. To address this challenge, we introduce D-Fusion, self-attention based method that can not only generates visually consistent and well-aligned samples from given images, but also retain the denoising trajectories. We
8

---

D-Fusion: DPO for Aligning Diffusion Models with Visually Consistent Samples
conduct comprehensive experiments using Stable Diffusion backbone, incorporating variety of prompts and RL algorithms. Both qualitative and quantitative experimental results demonstrate that, by training with visually consistent samples generated by D-Fusion, the RI.-based fine-tuning can achieve better prompt-image alignment.
Chi, C., Xu, Z., Feng. S., Cousineau, E., Du, Y., Burchfiel, B., Tedrake, R., and Song, S. Diffusion policy: Visuo- motor policy learning via action diffusion, 2024, URL https: pttp:://axxiv.rr// htt.v//orrr/abb//3...e3/
Clark, K., Vicol, P., Swersky, K., and Fleet, D. J. Di- rectly fine-tuning diffusion models differentiable rewards, 2024. URL https:/ /ttp///axvvvv/va///ea//eeeeaaaea org.
Acknowledgements
This work supported in part by the National Key Research and Development Program of China (2024YF00203700), National Natural Science Foundation of China (62376243, 62441605), and "Pioneer" and "Lead- ing Goose" R&D Program of Zhejiang (2025C02037). All opinions in this paper are those of the authors and donot necessarily reflect the views of the funding agencies.
Dhariwal, and Nichol, Diffusion models beat gans on image synthesis, 2021. URL https: /ttp/i.xxiiii////../e. abs
Dosovitskiy, A., Beyer, L.., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer,
M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale, 2021. URL https: / www.ii.ooggga..o...e.ooo...o..oe abs 2010 11929.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Fan, Y., Watkins, O., Du, Y., Liu, H., Ryu, M., Boutilier, C., Abbeel, P., Ghavamzadeh, M., Lee, K., and Lee, K. Dpok: Reinforcement learning for fine-tuning text-to- image diffusion models, 2023. URL https: / //rxxva.avvvv// org/ abs / 2305 16381.
References
Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., and Cohen-Or, D. Prompt-to-promp im- age editing with cross attention control, 2022. URL https: http://axxxvv...g.e wrgp/os///////////// 01626.
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J.,
Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKin- non, C., Chen, C., Olsson, C., Olah, C., Hernandez, D., Drain, D., et al. Constitutional ai: Harmlessness from ai feedback, 2022. URL https:://// /tt////axic /aaaaaixvvvooo//aa//aaa org/ abs/ 2212. 08073.
Hessel, J., Holtzman, A., Forbes, M., Bras, R. L., and Choi, Y. Clipscore: A reference-free evaluation metric for im- age captioning, 2022. URL
Bai, Z., Wang, P., Xiao, T., He, T., Han, Z., Zhang, Z.,, and Shou, M. Z. Hallucination of multimodal large lan- guage models: A survey, 2024. URL hwtph:tt///aat //axivv. org/ hrg/abs/2ae 2404 1893C.
abs 2204. 08718.
Ho, J. and Salimans, T. Classifier-free diffusion guid- ance, 2022. URL https://axxvv.org/ 2207 12598..59.
Black, K., Janner, M., Du, Y., Kostrikov, I, and Levine, S. Training diffusion models with reinforcement learn- ing, 2024. URL https: /ttp:///aaivvvviii wrwpa///.rrg//////////// 13301.
Ho, J., Jain, A., and Abbeel, Denoising diffusion prob- abilistic models, 2020. URL abs 2ba/2006.110 11239.
Brooks, T., Holynski, A., and Efros, A. A. Instructpix2pix: Learning to follow image editing instructions, 2023. URL https:// //tpvi/aaaivv.//aaeeeaeeaaeaiiia org/ abs/ 2211 09800.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L.., and Chen, W. Lora: Low-rank adaptation of large language models, 2021. URL httts:://// //arxiv.. org/ aaa//////// / 06. 09685.
Cao, M., Wang, X., Qi, Z., Shan, Y., Qie, X., and Zheng, Y. Masactrl: Tuning-free frrng mutual self self-attention control for consistent image synthesis and editing, 2023. URL. https://arkse org/ abs/ 304 08465.
Hu, Z., Zhang, F., Chen, L., Kuang, K., Li, Gao, K., Xiao, J., Wang. X., and Zhu, W. Towards better alignment: Training diffusion models with reinforcement learning against sparse rewards, 2025. URL https://aaaaet org abs atw.///........ 2524.11240.
Cao, P., Zhou, F., Song, Q. and Yang. L. Controllable generation with text-to-image diffusion models: sur- vey, 2024. URL https: //arxii.///xv.... axxiv... org absgg//bb/4333 04279.
Huang, L., Yu. W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q. Peng, W., Feng, X., Qin, B., and Liu,

---

D-Fusion: DPO for Aligning Diffusion Models with Visually Consistent Samples
T. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions Information Systems. November 2024. ISSN 1558-2868. doi: 10. 145/3703355 URL http:/ /dxx doi. org 17755151555
Peebles, W. and Xie, Scalable difftusion models with trans- abs/ formers, 2023. URL https://arxis: orrg/bs////////
Poole, B., Jain, A., Barron, J. T., and Mildenhall, B. Dreamfusion: Text-to-3d using 2d diffusion, 2022. URL https: /atpiviivvvv/////va///aa/////aia orgg hta//bbs//2200 14988.
Illarco, G., Wortsman, M., Wightman, R., Gordon, C.,, Car- lini, N., Taori, R., Dave, A., Shankar, V., Namkoong. H., Miller, J., Hajishirzi, H., Farhadi, A., and Schmidt, L. Openclip. July 2021. URL https: //doi... orq. 10. 5281 zenodo. 5143773. you use this software, please cite it as below.
Preechakul, K., Chatthee, N., Wizadwongsa, S., and Suwa- janakorn, S. Diffusion autoencoders: Toward meaning- ful and decodable representation, 2022. URL https: / //arxiv. org/ abs /11b://///1.... 15640.
Jiang, D., Song, G., Wu, X., Zhang, R., Shen, D., Zong, Z.,, Liu, Y., and Li, H. Comat: Aligning texi-lo-image diffu- sion model with image-to-text concept matching, 2024. URL https: aaaax.iivv..... 2200/44/5// 036533
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision, 2021. URL htwps:tppp/eooc aaaaiiiii.. www.wwwwwwg....e..........oowoee www.oorgggaaa.... hbtp//a//////
00020.
Kawar, B., Zada, S., Lang, O., Tov, O., Chang, H., Dekel, T., Mosseri, I., and Irani, M. Imagic: Text-based real image editing with diffusion models, 2023. URL https:
Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, D. and Finn, C. Direct preference optimization: Your language model is secretly reward model, 2024. URL https: / /aaaaiiiivv....../ org/ abs 220/5/305.. 18290.
//arxiv. v. org/ abs/ 221 0 0 09276.
Kong, A., Ma, W., Zhao, S., Li, Y., Wu, Y., Wang, K., Liu, X., Li, Q. Qin, Y.. and Huang, F. Sdpo: Segment-level direct preference optimization for social agents, 2025.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models, 2022. URL htttts://arxiv....
URL hhttps://arxiti www.aawwww..i.aaaaaaaaaaao abs 2501.5011100 01821.
abs/2112.10755
Lee, K., Liu, H., Ryu, M., Watkins, O., Du, Y., Boutilier, C., Abbeel, P., Ghavamzadeh, M., and Gu, S. S. Aligning text-to-image models using human feedback, 2023. URL https: petps://arxxe /atxiv..iivvv.a//////.aaa/aaaav/ org/ obs/ 2302 12192.
Ronneberger, O., Fischer, P., and Brox, T. U-net: Convolu- tional networks for biomedical image segmentation, 2015, URL httpp::/// //axxiv .rgg /abs 2ttp//ab/11055055577
Liu, Y., Yao, Y.. Ton, J.-F. Zhang, X., Guo, R., Cheng, H., Klochkov, Y., Taufiq, M. F., and Li, H. Trustworthy llms: survey and guideline for evaluating large language mod- els' alignment, 2024. URL https:ttp///o httphtttt::/aaeee /ttp::/aaxvvvvvooo abs/2308 05374.
Schulman, J., Wolski, F. Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algo- rithms, 2017. URL https: http:://axxvvvooee
Shi, Y., Xue, C., Liew, Pan, Yan, H., Zhang, W., Tan, V. Y. F., and Bai, Dragdiffusion: Harnessing diffusion models for interactive point- based image editing, 2024. URL https: /arxivi.vvvv..////va//.aa//aaaiv org/ abs abs/2306.... 14435.
Lu, C., Krishna, R., Bernstein, M., and Fei-Fei, L. Visual relationship detection with language priors, 2016. URL https://arrae org/ htwp/bs//////oo....o/ggg//..a 0obs/1608 16086661100 00187.
Mokady, R., Hertz, A., Aberman, K., Pritch, Y., and Cohen- Or, D. Null-text inversion for editing real images using guided diffusion models. 2022. URL https: /arxiv. org/abs 2211.2211 09794.
Sohl-Dickstein, J.. Weiss. E. A.. Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequilibrium thermodynamics, 2015. URL https: //aaxii..rg/aas/////....... http://www....//.aa//////
03585.
Mrini, K., Lu. H., Yang, L., Huang, W., and Wang, H. Fast prompt alignment for text- tont-tto-mmaee generation, 2024. URL https: // /aaaaaiivv...o org/ abs 2412, 08639.
Song, J., Meng, C., and Ermon, S. Denoising diffusion implicit models, 2022. URL MtLL:////aa/x/..orr abs/2C100 02502.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J.. Hilton, J., Kelton, F., Miller, L., Simens, M., et al. Training language models to follow instructions with human feedback, 2022. URL https: //a rxivv org/ /203 02155.
Song, Y. and Ermon, Improved techniques for training score-based generative models, 2020. URL https: wwriv.org//bbae 2006. 09011.
Tong, Y., Yuan, J., Zhang, M., Zhu, D., Zhang, K., Wu, F,, and Kuang, Quantitatively measuring and contrastively
10

---

D-Fusion: DPO for Aligning Diffusion Models with Visually Consistent Samples
exploring heterogeneity for domain generalization. In Proceedings of the 29th ACM SIGKDD Conference Knowledge Discovery and Data Mining, KDD '23, pp. 2189 2200. ACM, August 2023. doi: 10.1145/3580005. 3599481. URL http:://th/tt htp//xxxdoo// org/10 1145/
Yang, J., Jin, D., Tang, A., Shen, L., Zhu, D., Chen, Z., Wang, D., Cui, Q. Zhang, Z.., Zhou, J., al. Mix data merge models? balancing the helpfulness, honesty, and harmlessness of large language model via model merging. arXiv preprint S.Kiv::502.0667777 2025.
3580305 3599481.
Yang, K., Tao, J., Lyu, Ge, C., Chen, J., Li, Q., Shen, W., Zhu, X., and Li, X. Using human feedback to fine-tune diffusion models without any reward model, 2024a. URL https: aaaaaaiiiiii./ii org. abs 211.111111111 13231.
Tu, H., Cui, C., Wang. Z., Zhou, Y., Zhao, B., Han, J., Zhou, W., Yao, H., and Xie, C. How many unicorns are in this image? a safety evaluation benchmark for vision llms, 2023. URL /tttpp:///rxivv.. org/ abs/ 2311. 16101.
Yang. S., Chen, T., and Zhou, M. A dense reward view on aligning text-to-image diffusion with preference, 2024b. URL https: wwwpaaxivvvvvoo/... org/ abs 24002 08265.
Tumanyan, N.. Geyer, M., Bagon, S., and Dekel, T. Plug- and-play diffusion features for text-driven image-to- image translation, 2022. Utthttps: /ttpxii.///axxv.r.t.. abs/2211.1255 12572.
Yang, Y., Zhang. K., Ge, Y., Shao, W., Xue, Z., Qiao. Y., and Luo, P. Align, adapt and inject: Sound-guided unified image generation, 2023. URL BtLp:tts:/// httt:p::/aaxii.orrtt abs / 2206 abs/2306.1155 11504.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, Attention you need, 2023. URL https:tt///// /tttpps://axii...// org/ abs/1705 03762.
Yu, T., Yao. Y., Zhang, H., He, T., Han, Y., Cui. G., Hu, J., Liu, Z., Zheng, H.-T., Sun, M., and Chua, T.-S. Rlhf-v: Towards trustworthy milms via behavior alignment from fine-grained correctional human feedback, 2024. URL w21p.///..../////.2...... 00849.
Wallace, B., Dang, M., Rafailov, R., Zhou, L., Lou, A., Pu- rushwalkam, S., Ermon, S., Xiong, C., Joty, S., and Naik, N. Diffusion model alignment using direct preference op- timization, 2023. URL https::///// /arxivv...... org/ abs/ 2311. 12908.
Zheng, G., Zhou, X., Li, X., Qi. Z., Shan, Y., and Li. X. Layoutdiffusion: Controllable diffusion model for layout- to-image generation, 2024. URL https: /arxiv. org/ abs/ab//// /2bos2330 22003/3033000 ......11888 17189.
Wang. S., Saharia, C., Montgomery, C., Pont-Tuset, J., Noy, S., Pellegrini, S., Onoe, Y., Laszlo, S., Fleet, D. J., Soricut, R., Baldridge, J., Norouzi, M., Anderson, P., and Chan, W. Imagen editor and editbench: Advancing and evaluating text-guided image inpainting, 2023. URL https:// org// org. abs 221.222.22992
Zhu, D., Song. Y., Shen, T., Zhao, Z., Yang, J., Zhang, M.. and Wu, C. Remedy: Recipe merging dynamics in large vision vision-language models. In The Thirteenth International Conference on Learning Representations, 2025.
arxiv.
Wang, Z., Tu, H., Mei, J., Zhao, B., Wang, Y., and Xie, C. Attngeg: Enhancing jailbreaking attacks on llms with attention manipulation, 2024. URL org/abs/ httpb////21..0044000 0904C.
Wu, Z., Hu, Y., Shi, W., Dziri, N., Suhr, A., Ammanabrolu, P.. Smith, N. A., Ostendorf, M., and Hajishirzi, H. Fine- grained human feedback gives better rewards for language model training, 2023. URL https:://////. http///xivvv.///////e//ea.ae...t /rg/ fos/2336..0 01 693.
Xu, J., Liu, X., Wu, Y., Tong, Y., Li, Q., Ding, M., Tang, J., and Dong, Y. Imagereward: Learning and evaluating human preferences for text-to-image generation, 2023. URL https: /aarii/.//xvvv// org/ 20/s//3344 05977.
Xu, M., Yu, L., Song, Y., Shi, C., Ermon, S., and Tang, J. Geodiff: a geometric diffusion model for molecular con- formation generation, 2022. URL https: /rrxiv. org/abs. 2203. 02923.

---

D-Fusion: DPO for Aligning Diffusion Models with Visually Consistent Samples
The Appendix is organized as follows:
Appendix A: presents the list of abbreviations and sym-
bols in this paper.
Appendix B: presents comprehensive observations on variety of attention control methods.
reference image
(a) self Q*k
(b) self K.
[c] cross Q*K
Appendix C: provides more details on implementation
(e.g., computational resources and hyperparameters).
Appendix D: provides pseudo-code of training with D-
Fusion.
base Image
(d) self Q
(c) self feats
(1) self Q,v
Appendix E: presents more experimental results.
Appendix F: presents more image samples generated by diffusion models fine-tuned with D)-Fusion.
Appendix G: presents prompts and corresponding mask thresholds used in our experiments.
reference image
(a) MosaCtri
(b) mask A A
mask feats
A. Abbreviation and Symbol Table
The abbreviations and symbols used in this paper are pre- sented in Table 2.
base image
(d) D-Fusion with different timesteps and layers
B. Observations on Attention Control
Figure 8. Effects of different attention control methods.
In this section, we present some observations on attention control from three perspectives: (I) What are the effects of different attention control methods (i.e., fusing differ- ent components in the attention module). (2) How do the timesteps and layers in U-Net affect the fusion results. (3) How do the cross-attention maps look like. We use prompt "a cat playing chess" in these observations.
applies masks to two components: self-attention maps and image features (i.e., the output of the attention module). The masks are used to make the foreground of resulting images resemble the reference images, while the background resem- bles the base images. As shown in subfigures (a) to (c) of Figure 8(bottom), we test MasaCtrl and its two parts respec- tively. They fail to generate ideal images primarily because the image features are directly tied to the pixel structure. Therefore, applying masks image features often leads to confusion at the boundaries between the covered and uncov- ered areas. We can conclude that MasaCtrl is more suitable for fusing two images with similar foreground and back- ground boundaries, such as the images generated with same seed (but different prompts). Therefore, we turn to apply masks to keys and values D-Fusion, and get robust fusion effects, shown in subfigure (d) of Figure 8(bottom).
(1) What are the effects of different attention control meth- ods? ? We have observed varieties of different attention con- trol methods. As shown in Figure 8(top), subfigures (a) to (c) correspond to the previous work (Tumanyan et al., 2022; Cao et al., 2023; Hertz et al., 2022), where they inject the self-attention maps, the keys and values of self-attention, and the cross-attention maps from the reference image into the base image, respectively. Subfigure (a) illustrates that injecting self-attention maps can easily lead to significant blurring in the resulting image. Subfigures (b) and (c) inject the keys and values of self-attention, and the cross-attention maps, respectively. In the former, the features largely mimic those of the reference image, while in the latter, the features from the base image are better preserved. For instance, the table in subfigure (c) appears green, just like in base image, whereas subfigure (b) does not exhibit this characteristic. Meanwhile, we experiment with injecting additional compo- nents in attention mechanism, as shown in subfigures (d) to (f). The images in Figure 8(top) are either blurred or retain too few features from the base image.
(2) How do the timesteps and layers in U-Net affect the fu- sion results? The U-Net in Stable Diffusion has three down- sampling layers, one middle layer and three up-sampling layers. We number them in order from 0 to 7. As shown in Figure 9, we apply D-Fusion at different timesteps and dif- ferent U-Net layers. The x-axis represents fused timesteps, and the y-axis represents fused layers. We present some of the most representative layers, specifically layer 3 (i.e., middle layer), layers 2-4 (i.e., middle layer, the last down- sampling layer and the first up-sampling layer), layers 3-6 (i.e., the whole upssampling layers) and layers 3-5 (i.e., the whole up-sampling layers except for the last one).
In order to generate ideal images, we introduce masks as in the previous work MasaCtrl (Cao et al., 2023). MasaCtrl
12

---

D-Fusion: DPO for Aligning Diffusion Models with Visually Consistent Samples
Table 2. List of important abbreviations and symbols.
Abbreviation/Symmoo
Meaning
Abbreviations of Concepts
DM
Diffusion Model
RL
Reinforcement Learning
DPO
Direct Preference Optimization
SD
Stable Diffusion
LoRA
Low-Rank Adaptation
DDIM
Denoising Diffusion Implicit Model
DDPM
Denoising Diffusion Probabilistic Model
CLIP
Contrastive Language- Pre- Teeiirainnng
Abbreviations of Methods
D-Fusion
Self-attention based Denoising trajectory Fusion
DDPO
Denoising Diffusion Policy Optimization
DPOK
Diffusion Policy Optimization with KL regularization
Symbols in Diffusion Models
Xo
Generated image
Noisy image at timestep
{xr. BEr.XT-i .1- X0)
Denoising trajectory
Condition for image generation, also called prompt Parameters of the diffusion model
0
WO
Gaussian distribution
T
Total timesteps
Symbols in Reinforcement Learning
St
State timestep
Action timestep &
me
Action selection policy parameterized by 0
R
Reward function
f
Rewards after normalization
Symbols in D-Fusion
I
Set of real numbers
IB
Binary set {0,1}
1 19,1%,1 I"
Target image base image and reference image
Q.K.V
Query, key and value in attention mechanism
Apross Agels Oc
Cross-attention maps and self-attention maps at timestep Index list of the item-related tokens in prompt
M.
Cross-attention mask at timestep Hadamard product
o
+
Exclusive OR operation
13

---

D-Fusion: DPO for Aligning Diffusion Models with Visually Consistent Samples
20
==
16
c=
t =
t=1
reference
base
cat
2-4
Layer
playing
Layer
chess
Layer
cat
3-6
2:20-0
1:18-
t:14-0
t:10-0
1:6-0
chess
Figure 9. Impact of timesteps and U-Net layers on fusion results.
(3) How do the cross-attention maps look like? As shown in Figure 10, the cross-attention maps each timestep are highly correlated with the corresponding items in the gener- ated images. More specifically, in the early timesteps, the layouts of the images have not yet fully taken shape, so the cross-attention maps do not accurately identify the items' location. In the middle timesteps, the cross-attention maps gradually turn mark the items' position precisely. By the later timesteps, the fundamental features of each item have been established, thus the cross-attention maps shift focus to more fine-grained details (e.g., cat's face). For tokens that do not represent items, their cross-attention maps do not have an obvious meaning, but are generally highlighted in the areas of related items. the image fails to align with the prompt, such as when there is no cat present, the corresponding cross- attssnatttntion maps will also lack clearly highlighted areas.
cal
Figure I0. The heatmaps of cross-attention maps at different timesteps.
Further Alignment Verification. Although D-Fusion has demonstrated the ability to generate both well-aligned and visually consistent samples, it also generates failed cases sometimes. Excessive use of failed cases training data can have negative impact on fine-tuning the diffusion models. Therefore, we introduce additional verification before training, which is shown as follows:
R(x,, c) R(x c) 2 thr (R(x): c) R(x c)),
(7)
where thrado is a hyperparameter called adoption threshold (usually, 0.0 < thr I 1.0), and is reward function. the target image I does not meet the requirements, we will replace it with the reference image I". This replacement is reasonable, the pairing between reference image I" and base image Id is consistent with that used in the naive DPO.
C. Implementation Details
C.1. Detailed Implementation of Our Method
Fusion Layers and Timesteps. As shown in Appendix B, usually, employing self-attention fusion at all layers and all timesteps does not generate ideal images (i.e., images that are both well-aligned and visually consistent with given poorly-aligned images). Therefore, we only employ self- attention fusion at some layers and some timesteps. For layers, we employ self-attention fusion at the middle layer and the up-sampling layers (iee, from layer 3 to layer 6 in Stable Diffusion basee. For timesteps, we employ self-atten aelf-attention fusion from timestep 18
Compatibility with DDPO. Before optimization, the align- ment scores evaluated by CLIP need to be normalized first. In implementation, we calculate the mean and standard devi- ation of the alignment scores for current and all the previous rounds. The scores from previous rounds are also used in calculation in order to increase the sample size under the
We use 1 instead of I to represent the target image, f commonly refers timestep in diffusion models.
14

---

D-Fusion: DPO for Aligning Diffusion Models with Visually Consistent Samples
C.2. Experimental Resources
Table 3. Hyperparameters of experiments.
The experiments were conducted 24GB NVIDIA 3090 and 4090 GPUs. It took approximately 30 hours to reach training data volume of 25.6k when applying DPO and DDPO, and approximately 40 hours when applying DPOK.
Hyperpatameter
D-Fusion
Baselines
Denoising steps T
20
20
Noise weight ?
1.0
1.0
Sampling
Guidance scale
5.0
5.0
Batch size
4
4
C.3. Hyperparameters
Batch count
160
160
The hyperparameters of our experiments are listed in Table 3. Hyperparameters that are not listed keep consistent with the corresponding RL work (Wallace et al., 2023; Fan et al., 2023; Black et al., 2024).
Fusion U-Net layers
3-6
Fusion
Fusion timesteps
Adoption threshold
1.0
Optimizer
AdamW
AdamW
Learning rate Weight decay (3))
lc-4
le-4
Optimizer
le-4
le-4
D. Pseudo-Code
(0.9, 0.999)
(0.9, 0.999)
1c-8
1e-8
The pseudo-code of employing direct preference optimiza-
Grad. clip norm
1.0
tion with D-Fusion for one training round is shown in Algo- rithm 1.
Batch size
1
Training
Grad. accum, steps
320
320
Inner epoch
2
2
same prompt. With the mean and standard deviation of the image scores under the same prompt, where R is the reward function. The normalized scores serve as rewards in the pro- cess of DDPO fine-tuning. DDPO employs proximal policy optimization (PPO) algorithms (Schulman et al., 2017) via Po(xx-1/xx.. .)x,,)) and clipping. The gra- importance sampling Pe.r (x (x--1 xx,e]
normalize
them by R(xa,e) stx((((xxee))
dient when applying D-Fusion to DDPO goes follows.
Po(x
x². c)
E
Ve log xx c)pe
Pamn
(x
xf. c)
(x Po
x/i c)
Vo log (xf_1
x c)pb
pea (xt_)
xti
Compatibility with DPOK. Similar DDPO, DPOK also employs the clipping mechanism in PPO. Meanwhile, DPOK utilizes value function V(x, c) in their implemen- tation. In our implementation, we replace the value function with reward normalization same as DDPO. Therefore, the gradient when applying D-Fusion to DPOK goes as follows, where a 0.99 and 0.01.
Va log.oo x cm
x² c) Pawe (xL_ xxf())
Vo
x pbb
+ (Dx
x |a (x] x c))
15

---

D-Fusion: DPO for Aligning Diffusion Models with Visually Consistent Samples
Algorithm 1: Pseudo-code of employing direct preference optimization with D-Fusion for training round.
Input :Tottal denoising timesteps T, inner epoch E, number of samples each round N, prompt list C, reward function R. pre-trained diffusion model pg.
Pold = deepcopy(po):
// Sampling
D Daampting {c: [] for in C}: for n - to N do
Randomly choose a prompt e from C:
Randomly choose zT from from.N(O, (O, Set seed to random number s:
F(T-1):0 Denoise from ør with pe for T steps;
R(c, 2o)
D Dammpting end
// Constructing Visually Consistent Samples
D Deraining =]];
for c, T,
D Dsampling do
Diemp = sort (TT::, r, 8 1:K in descending order according to r Dreference = Dromp [0 K//2]:
Doase Diempl K/ 2 K];
for {xf.0: ,")"" (= sh} E zip( Dii((reference. Daase) do
Set seed to s
A7.1 Across KT V7.1
Denoise from o with pe for T steps;
Extract mask MT from Agross using (4)
Set seed to s²
29 = x
for - T to do x7 Denoise from zi with pe, incorporating Mc, KT. V7 using Eq.(5):
end
Draining append((a))))pppa 27:00 c));
end
end
// Training for e +- 1 to E do
D shuffle(Dtraininn))
with grad; for d E D do
d shuffflle
for (2t 22 x 1 do
update 0 with gradient descent using Eq. (6)
end
end end
16

---

D-Fusion: DPO for Aligning Diffusion Models with Visually Consistent Samples
E. More Experimental Results
ing to each item used in (q.(4). For each prompt template, collect 40 prompts for training and another 40 prompts for generalization test. The full prompt lists are shown in Table 4, Table 5 and Table 6.
0331
The mask thresholds are not hyperparameters that require meticulous tuning, and can be predetermined through low- cost methods. In our experiments, we predeternined the thresholds by sampling few images for each prompt, and assessing whether the chosen threshold value allows the mask to outline corresponding object. This selection pro- cess does not require high precision. As shown in the prompt lists, the thresholds we use (predominantly 0.005, 0.01, 0.015, 0.02, and 0.03) are fairly coarse values. To apply our method to new prompts, one can simply sample few images and determine appropriate thresholds through straightforward observation.
034
0.24
0356
Traeng Dea
Touerng Duts
100M Inversion
(>-huon DPO
DDIM Invenioon
DPO-i--Psiion - DPO
(a) DPO and D-Fusion on Tomplates 2(oft) and Strigai0
3.20
3.24
334
3200
0339
3219
Traving Dala
Trrkking Tamk
Plasion
D-Eusiin DOPO
(b) DDPO and D Pasion Tempplates 20ee end 30ighh)
3228
0333
0.42
3204
030
DDMM lnvession
DPOK-
Stsion DPOK
DDIM Saversion sion
DPOK Fusion
DPOK
(6) DPOK and D-Pasion Templates 20leth and 30ight)
Figure 11. More ablation studies on denoising trajectories and RL algorithms with templates 2 and 3.
As supplement to Section 4.4, we also conduct ablation studies on denoising trajectories and RL algorithms on tem- plates 2 and as illustrated in Figure 11. The experimental results show the same conclusion as the ablation studies template 1. That is, on the one hand, constructing denoising trajectories by DDIM inversion is not an effective approach for RL training. On the other hand, integrating D-Fusion can enhance the effect different RL algorithms, which can further improve the alignment of diffusion models.
F. More Samples
In this appendix, we present more samples generated by the diffusion models fine-tuned with visually consistent samples. In detail, Figure 12 shows more samples our method when compatible with DPO, DDPO and DPOK on template Correspondingly, Figure 13 and Figure 14 show more samples on templates 2 and 3. Moreover, Figure 15 shows more samples when generalized unseen prompts.
G. Prompt Lists
We present the prompt lists used in our experiments in this section. Meanwhile, we list the mask thresholds correspond-
17

---

D-Fusion: DPO for Aligning Diffusion Models with Visually Consistent Samples
DPO
DPO+D-Fusion
DDPO
DDPO-D-Fusion
DPOK
D-OKoD-Fusion
"a raccoon driving car"
"a horse playing basketball"
"a sheep eating sandwich"
"a dog driving car"
"a zebru playing basketball"
"a bear eating sandwick" "
"a lion driving a car"
"a cow playing basketball"
Figure More samples generated by the diffusion models with template The models fine-tuned by different RL methods with without D-Fuiion.
18

---

D-Fusion: DPO for Aligning Diffusion Models with Visually Consistent Samples
SD
DPO
DPO-D-Fusion
DDPO
DDPO+D-Fusion
DPOK
D-Okin-Fusion
"a yellow and metallic dog"
"a black and glass dog"
"a blue and fabric dog"
"a red and woolen dog"
yellow and woolen table"
"a green and metallic table"
"a yellow and marble lamp"
WWittee & Mamicc
white and marble bus"
Figure 13. More samples generated by the diffusion models with template 2. The models fine-tuned by different RL methods with without D-Fusion.
19

---

D-Fusion: DPO for Aligning Diffusion Models with Visually Consistent Samples
SD
DPO
DPO-D-Fusion
DDPO
DDPO-D-Fusion
DPOK
DPOK-D-Fusion
"the table above the dog"
"the dog on the right of the vase"
"the umbrella above the table"
"the suitcase on the left of the person"
"the building below the kite"
"the person on the left of the bear"
"the trees above the train" "
"the person on the right of the dog"
Figure 14, More samples generated by the diffusion models with template 3. The models are fine-tuned by different RL methods with without D-Fusion.
20

---

D-Fusion: DPO for Aligning Diffusion Models with Visually Consistent Samples
DPO
DPO-D-Fusion
DPO
DPO+D-Fusion
SD
DPO
DPO-D-Fusion
"a horse driving car"
"a yellow and marble tree"
"the below the clock
bear playing basketball"
"a green and metallic bow!"
"the motorcycle on the left of the car"
"a rat eating sandwich"
"a red and leather car"
"the shirt above the shoes" "
"a zebra driving car"
"a yellow and marble bike"
"the faucet right bottle" "
"a wolf playing basketball"
"a red and wooden table"
"the motorcycle on the left on the bus"
"a riger driving car"
"a blue and marble car"
"the kite above the building"
"a raccoon washing dishes"
"a white and leather cup"
"the helow the jacket"
tiger riding bike"
"a green and wooden chair" (b) Template
"the person the right of the bear"
(a) Template 1
(c) Template
Figure More samples when generalized unseen prompts on templates 1, and
21

---

D-Fusion: DPO for Aligning Diffusion Models with Visually Consistent Samples
Table 4. Prompt lists and mask thresholds for template 1.
Training list
Test list
Prompt
Mask Threshold
Prompt
a cat eating a sandwich
0.03; 0.005
a cat playing basketball a dog eating sandwich horse driving a car a monkey playing basketball a rabbit cating a sandwich zebra driving a car sheep playing basketball
a cat driving car a dog driving a car dog playing basketball horse playing basketball
0.03; 0.01
0.03; 0.01
0.03; 0.02
0.03; 0.02
horse eating a sandwich a monkey eating a sandwich
0.03; 0.005
0.03; 0.005
monkey driving car rabbit driving car a rabbit playing basketball
0.03; 0.01
deer eating a sandwich a cow driving car goat playing basketball a lion eating sandwich
0.03; 0.01
0.03; 0.02
zebra playing basketball
0.03; 0.02
a zebra eating a sandwich a sheep eating a sandwich
0.03; 0.005
a tiger driving a car a bear playing basketball a raccoon eating a sandwich
0.03; 0.005
a sheep driving a car a deer driving a car a deer playing basketball a cow playing baskethall cow eating sandwich goat eating sandwich
0.03; 0.01
0.03; 0.01
a fox driving a car a wolf playing basketball a lizard eating a sandwich a shark driving a car whale playing basketball dolphin eating a sandwich a squirrel driving a car a mouse playing basketball rat eating sandwich turtle driving car frog playing basketball a chicken eating a sandwich
0.03; 0.02
0.03; 0.02
0.03; 0.005
0.03; 0.005
goat driving car a lion driving a car lion playing basketball a tiger playing basketball a tiger eating a sandwich bear cating sandwich a bear driving car raccoon driving a car a raccoon playing baskethall a fox playing basketball a fox eating a sandwich
0.03; 0.01
0.03; 0.01
0.03; 0.02
0,03; 0.02
0.03; 0.005
0.03; 0.005
0.03; 0.01
0.03; 0.01
a duck driving a car a goose playing basketball a pig cating a sandwich a Ilama driving a car lion washing dishes tiger riding a bike bear playing chess a raccoon washing dishes
0.03; 0.02
0.03; 0.02
0.03; 0.005
wolf eating sandwich
0.03; 0.005
wolf driving car
0.03; 0.01
a lizard driving car a lizard playing basketball
0.03; 0.01
0.03; 0.02
a shark playing baskethall a shark eating a sandwich a whale eating a sandwich a whale driving car dolphin driving car
0.03; 0.02
a fox riding a bike a wolf playing chess a lizard washing dishes a shark riding a bike whale playing chess dolphin washing dishes
0.03; 0.005
0.03; 0.005
0.03; 0.01
0.03; 0.01
a dolphin playing basketball
0.03; 0.02
22

---

D-Fusion: DPO for Aligning Diffusion Models with Visually Consistent Samples
Table 5. Prompt lists and mask thresholds for template 2.
Training list
Test list
Prompt
Mask Threshold
Prompt
black and woolen bus black and plastic dog green and metallic table a black and glass table a white and marble bus a red and woolen dog a green and glass lamp a red and glass dog a black and glass bowl a green and stone lamp a yellow and wooden tree a black and marble vase yellow and metallic dog a yellow and woolen table a blue and woolen vase yellow and plastic bus a black and stone sculpture a red and marble table white and plastic lamp a yellow and leather table a red and leather toy a red and leather table a blue and plastic sword a black and fabric tree a yellow and fabric dog a black and plastic table a white and metallic sculpture
0.005
red and wooden table a green and metallic bowl white and woolen bowl a blue and stone sculpture a yellow and glass bus a black and woolen sculpture a yellow and fabric table a yellow and marble tree a blue and wooden toy a red and plastic sculpture a white and leather tree a yellow and metallic sculpture a white and wooden lamp a yellow and stone bus a red and fabric toy green and wooden dog white and woolen table a black and stone table white and metallic tree a green and plastic sculpture a white and marble car a white and leather boat a blue and fabric clock a white and stone chair green and leather chair a yellow and wooden cup white and leather cup a green and wooden chair a black and wooden car red and wooden plate red and glass car a yellow and stone car white and metallic cup a white and stone car blue and marble car a red and woolen bike a yellow and marble bike a blue and wooden chair a red and marble plate a red and leather car
0.03
0.02
0.02
0.03
0.01
0.03
0.015
0.03
0.03
0.02
0.03
0.015
0.005
0.005
0.03
0.025
0.02
0.03
0.02
0.01
0.02
0.03
0.02
0.015
0.02
0.02
a black and leather tree a blue and marble toy black and glass dog a black and fabric bus a green and wooden vase blue and plastic table a green and fabric dog black and stone bowl a black and stone tree a black and glass sculpture a yellow and marble lamp a blue and fabric dog a green and glass sword
0.02
0.015
0.015
0.03
0.03
0.02
0.015
0.02
0.02
0.015
0.03
0.015
0.03
23

---

D-Fusion: DPO for Aligning Diffusion Models with Visually Consistent Samples
Table 6. Prompt lists and mask thresholds for template 3.
Training list
Test list
Prompt
Mask Threshold
Prompt
the umbrella above the table the trees above the train the laptop above the table the table below the laptop the building below the tower the snowboard below the person the dog on the right of the vase the table above the dog the shirt above the pants the suitcase above the dog the suitcase on the left of the person
0.03; 0.01
the shirt above the shoes the jacket above the pants the kite above the building the kite above the sand the monitor above the keyboard the keyboard above the mouse the glasses below the laptop the shirt above the jeans the jeans above the shoes the bag below the sink the hat above the sunglasses the tree below the clock the clock above the tree the skis below the pants the pants below the jacket the sunglasses below the hat the car on the right of the umbrella the phone on the right of the monitor
0.01; 0.03
0.02; 0.01
0.01; 0.02
0.005; 0.01
0.03; 0.005
0.01; 0.03
0.01; 0.01
0.01; 0.01
0.03; 0.01
0.03; 0.005
the dog below the suitcase the dog on the left of the person the person on the right of the dog the person on the right of the suitcase the person on the right of the hand the helmet above the glasses the helmet above the person the roof above the bus the wheel below the engine the engine above the wheel the car on the right of the person the table below the glasses the building below the kite the sand below the kite the mouse below the keyboard the computer below the counter the person on the left of the ball the ball on the right of the person the person on the left of the pillow the bowl on the right of the plate the building on the right of the truck the person the left of the bottle the bottle on the right of the person the box the left of the post the truck on the right of the car the jacket on the left of the coat the monitor the left of the person the phone on the left of the person the person on the left of the bear
0.01; 0.03
0.02; 0.005
0.01; 0.02
0.01; 0.03
0.01; 0.01
0.03; 0.01
0.03; 0.01
0.01; 0.01
the shirt below the helmet the pants below the shirt the person on the right of the bear the bear on the right of the person the bear on the left of the person the bus on the right of the car the bus on the right of the motorcycle the car on the left of the bus the motorcycle on the left of the bus the motorcycle on the left of the car the table below the monitor the person below the monitor the basket on the right of the person the faucet the right of the bottle the pot on the right of the faucet the van on the right of the car the car the left of the van the person on the left of the train the hat on the left of the shirt the plate the left of the glasses the person on the left of the cart the bed on the left of the chair
0.01; 0.005
0.005; 0.01
0.01; 0.01
0.01; 0.01
0.01; 0.03
0.03; 0.03
0.03; 0.03
0.01; 0.005
0.01; 0.01
0.01; 0.01
0.01: 0.02
0.01; 0.01
0.01; 0.01
0.01; 0.02
0.02; 0.01
0.01; 0.01
0.01; 0.01
0.01: 0.01
0.01; 0.01
0.01; 0.01
0.01; 0.03
24


Reviews:
[{"review_id": "mYii8NRp6w", "reviewer": "Reviewer_UTBY", "summary": "This paper focuses on improving DPO for fine-tuning diffusion models. The key problem identified is visual inconsistency in training samples, which hampers the effectiveness of reinforcement learning-based fine-tuning methods like DPO.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper and providing a lot of suggestions! We appreciate your recognition of our paper's clarity and your time reviewing the paper and supplementary materials in detail.\n\nResponses to your suggestions are as follows. We provide additional experimental results **at this link** (https://anonymous.4open.science/r/ICML-25-Rebuttal-2703/rebuttal_2703.pdf). Please let us know if you require any further information, or if anything is unclear. \n\n## 1. More Benchmarks\n\n> Several meaningful benchmarks are not evaluated, such as Pick-a-Pic, HPSv2, Image Reward, and Aesthetic score.\n\nThank you for recommending these meaningful evaluation metrics.\n\nWe would like to first clarify that our work focuses on addressing the issue of **prompt-image misalignment**, which is a crucial challenge in controllable generation for diffusion models. The metrics you recommended are designed to assess **human preference** or **aesthetic quality**, which are not the primary tasks our work directly targets.\n\nNevertheless, we have conducted additional experiments using these metrics you recommended. The results are presented in Figure 2 at the provided link. As shown, when using these metrics as reward models, D-Fusion can still enhance the performance of the RL fine-tuning. This further demonstrates the effectiveness of D-Fusion. \n\n## 2. Ablation Study on Mask Threshold\n\n> There is no ablation study for the mask threshold hyperparameter. \n> How sensitive is D-Fusion to the mask threshold hyperparameter?\n\nThank you for raising this important question.  \n\nThe mask threshold is **not a hyperparameter that requires meticulous tuning**, and can be predetermined through low-cost methods. In our experiments, we predetermined the threshold by sampling a few images for each prompt, and assessing whether the chosen threshold value allows the mask to outline corresponding object. This selection process does not require high precision. As shown in Appendix G, the thresholds we use (predominantly 0.005, 0.01, 0.015, 0.02, and 0.03) are fairly coarse values. If users wish to apply our method to new prompts, they simply need to sample a few images and determine an appropriate threshold through staightforward observation.  \n\nEmpirically, our method can consistently deliver **stable improvements** even with less precise thresholds. We performed an ablation study on the mask threshold by setting it to 0.002, 0.005, 0.01, 0.02, 0.03 and 0.05, respectively. Each threshold is uniformly applied to **all** prompts in the list. And then we fine-tune the model. The results are shown in Figure 3 at the provided link. As the threshold increases, the fine-tuning performance initially improves and then declines. For thresholds of 0.002 and 0.05, the masks are too inaccurate, leading to a significant drop in performance. In contrast, for threshold values from 0.005 to 0.03, our method can **stably** improve the fine-tuning effect. The experimental results demonstrate the robustness of our method to threshold setting. \n\n## 3. More Baselines\n\n> * Lack of comparison with DPO-like methods. \n>   * Diffusion-NPO. \n>   * Diffusion-RPO. \n>   * SePPO. \n> * Lack of comparison with SFT. \n> * Lack of comparison with RL methods, such as PPO and PG. \n\nThank you for pointing out these valuable methods for comparison.  \n\n* For the comparison with **SFT**, we have added this experiment.  \n* For the comparison with **RL methods**, DDPO is a PPO-based diffusion model fine-tuning method [1], and we have already presented its results in the paper. Additionally, we conducted a comparison with PG. \n* For the comparison with **DPO-like methods**, we have carefully read the papers you recommended. However, we regret to find that none of them have released their implementation code (despite providing repository links), making it difficult for us to verify their details. Nevertheless, we would like to discuss them in the related work section of the final version. Additionally, We conducted a review of existing works, and added a new comparison with DenseReward [2].  \n\nThe experimental results are shown in Figure 1 at the provided link. D-Fusion can be well **compatible** with SFT, PG and DenseReward. As shown, incorporating D-Fusion into these methods consistently leads to better fine-tuning performance than the original methods.\n\n[1] Training Diffusion Models with Reinforcement Learning. \n\n[2] A Dense Reward View on Aligning Text-to-Image Diffusion with Preference.", "claims_and_evidence": "The claim that \"D-Fusion works across different RL algorithms\" is supported primarily through experiments on DPO, DDPO, and DPOK. While these are relevant methods, additional RL-based techniques could be explored to further validate generalizability.", "methods_and_evaluation_criteria": "Several meaningful benchmarks are not evaluated, such as Pick-a-Pic, HPSv2, Image Reward, and Aesthetic score.", "theoretical_claims": "There is no formal proof for D-Fusion, e.g., the optimality of D-Fusion.", "experimental_designs_or_analyses": "**Comparison.** \n\n1. Lack of comparison with DPO-like methods. See *Essential References Not Discussed*.\n\n2. Lack of comparison with SFT.\n\n3. Lack of comparison with RL methods, such PPO and PG. But this is not necessary.\n\n**Ablation Study.** There is no ablation study for the mask threshold hyperparameter.", "relation_to_broader_scientific_literature": "Lack of discussion on other Diffusion DPO variants.", "essential_references_not_discussed": "The paper does not compare D-Fusion to other Diffusion-PO methods. To name a few:\n\n[1] Diffusion-NPO: Negative Preference Optimization for Better Preference Aligned Generation of Diffusion Models. ICLR 2025.\n\n[3] Diffusion-RPO: Aligning diffusion models through relative preference optimization. arXiv:2406.06382\n\n[4] SePPO: Semi-Policy Preference Optimization for Diffusion Alignment. arXiv:2410.05255", "other_strengths_and_weaknesses": "The notations are clear and well written.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "How sensitive is D-Fusion to the mask threshold hyperparameter?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NEecvH0nqB", "reviewer": "Reviewer_tsGd", "summary": "This paper introduces D-Fusion, a novel approach to addressing the misalignment between generated images and their corresponding text prompts. D-Fusion constructs DPO-trainable visually consistent samples to mitigate the visual inconsistency present in previous DPO methods. First, the mask-guided self-attention fusion ensures that the generated images are not only well-aligned with the text prompts but also visually consistent with given poorly aligned images. Second, D-Fusion preserves the denoising trajectories of the resulting images, facilitating effective DPO training. Experimental results demonstrate significant improvements in prompt-image alignment.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time and effort in reviewing our paper and providing well-structured comments. We are delighted that you think our paper well-written and approve of our motivation and experiments. \n\nBelow are our responses to your concerns. Additional experimental results can be found **at this link** (https://anonymous.4open.science/r/ICML-25-Rebuttal-2703/rebuttal_2703.pdf). If anything remains unclear or if you need further information, please feel free to let us know! \n\n## 1. Applicability to Other Architecture\n\n> (Weakness 1) In D-Fusion, cross-attention mask extraction relies on findings from SD’s cross-attention maps, which limits its applicability to other diffusion architectures, such as DiT.\n\nMany thanks for this concern. \n\nAs you pointed out, many diffusion models no longer adopt the U-Net architecture. We take SD3.5, a representative example that uses DiT, as a case study. In SD3.5, there is no explicit cross-attention layer. However, in the self-attention layers, the prompt embeddings are concatenated with image features and processed together, effectively forming an **implicit cross-attention mechanism**. This allows us to extract cross-attention masks and further apply D-Fusion. Examples of this can be found in Figure 7 at the provided link.\n\nMore broadly, any model that **incorporates prompt guidance through attention mechanism** could potentially benefit from D-Fusion. However, since many of these models have only gained research attention in recent months, a more in-depth exploration will be pursued in our future work. \n\n## 2. Complex Prompts\n\n> (Weakness 2) The prompts used in this paper are simple, making it unclear whether D-Fusion can perform well with more complex prompts.\n\nThank you for this question! \n\nWe have observed that previous studies primarily fine-tuned models on short prompts [1,2], with some even fine-tuning on simple labels [3]. To ensure a fair comparison, we followed them to use simple prompts in our experiment.  \n\nMoreover, we carefully designed **three types of prompt templates**, which considers the behavior of the object, the attribute of the object and positional relationship between objects respectively. We suggest that these prompts are representative, as most complex prompts can be decomposed into these three fundamental types.\n\nDespite this, we appreciate your reasonable suggestion and have conducted **additional experiments using complex prompts**. As shown in Figure 4(left) at the provided link, D-Fusion can also construct well-aligned visually consistent samples for complex prompts. Furthermore, as demonstrated in Figure 4(right), fine-tuning with visually consistent samples also leads to improved performance under complex prompt conditions. \n\n[1] Training Diffusion Models with Reinforcement Learning. \n\n[2] DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models. \n\n[3] Aligning Text-to-Image Models using Human Feedback. \n\n## 3. Clarification on Efficiency\n\n> (Weakness 3) The efficiency of D-Fusion has not been evaluated.\n\nThank you for this suggestion. \n\nWe are willing to first analyze the computational overhead of D-Fusion and naive DPO theoretically. As shown in Figure 2 of the paper, each fine-tuning round of naive DPO consists of three steps: **sampling**, **evaluation**, and **training**. Among them, **evaluation** is conducted using AI models, which are highly efficient and can run concurrently with **sampling**. Thus, the total time required for one round in naive DPO can be expressed as $nT_1+nT_2$, where $n$ is the number of images per round, $T_1$ is the time needed to sample each image, and $T_2$ is the time required for training on each image.  \n\nD-Fusion introduces an additional **fusion** step following the **evaluation** step. During **fusion** step, half of the images (*i.e.*, those that are well-aligned) are resampled to extract their $K,V$ and mask values, while the remaining half (*i.e.*, those that are poorly aligned) undergo sampling with attention fusion. Consequently, the time required per round in DPO+D-Fusion can be expressed as $nT_1+\\frac{n}{2}(T_1+T_1')+nT_2$, where $T_1'$ represents the time needed to apply attention fusion. (Alternatively, storing $K,V$ and mask values during the **sampling** step could eliminate the need for repeated resampling, reducing the total time to $nT_1+\\frac{n}{2}T_1'+nT_2$. But we do not recommend this approach, as storing $K,V$ and mask values for all the $n$ images would require excessive memory.) \n\nThe computational overhead introduced by D-Fusion is **affordable**. In our experiments, the ratio of processing times is observed as $T_1:T_1':T_2=6:7:33$. This indicates that the majority of the time is spent on training, thus the computational cost of D-Fusion is acceptable. Specifically, when fine-tuning on an RTX 3090 GPU, naive DPO takes 78 minutes per round, whereas D-Fusion extends this to 91 minutes per round.", "claims_and_evidence": "Most are supported, but I believe there is room for improvement.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "No issue.", "relation_to_broader_scientific_literature": "D-Fusion is related to the diffusion models and DPO. The cross-attention mask extraction of D-Fusion is inspired by [1] and [2].\n\n[1] Hertz, et al. Prompt-to-prompt image editing with cross attention control.\n\n[2] Cao, et al. Masactrl: Tuning-free mutual self-attention control for consistent image synthesis and editing.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "### **Strengths**\n\n- The paper is well-written and easy to follow.\n- The motivation is clear, and the proposed method is solid.\n- The experiments are comprehensive and effectively demonstrate the effectiveness of D-Fusion.\n\n### **Weaknesses**\n\n- In D-Fusion, cross-attention mask extraction relies on findings from SD’s cross-attention maps, which limits its applicability to other diffusion architectures, such as DiT.\n- The prompts used in this paper are simple, making it unclear whether D-Fusion can perform well with more complex prompts.\n- The efficiency of D-Fusion has not been evaluated.", "other_comments_or_suggestions": "No.", "questions_for_authors": "See “Weaknesses”.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qC2JdTFvH2", "reviewer": "Reviewer_n8VD", "summary": "This paper introduces **D-Fusion**, a method for improving text-image alignment in diffusion models via **Direct Preference Optimization (DPO)**. It identifies a key challenge in prior DPO approaches—**visual inconsistency** between well-aligned and poorly-aligned images—which hinders the model's ability to learn effective alignment strategies. To address this, D-Fusion **constructs visually consistent training samples** using **self-attention fusion** while preserving denoising trajectories, ensuring compatibility with RL-based training. The paper demonstrates **significant improvements in prompt-image alignment** over naive DPO on Stable Diffusion (SD) 2.1 across various evaluation metrics, including CLIPScore and human preference tests. The method is also shown to be **compatible with multiple RL algorithms**, including DPO, DDPO, and DPOK.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for providing these detailed and helpful comments! We appreciate that you think our method well-motivated and approve of our experiments.\n\nResponses to your concerns are as follows. We provide additional experimental results **at this link** (https://anonymous.4open.science/r/ICML-25-Rebuttal-2703/rebuttal_2703.pdf).\n\n## 1. About Base Model\n\nThanks for recommending advanced models like SDXL. In fact, we have tried to use them. Unfortunately, due to limited computational resources (24GB 3090/4090 GPU), we are unable to fine-tune them and can only perform inference.\n\nIn Figure 6 and 7 of the above link, we show some images generated using D-Fusion on **SDXL** and **SD3.5**. As shown, D-Fusion is capable of producing well-aligned and visually consistent samples with these models. We suggest that fine-tuning with these samples still has the potential to enhance alignment of these models.\n\nWe chose SD2.1 as it is the most advanced model our resources allow. Notably, much influential related work conducted experiments on SD1.4 and SD1.5 [1,2,3]. We will appreciate your understanding regarding our limitation of base model.\n\n## 2. More Baselines\n\n> No comparison to alternative approaches (e.g., LoRA for alignment, reward-based RLHF)\n\nThanks for your suggestion. We would like to clarify that DDPO and DPOK are forms of reward-based RLHF. As follows, we provide a categorized list of the methods compared in our experiments, with **bold** ones indicating those newly added during rebuttal.\n\n* SFT: **Lora Fine-tuning**\n* Reward-based RLHF: DDPO, DPOK, **Policy Gradient**\n* DPO: DPO, **DenseReward**[4]\n\nResults of added experiments are shown in Figure 1 of the link. D-Fusion are well compatible with these methods. As shown, incorporating D-Fusion leads to greater alignment improvement.\n\nIf there exist methods you believe should be included, please don't hesitate to tell us!\n\n## 3. Real-World Task\n\n> Real-world applications (e.g., inpainting, video synthesis).\n\nThanks for your suggestion. To the best of our knowledge, inpainting and video synthesis are rarely explored in previous RL-based studies. Given the time constraints of rebuttal period, we were unable to implement them well. However, RL-based fine-tuning follows an **end-to-end** training paradigm, where simply modifying reward function allows adaptation to various downstream tasks. We believe that RL-based methods hold great potential to benefit these real-world tasks.\n\nAdditionally, we conducted experiments on tasks that are commonly studied in prior work, including improving **human preference** and **aesthetic quality**. As shown in Figure 2 of the link, D-Fusion can enhance the performance of RL fine-tuning on these tasks. Hope these tasks are sufficient to demonstrate the generalization ability of our method.\n\n## 4. Computational Overhead\n\nThanks for this concern. We provide a detailed analysis in response to reviewer tsGd under “*3. Clarification on Efficiency*”. We would appreciate it if you could take a look.\n\nIn our experiments, DPO takes 78 min per round, whereas D-Fusion extends it to 91 min. The additional overhead is affordable. \n\n## 5. About Diversity and Failure Cases\n\n> How does fusion affect diversity? Are there failure cases?\n\nThanks for these questions!\n\nIn fact, D-Fusion can **mitigate loss of diversity** caused by RL. It is widely recognized that RL may cause the models to only generate images of specific patterns [1]. For instance, when using template \"*a(n) [animal] [human activity]*\", cartoon images tend to receive high alignment scores, because these prompts are often depicted in cartoon styles in the pre-training data. During RL, the model may learn to generate only cartoon images to achieve high alignment, which reduces diversity.\n\nD-Fusion helps mitigate this issue. The visually consistent sample pairs constructed by D-Fusion typically adopt **similar styles**, but receive **different alignment scores**. When training with these samples, the model does not associate a specific style with high alignment, thus maintaining diversity.\n\nIn Table 1 of the link, we show the diversity of images generated by fine-tuned models. As shown, employment of D-Fusion helps alleviate reduction in diversity.\n\nD-Fusion indeed occasionally results in **failure cases**. We show some cases in Figure 5 of the link. As observed, the failures likely arise from **insufficient attention** given to certain objects when sampling, causing the cross-attention mask to poorly outline their positions. This leads to confusion during subsequent fusion. However, based on our experimental results, the presence of a small number of failure cases does not significantly impact fine-tuning performance.\n\n[1] Training Diffusion Models with Reinforcement Learning.\n\n[2] DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models.\n\n[3] Aligning Text-to-Image Models using Human Feedback.\n\n[4] A Dense Reward View on Aligning Text-to-Image Diffusion with Preference.", "claims_and_evidence": "The paper's central claims are supported by extensive empirical results:\n\n1. **D-Fusion improves text-image alignment**: Experiments demonstrate superior CLIPScore performance and higher human preference rates compared to standard DPO.\n2. **D-Fusion constructs visually consistent training samples**: Qualitative examples and ablation studies confirm that self-attention fusion effectively preserves alignment features while ensuring visual consistency.\n3. **D-Fusion generalizes across different RL methods**: The results indicate that integrating D-Fusion with different RL fine-tuning approaches (DPO, DDPO, DPOK) leads to improved alignment.\n\nHowever, one limitation is that all experiments are conducted on **SD 2.1**, an older model, without evaluation on **more advanced diffusion architectures** such as **SDXL** or **Flux**, which may better reflect the state of the field.", "methods_and_evaluation_criteria": "The method is well-motivated and addresses a real challenge in diffusion model alignment. The **benchmarking approach is reasonable**, using:\n\n- **CLIPScore** as a quantitative metric\n- **Human preference testing** as a qualitative evaluation\n- **Three types of prompts (actions, attributes, spatial relationships)** to ensure diversity\n\nOne concern is **the absence of comparison to state-of-the-art models** such as SDXL or Flux. Given the rapid advancements in diffusion models, it is unclear whether the improvements hold in **larger-scale architectures**.", "theoretical_claims": "The paper primarily focuses on empirical improvements rather than theoretical derivations. The **formulation of diffusion models as a sequential decision-making process** and the **DPO objective function** appear correct and align with prior work.", "experimental_designs_or_analyses": "The experimental setup is **mostly sound**, but there are some limitations:\n\n1. **Model Selection Bias**: The choice of SD 2.1 for evaluation is outdated, raising concerns about applicability to modern architectures.\n2. **Lack of Baselines Beyond DPO Variants**: While the paper evaluates multiple DPO-based methods, it does not compare against alternative alignment techniques (e.g., **reward models, preference learning in SDXL**).\n3. **Generalization to Real-World Tasks**: The evaluation primarily focuses on synthetic prompt-image pairs, with no discussion of real-world generative applications (e.g., inpainting, video synthesis).", "relation_to_broader_scientific_literature": "The work aligns with ongoing research in **reinforcement learning for generative models** and **diffusion model alignment**. Specifically, it builds on:\n\n- **DPO-based fine-tuning for LLMs and diffusion models** (Wallace et al., 2023; Fan et al., 2023)\n- **Attention control techniques for image editing** (Cao et al., 2023; Hertz et al., 2022)\n- **CLIP-based alignment evaluation** (Radford et al., 2021)\n\nHowever, the paper does not **explicitly compare to recent SDXL-related studies** or Flux-based advancements, which limits its impact.", "essential_references_not_discussed": "**SDXL and Modern Diffusion Architectures**", "other_strengths_and_weaknesses": "### **Strengths:**\n\n1.  **Well-motivated problem** (prompt-image misalignment)\n2.  **D-Fusion is simple yet effective** (self-attention fusion for visual consistency)\n3.  **Extensive evaluation** (multiple RL algorithms, CLIPScore, human preference)\n\n### **Weaknesses:**\n\n1.  **Choice of model (SD 2.1) is outdated** → Lacks validation on SDXL/Flux\n2.  **No comparison to alternative fine-tuning approaches** (e.g., LoRA for alignment, reward-based RLHF)\n3.  **Does not explore computational efficiency** (D-Fusion adds complexity—does it scale?)", "other_comments_or_suggestions": "1. **Why was SD 2.1 chosen instead of SDXL?**\n2. **Does D-Fusion improve alignment in other generative tasks (e.g., inpainting, video synthesis)?**\n3. **How does D-Fusion compare to reward-based RL approaches for alignment?**\n4. **What is the computational overhead of self-attention fusion?**", "questions_for_authors": "1. **How does mask-guided fusion affect diversity?**\n   - Does it introduce mode collapse (favoring specific visual patterns)?\n2. **Are there failure cases?**\n   - Are there scenarios where D-Fusion reduces prompt-image alignment instead of improving it?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uuVNMTYStP", "reviewer": "Reviewer_ri7A", "summary": "This paper addresses the issue the effectiveness of preference learning (e.g. Direct Preference Optimization (DPO)) for text-to-image diffusion model is limited by the visual inconsistency between the sample that aligns better with the text and the sample that aligns less with the prompt, making it difficult to focus on the relevant differences between the positive and negative samples. The authors introduce D-Fusion (Denoising Trajectory Fusion), a method to construct visually consistent samples for preference learning. D-Fusion works by performing mask-guided self-attention fusion between a well-aligned reference image and a poorly-aligned base image, resulting in a target image that is both well-aligned and visually consistent with the base image, while also retaining the denoising trajectories necessary for DPO training. Through comprehensive experiments on Stable Diffusion, the authors demonstrate that applying D-Fusion can significantly improve prompt-image alignment when used with different reinforcement learning algorithms like DPO, DDPO, and DPOK. The paper highlights the necessity of using visually consistent image pairs for fine-tuning diffusion models with DPO and introduces D-Fusion as a compatible approach to address this challenge", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for your time and efforts in reviewing our paper. We are happy that you think the intuition makes sense. In addition, we feel pleased that you approve our experiment design. \n\nBelow are responses to your concerns and suggestions. Please let us know if you require any further information, or if anything is unclear. \n\n## 1. Clarification on Comparison with DDIM Inversion\n\n> As mentioned, can you expand the ablation study section to provide more clarity on the motivation and the value for having the comparison with DDIM inversion? \n\nThanks for this concern! We would like to clarify on it in a clearer manner. \n\nFirstly, fine-tuning with reinforcement learning requires access to the **denoising trajectories** of the images. This paper highlights the necessity of fine-tuning diffusion models using visually consistent samples. However, constructing visually consistent samples through **image editing** is not feasible, as the editing process disrupts the denoising trajectory (See right column of Line 90 in the paper for detail). The advantage of D-Fusion lies in its ability to generate visually consistent samples while **preserving their denoising trajectories**.\n\nHowever, there exist two straightforward methods to create a denoising trajectory for any given image: the **forward process** of diffusion models and the **DDIM inversion**. (The former simply adds noise to an image, which is so trivial that we did not mention it in the paper.) The created denoising trajectories can also be used for reinforcement learning fine-tuning.\n\nTherefore, we designed this comparative experiment, where the training data consists of the **same visually consistent samples**, but the **denoising trajectories come from D-Fusion and DDIM inversion respectively**. The results indicate that training with denoising trajectories created by DDIM inversion performs poorly. This effectively rules out the method of fine-tuning the model using DDIM Inversion through the following steps: \n\n* generating visually consistent samples via **image editing**, \n* creating denoising trajectories using **DDIM inversion**, \n* fine-tuning with samples and trajectories. \n\nThus, this comparison further demonstrates the advantage of D-Fusion (*i.e.*, generating visually consistent samples while preserving their denoising trajectories).\n\nWe would like to further discuss the **forward process**. Adding noise to images through the **forward process** essentially turns the training to supervised fine-tuning (SFT). We have also provided experimental results on SFT. if you're interested, please refer to Figure 1(a) at this link (https://anonymous.4open.science/r/ICML-25-Rebuttal-2703/rebuttal_2703.pdf). As shown, SFT also performs poorly in improving alignment, further demonstrating the effectiveness of RL training with D-Fusion. \n\n## 2. About Writing Suggestions\n\n> * Writing quality has room to improve.\n> * (Line 25, abstract) On the one hand -> On one hand. \n> * I suggest using $I^t$ instead of $I^a$ for the target image, following $I^b$ for base image and $I^r$ for reference image.\n\nThank you again for providing these suggestions. In the final version, we will conduct a **thorough review** to correct errors and enhance writing quality. For instance, we will revise the phrase \"on the one hand\" in the abstract, as you pointed out, to \"on one hand\". We will also revise $I^a$ to $I^t$, in order to keep consistent with $I^r$ and $I^b$.", "claims_and_evidence": "The claims made in the submission appear to be supported by evidence. Both qualitative results (Figure 4) and quantitative results (Figure 5, Table 1), including CLIPScore metrics and human preference tests (Figure 6), consistently demonstrate the effectiveness of D-Fusion in improving text-image alignment, compared to the plain DPO and the base model. These support the hypothesis that preference optimization can work better with visual similar pairs, and demonstrate the effectiveness of the proposed method.", "methods_and_evaluation_criteria": "Denoising Trajectory Fusion is an empirical method, motivated with intuition. \n\nIt aims to generate well-aligned and also visually similar positive image samples for preference learning. The process involves two key steps:\n1. Cross-Attention Mask Extraction: A mask is extracted from the cross-attention maps of the reference image's denoising process. This mask identifies the image regions corresponding to the prompt's content.\n2. Self-Attention Fusion: Starting with the same random noise as the base image, a target image is denoised. During this process, the self-attention keys and values of the target image are fused with those of the reference image using the extracted mask. In the masked (prompt-related) areas, the target image adopts the self-attention information from the reference image, enhancing alignment. In the unmasked areas, it retains its own information, ensuring visual consistency with the base image.\n\nAlthough there is no theoretical support, the intuition makes sense and the evaluation criteria used in the paper are standards for evaluating text-to-image diffusion models.", "theoretical_claims": "This is a pure empirical paper.", "experimental_designs_or_analyses": "The experiment designs are pretty standard for evaluating text-to-image diffusion models. On the other hand, I'm not very sure I understand the motivation and the value for having the comparison with DDIM inversion in ablation study (or whether it should be called an ablation studies). I'd like to see more details about that.", "relation_to_broader_scientific_literature": "It's related the literature of text-to-image diffusion model alignment.", "essential_references_not_discussed": "I'm not aware of significant missing references.", "other_strengths_and_weaknesses": "No additional comments.", "other_comments_or_suggestions": "- Writing quality has room to improve.\n- (Line 25, abstract) On the one hand -> On one hand\n- I suggest using $I^t$ instead of $I^a$ for the target image, following $I^b$ for base image and $I^r$ for reference image.", "questions_for_authors": "As mentioned, can you expand the ablation study section to provide more clarity on the motivation and the value for having the comparison with DDIM inversion?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces D-Fusion, a method for improving text-image alignment in diffusion models by generating visually consistent training pairs. Reviewers found the approach well-motivated, empirically strong, and broadly compatible with existing RL fine-tuning methods.

While one reviewer noted missing baselines and limited theoretical grounding, the majority considered the method practical and clearly presented. Given that the rebuttal effectively addressed most concerns, the reviewers and ACs recommend acceptance. Congratulations to the authors on the good work!
Decision: Accept (poster)