CHATS: Combining Human-Aligned Optimization and Test-Time Sampling for Text-to-Image Generation
Authors: Minghao Fu
Guo-Hua Wang
Liangfu Cao
Qing-Guo Chen
Zhao Xu
Weihua Luo
Kaifu Zhang
Venue: ICML 2025 poster (2025)
Keywords: Diffusion Models
Flow Matching Models
Human Preference Optimization
Classifier-Free Guidance
Text-to-Image Generation
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: We propose CHATS, a novel generative framework that facilitates the collaboration between human preference alignment and test-time sampling.

Abstract: Diffusion models have emerged as a dominant approach for text-to-image generation. Key components such as the human preference alignment and classifier-free guidance play a crucial role in ensuring generation quality. However, their independent application in current text-to-image models continues to face significant challenges in achieving strong text-image alignment, high generation quality, and consistency with human aesthetic standards. In this work, we for the first time, explore facilitating the collaboration of human performance alignment and test-time sampling to unlock the potential of text-to-image models. Consequently, we introduce CHATS (Combining Human-Aligned optimization and Test-time Sampling), a novel generative framework that separately models the preferred and dispreferred distributions and employs a proxy-prompt-based sampling strategy to utilize the useful information contained in both distributions. We observe that CHATS exhibits exceptional data efficiency, achieving strong performance with only a small, high-quality funetuning dataset. Extensive experiments demonstrate that CHATS surpasses traditional preference alignment methods, setting new state-of-the-art across various standard benchmarks. The code is publicly available at github.com/AIDC-AI/CHATS.

CHATS: Combining Human-Aligned Optimization and Test- Sampling for Text-to-lmage Generation
Minghao Fu 1231
Guo-Hua Wang"
Liangfu Cao
Qing-Guo Chen 3
Zhao Xu
Weihua Luo
Kaifu Zhang
CHATT
Figure 1. Images generated by CHATS with SDXL, an innovative text-to- toaeio--eeeeaa generation framework that, for the first time, facilitates collaboration between human preference alignment and test-time sampling, establishing new paradigm in text-conditioned image synthesis.
Abstract
in achieving strong text-image alignment, high generation quality, and consistency with human aesthetic standards. this work, we for the first time, explore facilitating the collaboration of human performance alignment and test-time sampling to unlock the potential of text-to-image models. Consequently. we introduce CHATS (Combining Human-Aligned optimization and Test-time Sampling), a novel generative frame- work that separately models the preferred and dispreferred distributions and employs a proxy- prompt- based sampling strategy to utilize the use- ful information contained in both distributions. We observe that CHATS exhibits exceptional data efficiency, achieving strong performance with only small, high-quality funetuning dataset. Ex- tensive experiments demonstrate that CHATS sur- passes traditional preference alignment methods,
Diffusion models have emerged as dominant approach for text-to-image generation. Key com- ponents such as the human preference alignment and classifier-free guidance play crucial role in ensuring generation quality. However, their independent application in current text-to-image models continues face significant challenges
Work done during the internship at Alibaba Group. National Key Laboratory for Novel Software Technol- ogy, Nanjing University "School of Artificial Intelligence, Nanjing University "Alibaba Group. Correspondence to: Minghao Fu <ffum lamda.nju.cdu.nn>. Guo-Hua Wang <wangguohuae @alibaba- cattbbbba-in.coo... inc.com..
Proceedings of the nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

CHATS: Combining Human-Aligned Optimization and Test-Time Sampling for Text-to-Image Generation
setting new state-of-the-art across various stan- dard benchmarks. The code is publicly available at github.cm/AAIDC--A/CHAA
novel, theoretically grounded objective function based the general principle of direct preference optimization (DPO) (Rafailov et al., 2023). During sampling, two mod- els collaborate to guide the denoising process towards the preferred model while concurrently repelling it from the dispreferred model using a proxy-prompt prompt-based sampling strategy to leverage conditional and unconditional informa- tion.
1. Introduction
Diffusion models (Cao et al., 2024) have emerged as a lead- ing generative framework (Kingma & Welling, 2014; Sohl- Dickstein et al., 2015; Goodfellow et al., 2020) by iteratively transforming random noise into structured images (Ho et al., 2020; Rombach et al., 2022; Song al., 2022). Text-to- image generation, as a key application, offers broad versatil- ity, supporting tasks like image editing (Brooks et al., 2023) and generative conversational AI (Brown et al., 2020).
Our method is simple, intuitive, has robust theoretical foundation, while also demonstrating high efficiency and effectiveness, In particular, training our approach requires only 7,459 high-quality preference pairs, exhibiting sig- nificantly greater data efficiency compared to traditional preference optimization methods (Wallace et al., 2024).
Empirical evaluations on two mainstream texl-to-image gen- eration frameworks, diffusion (Podell et al., 2024) and flow matching (Liu et al., 2023; Lipman et al., 2023), under- score the superiority of our method. Our approach not only surpasses naive CFG sampling strategies but also outper- forms the most effective preference optimization methods currently available (Wallace et al., 2024). Furthermore, the default configuration of hyperparameters in our method performs robustly across different model architectures, high- lighting its practical utility.
In particular, high-quality image generation largely depends on two stages: 1) human preference alignment, in which ranked pairs reflecting human preference are utilized to train the model for maximizing the distinction between pre- ferred and dispreferred generated images; and 2) sampling, where Classifier-Free Guidance (CFG) (Ho & Salimans, 2022) plays a vital role by merging conditional and un- conditional noise predictions to focus on regions of higher eox--conditioned density.
Preference alignment involves human experts categorizing images into preferred and dispreferred groups based their aesthetic desirability, and using these two groups of images to train a model for capturing desirable character- istics from preferred images while avoiding undesirable features or errors present in dispreferred ones. Similarly, the sampling procedure of CFG can be conceptualized as denoising from preferred distribution (conditional) while simultaneously steering away from dispreferred distribu- tion (unconditional). This dual- framewrrk naturally aligns with the principles of preference alignment. Nevertheless, previous efforts (Dhariwal& & Nichol, 2021; Fan et al., 2023; Yang et al., 2024; Ahn et al., 2024; Black et al., 2024) focus on optimizing these two processes in isolation, leaving the integration of the two for leveraging their mutual benefits unexplored and resulting in suboptimal outcomes. This presents an opportunity for us to develop unified method that enables a more effective solution.
As far as we know, our work marks the first attempt to unify the human preference alignment and sampling stages within the current text-to-image generation framework. We hope that our research can shed new light on the paradigm design of such generative systems.
The contributions of this paper can be summarized as:
We are the first to integrate human preference optimiza- tion finetuning with the sampling process, leveraging their inherent synergy to refine image generation.
We propose CHATS, a novel framework that offers precise control over preference distributions while en- suring efficient sampling. CHATS requires only small high-quality dataset for training, maintaining high data efficiency.
Extensive experiments are conducted to demonstrate that the collaboration between preference alignment and sampling together yields significantly improved generation performance compared to baseline methods.
Motivated by this, we introduce CHATS (Combining Human- Aliman-Aligned optimization and Test-time Sampling), a novel approach that integrates human preference alignment with the sampling stage, thereby further enhancing the per- formance of text-to-image generation systems.
2. Related Work
Specifically, CHATS employs two distinct models: one tasked with learning the preferred distribution (referred to as the preferred model) and another dedicated to capturing the dispreferred and unconditional distributions (referred to as the dispreferred model). These models are trained using
2.1. Human Preference Optimization
Human preference optimization aims to improve the aes- thetic quality of generated images. One line of research (Xu et al., 2024; Wu al., 2023; Kirstain et al., 2023b; Lee et al.,
2

---

CHATS: Combining Human-Aligned Optimization and Test-Time Sampling for Text-to-Image Generation
Direct Preference Optimization (DPO)
Classifier-Free Guidance (CFG)
EC
Go (Z 8)
Ic+ 6
(1 S)E SE4
Eo
purfer disprefer
CHATS
Egt (2,,t)
(xt,t)
s)E SEc
vs.
E (ze, 1)
EB (Zp. t)
prefer disprefer
mu a))
Figure 2. Illustration of CHATS. Both DPO and CPG adopt preference-haset frameworks in their formulations. DPO derives preferences from user feedback, while CFG utilizes diverse prompting strategies to capture such discrepancies. However, their current independent working paradigms present significant potential for integration. Our CHATS addresses this by separately modeling preferred and dispreferred information with two distinct models. Additionally, it employs proxy-prompt-bane sampling strategy to facilitate effective collaboration between two models.
2023) focused on developing reward models to evaluate the alignment between generated images and their correspond- ing prompts. These reward models are subsequently used to finetune diffusion models, guiding them to better capture desirable characteristics.
biasing the noise towards areas with higher semantic den- sity. Numerous methods (Hong et al., 2023; Bansal et al., 2023; Luo et al., 2024; Nichol et al., 2022) were proposed to further refine CFG. In contrast to these approaches, as shown in Fig. 2, our CHATS enhances sampling quality by facilitating collaboration with DPO, enabling both processes to mutually reinforce each other and thus achieve superior results.
Another line of research explored direct finetuning on preferred images (Dong et al., 2023; Dai et al., 2024) or adopted reinforcement learning from human feedback (RLHF) (Black et al., 2024; Fan et al., 2023; Yang et al., 2024; Wallace et al., 2024). Specifically, DDPO (Black et al., 2024) formulated denoising as a multi-step decision- making problem. DPOK (Fan et al., 2023) incorpo- rated KL. regularization as an implicit reward to stabilize the finetuning process. D3PO (Yang al., 2024) and Diffusion-DPO (Wallace et al., 2024) extended the con- cept of DPO (Rafailov et al., 2023) to diffusion models by optimizing the respective policies. Different from these methods, CHATS employs two distinct models to separately capture preferred and dispreferred information.
3. Preliminaries
3.1. Diffuusio Models
leffusion-basedd generative frameworks (Sohl-Dickstein et al., 2015; Ho al., 2020) operate by iteratively injecting Gaussian noise e sampled from N I) into clean data zg- creating progressively noisier versions 2, at each timestep (. This forward diffusion is described by:
q(z, zn) = -N(2 Vo, 20, v1 - og I).
(1)
2.2. Guided Sampling for Text-to-Image Generation
where a, = 1 Br and ā = II x= a2 emerge from a pre- determined schedule ( 3r). The denoising model co is trained to predict the noise under the objective:
Guided sampling enhances fidelity by steering generation towards specific directions determined by various strategies. CG (Dhariwal & Nichol, 2021) utilized the gradients from classifier the signal to achieve class-conditional gener- ation at the cost of diversity. CFG (Ho & Salimans, 2022) extended this concept to the open-vocabllary domain by combining conditional and unconditional outputs, thereby
LDifiusion = Be.t.e Ea EE 1)
(=)
(2)
where it uniformly sampled from (1...., T).
3

---

CHATS: Combining Human-Aligned Optimization and Test-Time Sampling for Text-to-Image Generation
3.2. Rectified Flow
generation by estimating the diffusion score (Song al., 2022) as:
Rectified Flow (Liu et al., 2023; Lipman al., 2023) learns direct mapping from noise target data through reversible ODE:
éB = -1 s)e SEB (za).
(8)
dæ
vs (2(, t).
(3) CFG biases the generation process towards the conditional distribution while reducing reliance on the unconditional counterpart, effectively aligning the generated content with the provided textual prompt.
dt
The intermediate state o, during training is approximated as:
= (1 - 20 + te,
(4)
4. CHATS: Combining Human-Aligned Optimization and Test-Time Sampling
where t E [0, 1] controls the progression from 2g to noise
3.3. Direct Preference Optimization
DPO and CFG both enhance generation performance by leveraging preferred and dispreferred information, yet they rely on a single model to process and balance these conflict- ing distributions. In contrast, as shown in Fig. 2, our CHATS adopts a more nuanced approach by employing two models, initialized from the same reference model, independently capture the characteristics of the preferred and dispreferred distributions. During sampling, these models collaborate using a proxy-prompt- based conditional strategy, seamlessly integrating their strengths.
DPO (Rafailov et al., 2023) formalizes the preference op- timization problem as single-stage policy training based on human preference data, without explicitly relying on a reward model. Follow this rule, given condition c, only preference-ranked pairs 20 > 20 (preferred vs. non- preferred) are available. The Bradley-Terry (Bradley & Temyy 1952) model characterizes the human preference dis- tribution as:
p(z > (c)) - (r((ttc)) r(āū,)))
4.1. Training
where (-) denotes the sigmoid function and r is reward model that quantifies the coherence between the input im- age and text. Approximated by neural network param- eterized by 0, is optimized by minimizing the negative log-likelihood, as follows:
We aim for both the preferred and dispreferred models to maximize their reward scores, defined by rg+ and ra-, re- spectively. However, the introduction of two independent models brings a new challenge in designing a valid learning objective. In this scenario, the naive DPO loss defined in Eq. 6 is no longer suitable, its direct would lead to the collapse of the dispreferred model. To address this issue, we reformulate the training objective as follows:
Looo -E to WDD [logpo(=) > 20 [c)]
where D {(c, at 2 ) represents the preference dataset obtained through human feedback. They build the work of Jaques et al. (2017; 2020) by implicitly employing reward model to optimize the task distribution Po using reference distribution Pref and demonstrating that pe admits a global optimum. Leveraging this property. DPO derives the following final preference alignment objective:
L= Fc. :z ->>> [log a (re+ (ot, c)]] (9)
where the sum of reward scores from two models is treated as the target. This formulation enables both models to leam meaningful image distributions, with the preferred model capturing high-quality characteristics and the dispreferred model fitting less optimal but complementary image dis- tribution.
LDOO = - [e, 25 jwD log of log po(= log (o (e)
pur(= (c) p=t 1 (c)
Training using this objective poses significant chal- lenges in deriving the task distributions Pot (2,_1 ||,²)) and Ps- 1 as we need to account for all potential de- noising trajectories from žт to 2g under conditioning c. This complexity arises from the intractability of marginalizing over all intermediate steps in the reverse diffusion process.
3.4. Classifier-Free Guidance
To enhance the alignment between generated images and textual descriptions, Classifier-Free Guidance (CFG) (Ho & Salimans, 2022) modifies the sampling distribution as:
po (z((cc x po(2:
To address this challenge, following Wallace et al. (2024), we define R(z c) as the reward score associated with the entire denoising chain from 27 to 2g given the condition c. We further express the per-sample reward r(26, c) as the expectation of this reward score over the intermediate steps
where e denotes the textual input, Pe(zecc) represents the conditional generative distribution, (2)) is the uncondi- tional distribution and s is a scale scalar. CFG guides the

---

CHATS:
Combining Human-Aligned Optimization and Test-Time Sampling for Text-to-Image Generation
zi:r
4.2. Sampling
r( r(2b.c) c) = E, (ff.r (=6.)) [R(eh: d]]
(10)
After training, we obtain two models: one corresponds to the preferred distribution, denoted as Pø+, and the other corre- sponds to the dispreferred distribution, denoted po-. Mo- tivated by the mechanism of CFG (Ho & Salimans, 2022), we propose a novel strategy to leverage the contribution of these two models. Let 2t be the noisy image at step t and c be the condition. The sampling distribution of CHATS is expressed as:
where i E {+. represents the preferred and dispreferred distributions, respectively. We abbreviate c to simplify the notation. By substituting Eq. 10 into Eq. 9, assuming <i.7 ~ Pe- 1 and 21:T ~ Pø- (si-r =o ), with = and #6 treated as imput variables, the loss for each ranked pair is defined as:
Po (z( (zxlc) (x- (2)((2x) (1+a)s (15)
[(=),2) . =-=== = -looo E log Put (= +log Pu- (tor) Par(zorr))
Parl=
Please refer to Sec A.4 for the details of its theoretical foundations. In this formulation, po+ (=+]c)+++ encourages the samples to align closely with the preferred distribution. When 0 < a < Its the dispreferred distribution Pe- is partially incorporated, allowing to contribute poten- tially useful patterns while remaining less influential than Pot. Conversely, when o O, the terms pe- (zylc) and Pg (z,) (1++a) actively push the samples away from unde- sired modes in the dispreferred distribution, both condition- ally and unconditionally.
(11)
We approximate Pai (26) using ((=1:1/20) as the di-
rect estimation of pos (=)) is intractable. Leveraging Jensen's inequality, given (=Ø :6) ~ D. 1 ~ u(0, T).
ztt ~ q(=5 |=5) and of ~ g(2 (zz). we obtain the follow- ing upper bound:
C(00. =00 S -E,stin log o T
Dxr. (a(z/t / =0 pp+ (z/=)|=)))
(x( 20 Peel ( -======+))
(12)
This balanced approach ensures that the sampling process leverages the strengths of the preferred distribution while selectively integrating or excluding aspects of the dispre- ferred distribution, depending on the value of a. Empirically, observe that this strategy is not sensitive to the precise choice of a and default value of 0.5 performs well across diverse text-to-image models.
+ DxI.(1(=T111z) 20 PP0 (0 ((
[x( ||
For diffusion models, the loss function of CHATS is explic- itly formulated as:
Similar to (Ho & Salimans, 2022), the predicted noise of CHATS is combined as follows:
- L .0 ).( (..47777))) logo -1
+ ||e* -eot (zt", 2)||2-1|e"* Garlzt t)]
(13)
(0 c) = (1 + s) (ze: c)
(16)
+ -||---o - (zt ,) -1kE Gar(2 -,al( (00l,,))))))) 001²)]
oc (2,, c) + (1 (1+ (z))
For flow matching models, the final expression remains sim- ilar, with eo simply replaced by vg. To improve the sampling efficiency of our method, we introduce an approximation that constructs a proxy prompt by linearly combining the prompt e with a null prompt Ø after they are transformed into text embeddings, i.e., = -ac + (1 + a) We then use this proxy prompt as the input to the dispreferred model:
For flow matching models, q( ((2=11zz, 20) is expressed using the Dirac delta function. In this setting, is uniquely deter- mined by Eq. 3. Consequently, the loss function is defined as:
:o 35500 ( log o [ (-T(
CCHATS
E
+ ||ot -ott-vo+ (ztt 1)12- -1lot vmr(=t,1)) 1)1 vot
(14)
(z e) = (1 + s)ca+ (z,, - seg- (2t,
(17)
+ -110--oo uo- (zT 0012 - -00 (anlt:.00]])]
Note that when a O, the method degenerates into CFG but utilizes two models. This modification significantly re- duces the number of forward passes required for generation by unifying and Ø into a single representation. Empirical evaluations demonstrate that this substitution achieves ro- bust performance and serves as an effective surrogate for incorporating dispreferred information.
Please refer to Supp. A.2 for detailed derivation of the training objective. Our method ensures that the two models, responsible for learning the preferred and dispreferred dis- tributions, achieve the dual objectives of accurately fitting their respective targets while not shifting too far from the reference model.
5

---

CHATS: Combining Human-Aligned Optimization and Test-Time Sampling for Text-to-Image Generation
Table HPS v2 results (1) on multiple benchmarks. The best results are bold face.
Model
Method
Anime
Concept-Art
Paintings
Photo
GenEval
DPG-Bench
Standard
19.66
18.11
18.25
20.13
21.20
18.88
SD1.5
Diffusion-DPO
21.65
20.12
19.87
20.74
22.04
19.87
CHATS
27.74
26.17
26.04
26.43
26.17
24.97
Standard
30.20
28.38
28.19
26.88
28.19
27.38
SDXL
Diffusion-DPO
31.69
29.64
29.83
28.34
29.51
28.71
CHATS
32.96
30.94
31.08
29.62
29.81
29.51
Standard
25,00
23.60
23.18
26.09
27.23
25.68
In-house T21
Diffusion-DPO
26.03
24.87
24.71
26.56
27.62
26.16
CHATS
30.41
29.87
29.95
29.30
29.84
29.53
5. Experiments
on the PaP v2 dataset, whereas CHATS achieves superior results when trained on OIP. Consequently, by default we report results from Diffusion-DPe finetuned on PaP v2 alongside CHATS finetuned on OIP for a fair comparison. More details about data efficiency can be found in Sec. 5.3 and Table 5.
In this section, first provide detailed description of our experimental settings. We then compare our method with state-of-the-art approaches on multiple text-to-image gener- ation models and benchmarks to highlight its effectiveness. Finally, we conduct essential ablation studies to investigate the impact of each component in our CHATS.
In table I we report the HPS v2 results over diverse prompt groups as this aesthetic metric is better aligned with human judgment compared to ImageReward and PickScore met- rics (Wu et al., 2023) (see Supp. C for full results). CHATS consistently surpasses two baselines in cases, underlining its robustness in handling various visual styles. Moreover, CHATS effectively narrows the aesthetic gaps among differ- ent models. For example, the largest discrepancy in average scores across the six groups occurs between SD1.5 and SDXL. Under the Standard setting, this gap is 8.83, but it decreases to 4.41 when adopting CHATS.
5.1. Experimental Settings
Models and datasets. We utilize two model families: dif- fusion models and flow matching models. For diffusion models, we employ Stable Diffusion 1.5 (SD1.5) (Rombach al., 2022) and Stable Diffusion XL-1.0 (SDXL) (Podell al., 2024). For flow matching models, deploy In-house T21, text-to-image generation model optimized for pho- torealistic images in e-commerce scenarios, based on the DiT architecture (Peebles & Xie, 2023). We conduct exper- iments primarily on two preference optimization datasets, Pick-a-Pic v2 (PaP v2) (Kirstain al., 2023a) and OpenIm- agePreferences (OIP) (Data is Better Together, 2024) (see Supp. B.I for more details).
Table 2 presents the evaluation results on the GenEval bench- mark, averaged over four random seeds. In this challenging scenario, CHATS outperforms baselines in 12 out of 18 tasks, achieving average improvement of 2.39% com- pared to Diffusion-DPO. Notably, CHATS excels in difficult tasks such as "Counting" (+10.18% over Diffusion-DPO) and "Color attribution" " (+2.13%) underscoring its capa- bility to capture complex relationships multi-object or attribute-focused prompts. Furthermore, the results on DPG- Bench (Table 3) show similar trend, with CHATS achiev- ing the highest overall evaluation scores. Combined with the previous findings, these outcomes confirm the superior- ity of CHATS across diverse evaluation metrics and model architectures, effectively establishing a new state-of-the
Evaluation. We utilize publicly available benchmark prompts from GenEval (Ghosh et al., 2023), DPG- Bench (Hu et al., 2024), and HPS v2 (Wu et al., 2023). Detailed information about these benchmarks is provided in Supp. B.3. We employ multiple evaluation metrics, in- cluding HPS v2 (Wu et al., 2023), ImageReward (Xu al., 2024), and PickScore (Kirstain et al., 2023b), which are trained on extensive preference datasets, offering robust in- sights about authentic human preferences. During sampling, by default we keep and a as and 0.5, respectively.
5.3. Identifying Key Components of CHATS
5.2. Main Results
Modular ablation. To elucidate the critical elements of our CHATS, we conduct an ablation study by examining differ- ent fintuning and sampling configurations. The CHATS dis- tinguishes itself from traditional finetuning methods through two primary innovations: (1) the introduction of two sep- arate models dedicated to learning the preferred and dis-
We compare our CHATS against two baselines: (1) the Standard versions of SD1.5, SDXL, and In-house T21, and (2) Difffusion-Di DPO (Wallace et al., 2024), the current state- of the DPO method for text-to-image generation. We find that Diffusion-DPO performs better when finetuned
6

---

CHATS: Combining Human-Aligned Optimization and Test-Time Sampling for Text-to-Image Generation
Table 2. Quantitative results (%) averaged by random seeds GenEval.
Model
Method
Single object
Two object
Counting
Colors f
Position t
Color attribution
Overall f
Standard
95.31
38.13
5.94
75.27
7.25
9.25
38.52
SD1.5
Diffusion-DPo
96.56
49.24
5,94
82.71
7.75
9.75
41.99
CHATS
98.75
47.73
20.94
78.72
7.75
10.75
44.11
Standard
98.44 44
68.43
40.62
83.78
10.25
20.75
53.71
SDXL.
Riffusion-DPO
99.53
81.94
45.47
88.70
13.25
26.75
59.27
CHATS
99.38
79.09
53.50
87.45
13.90
27.40
60.12
Standard Diffusion-DPO
98.75
75.25
37.81
85.37
25.75
44.00
61.16
In-house T21
99.06
75.25
40.00
83.78
27.25
48.00
62.22
CHATS
99.06
82,58
47.50
90.16
26.50
52.75
66.42
Table Quantitative results (%) on DPG-Bench.
Model
Method
Global f
Entity
Attribute
Relation
Other
Overall f
Standard
74.77
68.74
69.72
79.11
39.20
57.92
SDI.5
Diffusion-DPO
81.76
71.02
72.78
80.66
38.
61.67
CHATS
77.51
73.67
74.34
80.23
58,80
64.44
Standard
85.41
81.24
78.90
87.04
60.80
74.37
SDXL
Diffusion-DPO
84.50
82.00
80.16
87.39
67.20
74.66
CHATS
84.50
82.91
80.12
87.47
70.80
76.08
Standard Diffusion-DPP
85.11
87.98
87.91
92.03
75.60
81.27
In-house T21
86.63
88.98
87.87
91.91
75.20
82.18
CHATS
82.67
90.11
88.17
92.34
84.40
83.72
Table 4. Ablation study on various configurations using In-house with prompts from Photo. "s" denotes guidance scalar in CFG as in Eq. 8.
Table 5. Ahlation studies on different finetuning datasets. HPS v2
results are reported quantitative comparison.
Finetuning Dataset
Benchmarks Photo
Method
Model
Configuration
Paintings
HPS v2 (1)
Finetuning
Sampling
PaP v2
28.34
29.83
SDXL
single model (full data)
5=5
28.64
Diffusion-DP2
OIP
28.16
29.03
single model (preferred data)
s=5
28.70
In-house T21
PaP v2
26.56
24.71
two models (w/o ref)
s=5
28.99
26.47
24.10
two models
$=
29.15
PaP v2
28.67
30.12
models
0.5
29.30
SDXL
29.62
31.08
CHATS
PaP v2
28.73
28.70
In-house T21
OIP
29.30
29.95
preferred distributions, and (2) the integration of guidance signals from two models.
As shown in Table 4, the baseline configuration, referred to as "single model (full data)", trains single model the entire DPO dataset by converting all preference pairs (2 2 c) into individual samples (z+,, c) and (z c). These samples are combined to construct a dataset without explic- itly ranked preference pairs and are subsequently used to train the model with a standard flow matching loss (Liu al., 2023). This configuration achieves an HPS v2 score of 28.64. Restricting the training to only preferred data results in slight performance improvement to 28. 70, sug- gesting that a single model fails to effectively leverage all preference information when full data is provided. Intro- ducing second model without reference policy leads to a more substantial increase in performance, yielding an HPS
v2 score of 28.99, thereby underscoring the advantages of independently modeling preferred and dispreferred distri- butions. By further integrating reference model to guide the finetuning process elevates the score to 29.15. The most significant improvement observed when incorporating the sampling strategy CHATS with a 0.5, resulting the highest HPS v2 score of 29.30.
These findings demonstrate that both the dual-model archi- tecture and the cooperative sampling strategy are crucial for enhancing the alignment of generated outputs with human preferences, thereby validating the critical design choices underpinning our CHATS.
A small high-quality preference dataset is enough. We

---

CHATS: Combining Human-Aligned Optimization and Test-Time Sampling for Text-to-Image Generation
Standard
Diffusion-DPO
CIATT
Table Throughput with 50 sampling steps, measured on NVIDIA A100 GPU with BF16 inference. "bs" specifies the batch size.
Throughput (img/sec)
Model
Method
bs=1
bs=4
Standard CHATS
0.187
0.217
SDXL
0.166 (-11%)
0.210 (-3%)
Standard CHATS
0.153
0.162
In-house T2I
0.148 (-3%)
0.158 (-2%)
SDXL
In-house T21
29.8
29.35
29.7
29.30
29.6
29.25
HPS
29.5
29.20
29.4
29.15
29.3
29.10
WELCOME
FREUNDS
a
Figure 3. HPS v2 results by varying of CHATS. The evaluation are conducted with SDXL and In-house T2I, using prompts from Photo.
Figure 4. Qualitative comparison among various methods. The im- ages generated by CHATS exhibit stronger text-image alignment, higher visual fidelity, and align more closely with human aesthetic standards. Prompts: 1) Three friends working in magical bukery. 2) An anthropomorphic cat riding a Harley Davidson in Arizona with sunglasses and leather jacket. 3) A kangaroo in an orange hoodie and blue sunglasses stands on the grass in front of the Sydney Opera House holding a sign that says Welcome Friends.
further conduct ablation studies on two finetuning datasets with distinet characteristics: PaP v2 (Kirstain al., 2023a) and OIP (Data is Better Together, 2024). PaP v2 contains large number of preference pairs but suffers from lower data quality, whereas OIP comprises ~ 1% of PaP v2's entries yet features significantly higher-qualin generated images.
ing preference optimization with only a small set of images. Robustness of a. The hyperparameter a in CHATS con- trols the balance between conditional and unconditional textual embeddings and regulates the contribution of the dispreferred distribution during sampling. We analyze its impact by varying a. As shown in Fig. 3 (see Supp. D for more results), both SDXL and In-house T2I models exhibit consistent trends: as a increases from approximately 0.5, the HPS v2 results improve significantly. Beyond this range, the performance gradually saturates. Notably, com- pared to the naive CFG case (a = 0), the proposed sampling strategy leveraging two models consistently enhances aes- thetic scores across different model families. These results indicate that a is robust to variations in its value, with the default setting of 0.5 performing reliably well.
As presented in Table 5, CHATS consistently outperforms Diffusion-DPO by substantial margin across all scenarios. Notably, while Diffusion-DDPO achieves superior perfor- mance when finetuned on PaP v2 dataset, CHATS excels when leveraging the smaller, high-quality OIP dataset.
The contrasting behaviors between Diffusion-DPO and our CHATS stem from their distinct training objectives. Diffusion-DPP maximizes the difference in diffusion squared-loss values (ef. Eq. 2) from ranked preference pairs. mini-
However, in the OIP dataset, these differences mal because the ranked pairs are derived from two closely matched cop-performing models. As a result. Diffusion- DPO struggles to distinguish and model preferred and dis- preferred distributions, leading to suboptimal performance compared to training on the larger PaP v2 dataset with stronger preference signals.
6. Conclusions and Limitations
In contrast, CHATS adopts more direct approach by inde- pendently modeling the preferred and dispreferred distribu- tions using two distinct models, while explicitly facilitating collaboration between them. This design enables CHATS to effectively utilize high-quality data and capture nuanced preference information, offering the possibility of perform-
In this paper, we for the first time explored integrating hu- man preference optimization and sampling process to ad- vance -otate synthesis. We proposed CHATS, a novel training and sampling framework that enables better utilization of preference distributions by independently mod-
8

---

CHATS: Combining Human-Aligned Optimization and Test-Time Sampling for Text-to-Image Generation
eling preferred and dispreferred data. Our method exhibits high data efficiency, requiring only small, high-quality dataset for finetuning to achieve strong results. Extensive experiments demonstrated the superiority of CHATS across various benchmarks, setting a new state-of-the-art compared to traditional non-synergistic baselines.
are few-shot learners. In Advances in Neural Information Processing Systems, pp. 1877-1901, 2020.
Cao, H., Tan, C., Gao, Z., and others. A survey on generative diffusion models. IEEE Transactions on Knowledge & Data Engineering, 36(07):2814-2) 14)-2830,, 2024.
Our method has a limitation: in CFG, conditional and un- conditional prompts are typically concatenated for a single forward pass with batch size of 2, but model processes them separately in two forward passes with a batch size of I each. Consequently, our approach slightly reduces infer- ence throughput as shown in Table 6. However, this impact diminishes as the batch size increases. In industrial applica- tions, the primary focus on generation quality. In practice this additional overhead becomes negligible, model dis- tillation (Meng al., 2023) can effectively unify the two models into single, optimized model, such as FLUX.11des
Dai, X., Hou, J., Ma, C.-Y., et al. Emu: Enhancing image generation models using photogenic needles in a haystack. In International Conference on Learning Representations, pp. 1-13, 2024.
Data is Better Together.
Open Image Pref-
https://huggingfaaee co/ detaastt//attr-ss-ettettttttppp o wwteeerrrrreeeeeeel----ee
erences.
2024.
Dhariwal, P. and Nichol, A. Diffusion models beat GANs image synthesis. In Advances in Neural Information Processing Systems, pp. 8780-8794, 2021.
Impact Statement
This paper presents work whose goal is to advance the field of machine learning in text-to-image generation. The ro- bust generation capability of our method holds the potential to benefit domains such design, data augmentation, and education, fostering interdisciplinary collaboration and in- spiring creative exploration.
Dong, H. Xiong, W., Goyal, D., et al. RAFT: Reward rAnked FineTTuning for generative foundation model alignment. Transactions on Machine Learning Research, 2023.
Esser, P., Kulal, S., Blattmann, A., et al. Scaling rectified flow Transformers for high-resolution image synthesis. In International Conference on Machine Learning, pp. 12606-12633 2024.
References
Ahn, D., Cho, H., Min, J., et al. Self-rectifying diffusion sampling with Perturhed-Attention Guidance. In Euro- pean Conference on Computer Vision, volume 15131 of LNCS, pp. 1-17, 2024.
Fan, Y., Watkins, O., Du, Y., et al. DPOK: Reinforcement learning for fine-tuning text-to-image diffusion models. In Advances in Neural Information Processing Systems, pp. 79858 79885, 2023.
Bansal, A., Chu, H.-M., Schwarzschild, A., Sengupta, S., Goldblum, M., Geiping, J., and Goldstein, T. Universal guidance for diffusion models. In JEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 843-852, 2023.
Ghosh, D., Hajishirzi, H., and Schmidt, L. GENEVAL: object-focused framework for evaluating text-to-image alignment. In Advances in Neural Information Processing Systems, 2023.
Black, K., Janner, M., Du, Y., Kostrikov, I., and Levine, S. Training diffusion models with reinforcement learning. In International Conference on Learning Representations, pp. 1-14, 2024.
Goodfellow, I., Pouget-Abadie, J., Mirza, M., et al. Genera- tive adversarial networks. Communications of the ACM, 63(11):139-11 2020.
Ho, and Salimans, T. Classifier-free diffusion guidance. 1.Xi::207.112998. 2022.
Bradley, R. A. and Terry, M. E. Rank analysis of incom- plete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-344 1952.
Ho, J., Jain, A., and Abbeel, Denoising diffusion prob- abilistic models. In Advances in Neural Information Processing Systems, pp. 6840-6600 2020.
Brooks, T., Holynski, A., and Efros, A. A. Instruct- Pix2Pix: Learning to follow image editing instructions. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18392-18402, 2023.
Hong, S., Lee, G., Jang, W., and Kim, S. Improving sample quality of diffusion models using self-attention guidance. IEEE/CVF International Conference on Computer Vi- sion, pp. 7428 7428-7437 2288-737, 2023.
Brown, T. B., Mann, B., Ryder, N., et al. Language models

---

CHATS: Combining Human-Aligned Optimization and Test-Time Sampling for Text-to-Image Generation
Hu, X., Wang, R., Fang, Y., Fu, B., Cheng, P., and Yu, G. EIlA: Equip diffusion models with LLM for enhanced semantic alignment. arXiv:2403.055135, 2024.
diffusion models. In IEEE/CVP Conference Computer Vision and Pattern Recognition, pp. 14297-14306, 2023.
Nichol, A., Dhariwal, P., Ramesh, A., al. Glide: Towards photorealistic image generation and editing with text- guided diffusion models. In International Conference Machine Learning, pp- 16784- 16804, 2022.
Jaques, N., Gu, S., Bahdanau, D., Hrrmández-Lobattor J. M., Turner, R. E., and Eck, D. Sequence Tutor: Conser- vative fine-tuning of sequence generation models with KL-control. In International Conference Machine Learning, pp. 1645-1654, 2017.
Peebles, W. and Xie, S. Scalable diffusion models with Transformers. In IEEE/CVF International Conference on Computer Vision, pp. 4172-4182, 2023.
Jaques, N., Shen, J. H., Ghandeharioun, A., et al. Human- centric dialog training via offline reinforcement learning. In Conference on Empirical Methods in Natural Lan- guage Processing, pp.
Podell, D., English, Z., Lacey, K., et al. SDXL: Improving latent diffusion models for high-resolution image synthe- sis. In International Conference on Learning Representa- tions, pp. 1-13, 2024.
2020.
Kingma, D. P. and Welling, M. Auto-encoding variational bayes. In International Conference on Learning Repre- sentations, pp. 1-14, 2014.
Rafailov, R., Sharma, A., Mitchell, E.,, Ermon, S., Manning, C. D., and Finn, C. Direct preference optimization: your language model is secretly a reward model. In Advances in Neural Information Processing Systems, pp. 53728- 53741, 2023.
Kirstain, Y., Polyak, A., Singer, U., Matiana, S., Penna, J., and Levy, O. Pick-a-Pic: An open dataset of user preferences for text-to-image generation. In Advances in Neural Information Processing Systems, pp. 36652 36663, 2023a.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10674-10685, 2022.
Kirstain, Y., Polyak, A., Singer, U., Matiana, S., Penna, J., and Levy, O. Pick-a-Pic: An open dataset of user preferences for text-to-image generation. In Advances in Neural Information Processing Systems, pp. 36652- 36663, 2023b.
Shazeer, N. and Stern, M. Adafactor: Adaptive learning rates with sublinear memory cost. In International Con- ference on Machine Learning, pp. 4596-4604, 2018.
Labs, B. F. FLUX.I-dev. https:: https::/hugginggace.
Sohl-Dickstein, J., Weiss, E. A., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequi- librium thermodynamics. In International Conference on Machine Learning, pp. 2256 2265, 2015.
Lee, K., Liu, H., Ryu, M., et al. Aligning text-to-image models using human feedback. 12X11 2023.
Lipman, Y., Chen, R. T., Le, M. Flow matching for generative modeling. In In- ternational Conference on Learning Representations, pp. 1-13, 2023.
H., Nickel, M., and
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. In International Conference Learning Representations, pp. 1-12, 2022.
Liu, X., Gong, C., and Liu, Q. Flow straight and fast: Learning to generate and transfer data with rectified flow. In International Conference on Learning Representations, pp. 1-15, 2023.
Wallace, B., Dang, M., Rafailov, R., et al. Diffusion model alignment using direct preference optimization. In IEEE/CVF Conference Computer Vision and Pat- tern Recognition, pp. 8228-8238, 2024.
Loshchilov, and Hutter, F. Decoupled weight decay reg- ularization. In International Conference on Learning Representations, pp. 1-18, 2019.
Wu, X., Hao, Y., Sun, K., Chen, Y., Zhu, F., Zhao, R., and Li, H. Human Preference Score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. 2023.
Luo, G., Darrell, T., Wang, O., Goldman, D. B., and Holyn- ski, A. Readout Guidance: Learning control from dif- fusion features. In IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 8217-8227, 2024.
Xu, J., Liu, X., Wu, Y., et al. ImageReward: Learning and evaluating human preferences for text-to-image gen- eration. In Advances Neural Information Processing Systems, pp. 15903-15935, 2024.
Meng, C., Rombach, R., Gao, R., Kingma, D., Ermon, S., Ho, J., and Salimans, On distillation of guided
10

---

CHATS: Combining Human-Aligned Optimization and Test-Time Sampling for Image Generation
Yang, K., Tao, J., Lyu, J., et al. Using human feedback to fine-tune diffusion models without any reward model. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8941-8951, 2024.

---

CHATS: Combining Human-Aligned Optimization and Test-Time Sampling for Texa-lo-Image Generation
A. Mathematical Derivations
In this section, we present the mathematical derivation of the training objective for CHATS. First, we identify global optimum the target generative distribution (zplc) by maximizing the Lu-ddvergence-constraini reward score in the framework of reinforcement learning from human feedback (RLHF). Subsequently, we utilize this global optimum derive the training loss for proposed method, ensuring clear and rigorous formulation.
A.1. Global Optimum of Reward Maximization
RLHF refines a generative model (a(o)))) e) by maximizing the scores defined by a reward model r, simultaneously minimizing the KL-diieegeecc between po (z ()) and reference distribution Prer(2olc) as:
max E (2o) e) 55 (po 20lc) [Pee((
(18)
Po
where D {c} is the conditional dataset and is scale scalar. Following Wallace et al. (2024), we have:
max EeeD
E ((0 c) 3I (Poalllla))))
(19)
Pe
pe (zo (c)
max (oppoo (r c) 3 log Peer( zo c)-
(20)
max B 2n mpoo (a (R(zo:T, c) - 3 log (n(2)lc)
(ro )
(21)
po (0(2)))
max B D. R( (200.) c) Blogg
(22)
Pref(zo) c)-
po (o=zorTlo) (c) Prerl (2orr
max E Bow R(zu:r, c) - Blog
(23)
which is equivalent to:
Po Po (20:r (c)
Risggr.e)
min E (p log
log
P6
Pref (20.T(c)))))
(24)
Po (0(
min Rew D P...: -oorr log Po
log Z(c)];
Pref(z0.Tle
/Z(c)
minmm is the regularization term. We define:
where Z(c)= = Ex ccle
p² (zo:T(c) Pref (zarTcel Bizugr.s) /Z(c),
(25)
which is valid distribution since p² (2o:Tlc) for any zo.T and Eoorr p" (zo.Tle) = Therefore Eq, 24 corresponds to:
po(20:ri (c)
min En poor log
log Z(c)]
(26)
Pe
p (= (c)
Pe (o0zoT/e)
min EE-D (B (cootroo))))) log p¹ (z0:T (c)
log [(c)
(27)
P6
min E DKL (po e) 11p" (20:T/c) c)) log [cc)
(28)
This optimization objective shows a global minimum as:
Po = p" (zo:rle) = Pra(Zo:rlel Migggr c)
/Z(c).
(29)
A.2. Training Objective of CHATS
According to Eq. 29, the reward function can be reformulated as:
R(zo.T: log ()
+ 3 log ((c).
(30)
p (re((zo.1(c)
12

---

CHATS: Combining Human-Aligned Optimization and Test-Time Sampling for Texaeto-Image Generation
Hence the r is expressed as:
r c) = (taalla)))))) (o( c)]
(31)
E (prrro( goog P Po(zazr()) + P log 2 Pra((o:rle)
(32)
Since CHATS employs two different models for preferred and dispreferred distributions, parameterized by 0+ and 0_, the loss for each ranked pair can be recast by substituting Eq. 32 into Eq. 9, omitting c, and setting 1 for simplicity, as follows:
po+ (=0-1) + +log po- (20.T) K
((z0= : 20)== E - ) h (A 1-5
log Piet (2O.T)
(33)
Piet (
where K is a variable that depends only on e, and can therefore be treated as a constant during optimization. Consequently, the objective can be simplified as:
log po+ (Z0.r) pret (+077)
Pa- (20:T) pref (ref(#a:T)
C(z : 20 ) logo p (eterltd - (T 1=6
+ log
(34)
Using (( 2o 2o)to approximate (=1 for both and 0_, we have:
po+ (z
+ log P8- (20.T) }]
[(2). 20) = -log o E
1s (e "o ( (log
(35)
(ra(
(
po (2/t1lzz²) 1zt)
Po (2 1 12 )
=olog log a of ملاا (log
log
(36)
Piet (z,-11zz²)
Pret 2/- 12
(T Pat (2 lzt)
Pa (2-=1 1,5 )
= log a [E o ).
+ log
(37)
Pref Pre( l2,))
Pret (- 1/
Pat (z lzt)) + log
Pa (2
2 )
= log a (T (= )rgl= lzt)
Bellog
(38)
Peel (2 1l+t)
Peer (2 26
(- 1 1=5
= -- -loo o TE,E = g(s) (= B - lzt )tt) (log po+ (0t1((z))
Po- (z
+ llog Pref [ 1 2
(39)
Pret (2t- (zt)
x (55)
i0lo 1l77 .4)
(log po- (2 12
pa- (2g || )
S E at xt>o(==] )
log TE
- ~ | :20
1 a
Prer Prer(2 2tt)
+ log
P |,
(40)
2 ~g(=; 1z6
= E + (((()(==)))
log 1ogol-T
B B ~
1- x0))
2, ~o(2; (a0)
( 11,tt ,20)
log q |as ,
+ log qل .26) log (z 1,,,5,,,2)) ,5))) pe- (zū-11²5)
+ log
(41)
po+ (zt 1b+))
Pref(zit 1lzt)
Prot(zū_llz
E x t (at) log TT
e < )=
+ Dxx. (g(= |2 - l1 (=tal=))) lxx (a(=ī1l0t =0 Illptrl Illttr (1
+ kL (g(=5)12 z Ilpa (z 1D (1(25 |4 क
(42)
For diffusion models, minimizing 1Kx ( (1(=,11=,, =0) po (20-1111z)) is equivalent to minimizing ||c <0(2,, 1)1 Accord- ingly, the training loss of CHATS is formulated as:
CHATT (=t ( = -E
= (+=((=)==) log - T t=
"5 ~ 1z0
(43)
+ -||ot-coo cat (at, (,,,,)]==== Ger f)]l1] + 1-111-co-- (0- (z,,t) - -Ger(2 - so(67_0)113]
13

---

CHATS: Combining Human-Aligned Optimization and Test-Time Sampling for Texa-lo-Image Generation
where et,et (O, and ;
= are defined in Eq.
For flow matching models, the minimization of De (q(zz-1 p0) is equivalent to the minimization of - (2, (, t)lla. In this case, the loss function is defined as:
t021() of xxt00== log a [-T( ~a(== =o
FFlow
=0) = -E
(44)
+ 11o Ugl (= t)]l -|11-1100 -1= - vorl t)]l + -110--00 (2r (z], -111111110- - +
where zi" and zi are defined as the linear interpolation between noise ~ (N I) and the data samples =o or following Eq.4 4.
A.3. Convergence Properties of CHATS
Given that DPO is invariant to affine transformations of the reward, for reward: I'' R' c) = a. R[ZO,T, c) + b, the optimal policy becomes (cf. Eq. 29):
p² (=o.T c) Prof (20.7 c)e ((
(45)
('(c)
CHATS decomposes reward of traditional DPO (Eq. 30) into two parts:
Poi (Z c)
R (0-) log (Z c)
R$ (0+) log
(46)
Prof ( c)
Prof 2 c)
with B and log (c) omitted since they are constants for optimization. Defining the effective reward as: RCHATS
Rt (0+) + RtT (0-) the resulting joint optimal distribution becomes:
Pref Pree(Zuur: c)e ReHATS
p" (zeer
cc
(47)
ZCHATS
Under the assumption of L-smoothness and using standard gradient descent, we obtain the recursive inequality:
Cx+1 Lx (V
(48)
which ensures the CHATS loss (Eq. 9) decreases monotonically and converges. Since their combination recovers the same optimal joint distribution as traditional DPO methods (Wallace et al., 2024), CHATS preserves theoretical foundations of DPO.
A.4. Theoretical Foundations of CHATS Sampler
Start from Bayes' rule for a classifier:
p(c 2,) p(ze c)p(c)
(49)
p(2))
since p(c) can be regarded as constant during optimization, CFG defines the guided distribution by raising p(c 2)) with guidance scale s:
p( c) x p(z(l))pp( xx)]".
(50)
Substitute the expression for p(c 2)) and omit p(c) yields:
p c) x p(zz (()())))
(51)
In CHATS, two models are used:
14

---

CHATS: Combining Human-Aligned Optimization and Test-Time Sampling for Tmxae Generation
The preferred model Po+ (Et c).
The dispreferred model Pa (ze e) (with its unconditional form Pa (zt)).
For each model, we can write a classifier-like term via Bayes' rule. For the preferred model:
Pot (c 2x) Po+ (ze c)p(c)
(52)
Po+ (zt)
and for the dispreferred model:
Po- (ze c)p(c)
Pø- (c 21)
(53)
Pg (2)
Assuming Pol (2t) Re po (zt), we combine the two signals by defining a composite log-odds score:
Po (=t le) (z+)
log Po (ze c)
A c)
log
(54)
Pg-
Pg- (2t)
The first term tends to generate features favored by the preferred model while suppressing the background features typically produced by the dispreferred model in its unconditional output (similar CFG), and the second term further accounts for the shift in the output of the dispreferred model when conditioned c, with its impact regulated by scalar In this form, the useful information in Pa- (z: c) is effectively utilized as well.
Following CFG, we define the CHATS guided distribution as:
(2 c) x po+ (2 c)xxp(s (xxp(s-A(zz,c))
(55)
Substitute A(2,, c):
Po(2 c) x Pot (z+ 1e) -e)pepppp (s log Po+ (2x [c) a log
Pg- (2x 1 c)
(56)
pø (2t)
Pe (2t)
Using exp(s log A) A", we have
Pa- (2e [c) ))
"
Po (2e c)
Pa(z1 e) x po+ (21 c)
(57)
Po- (2+)
Po (2c)
Grouping terms, obtain
Po c) Pai (#e c) pp (ze c) (z+)-
(58)
which the same with Eq. 15. The final guided distribution is not merely sharpened version of Po+ (2, c). It also leverages the dispreferred model. The term dispreferred model's 's behavior. This derivation, starting from p(c 22) for both models, provides theoretical foundation for the CHATS sampling distribution analogous that of CFG.
Pe- (zele) as
adjusts the output based on how conditioning on changes the
Pa- (#t)
B. Full Experimental Settings
B.I. Models and Finetuning Datasets
We utilize two model families: diffusion models and flow matching models. For diffusion models, we employ Stable Diffusion 1.5 (SD1.5) (Rombach et al., 2022) and Stable Diffusion XL-1.0 (SDXI.) (Podell al., 2024). For flow matching
15

---

CHATS: Combining Human-Aligned Optimization and Test-Time Sampling for Text-lo-Image Generation
Table 7. Full aesthetic results (1) on "Anime" "Concept-Art".
Model
Method
Anime HPP v2
Concept-Art HPS v2
ImageReward
PickScore
ImageReward
PickScore
Standard
-134.51
19.66
19.46
-121.69
18.11
19.47
SD1.5
Diffusion-DPO
-95.95
21.65
20.13
-89.71
20.12
20.14
CHATS
34.35
27.74
21.38
30.74
26.17
20.80
Standard
94.99
30.20
22.80
90.24
28.38
22.19
SDXL.
Diffusion-DPO
111.73
31.69
23.09
98.26
29.64
22.27
CHATS
122.73
32.96
23.17
107.21
30.94
22.38
Standard Diffusion-DPO
49.23
25.00
20.99
53.89
23.60
20.50
In-house T21
65.49
26.03
21.27
69.82
24.87
20.83
CHATS
112.41
30.41
22.13
114.07
29.87
21.66
models, we deploy In-house T2I, a text-to-image generation model optimized for photorealistic images in e-commerce scenarios, based on the DiT architecture (Peebles & Xie, 2023).
We finetune our models on two preference optimization datasets. The first, Pick-a-Pic (PuP v2) (Kirstain al., 2023a), comprises $51,293 tie-free preference pairs collected through user interactions with SD2.1, SD1.5 variants, and SDXL variants (Rombach al., 2022) under various CFG values. The second, OpenImaggerreferenee (OIP) (Data is Better Together, 2024), is recently released high-quality image preference dataset containing 7,459 preference pairs of 1024x 1024 images generated by SD3.5-large (Esser et al., 2024) and FLUX.1-de (Labs, 2024). Compared to PaP v2, OIP offers superior image quality, including enhanced resolution and textual fidelity. We leverage both datasets to demonstrate the distinct advantages of our proposed method.
B.2. Finetuning Details
Following the approach from Wallace et al. (2024), we employ Adafactor (Shazeer & Stern, 2018) to finetune the SDXL model and A (Loshchilov & Hutter, 2019) for the SD1.5 and In-house T2I models. Training is conducted with an effective batch size of 512, maintaining an image resolution of 1024. The default learning rate set to x 10-5, and learning rate scaling strategy based batch size increases is utilized accelerate the finetuning. T (cf. Eq. 13 and Eq. 14) fixed as 1000.
B.3. Evaluation Prompts
HPS v2. The HPS v2 (Wu al., 2023) comprehensive benchmark for aesthetic evaluation, comprising 3,200 prompts evenly distributed across four distinct styles: "Anime". "Concept-Art", "Paintings", and "Photo" with 800 prompts per style. We select this benchmark due its extensive number of prompts, which ensures the stability of results and strengthens the reliability of the evaluation.
GenEval. GenEval (Ghosh et al., 2023) is an object-centric evaluation framework designed to assess text-to-image models on compositional image attributes, such as object no-occurrence, spatial positioning, quantity, and color. It comprises 553 prompts and typically used in conjunction with object detection models report compositional scores.
DPG-Bench. DPG-Bench (Hu al., 2024) is challenging benchmark consisting of 1,065 lengthy and dense prompts, with each describing multiple objects characterized by wide range of attributes and complex relationships.
B.4. Evaluation Metrics
For aesthetic evaluation, employ three aesthetic evaluators:
HPS v2. HPS v2 (Wu et al., 2023) is built upon CLIP (ViT-H/14) model, finetuned on the Human Preference Datasets v2. This dataset includes 798,090 099000 human preference annotations across 433,760 image pairs, enabling HPS v2 to accurately predict human preferences for images generated from textual prompts.
ImageReward. ImageReward (Xu et al., 2024) is the first general-purport human preference scoring model for text-to-image generation. It trained using a systematic annotation pipeline, incorporating both rating and ranking methodologies, and
16

---

CHATS: Combining Human-Aligned Optimization and Test-Time Sampling for Text-to-Image Generation
Table Full aesthetic results (f) on "Paintings" and "Photo".
Model
Method
Paintings HPS v2
Photo ImageReward
ImageReward
PickScore
HPS v2
PickScore
Standard
-123.31
18.25
19.53
-84.69
20.13
20.19
SD1.5
Diffusion-DPO
-97.24
19.87
20.08
-72.05
20.74
20.47
CHATS
40.53
26.04
20.94
18.71
26.43
21.44
Standard
91.39
28.19
22.31
67.31
26.88
22.26
SDXL.
Diffusion-DPO
108.19
29.83
22,42
82.89
28.34
22.51
CHATS
114.10
31.08
22.55
88.72
29.62
22.61
Standard Diffusion-DPP
53.03
23.18
20.48
73.00
26.09
21.86
In-house T21
70.20
24.71
20.88
80.51
26.56
21.98
CHATS
116.36
29.95
21.54
97.02
29.30
22.17
Table 9, Full aesthetic results (f) on "GenEval" and
Model
Method
GenEval HPS v2
DPG-Bene's HPS v2
ImageReward
PickScore
ImageReward
PickScore
Standard
-11..88
21.20
20.46
-117.39
18.88
19.70
SD1.5
Diffusion-DPO
-95.61
22.04
20.72
-100.05
19.87
20.01
CHATS
-1.774
26.17
21.55
-21.47
24.97
20.79
Standard
55.14
28.19
22.65
48.84
27.38
21.99
SDXL
Diffusion-DRR
79.19
29.51
22.94
57.17
28.71
22.06
CHATS
84.72
29.81
22.96
65.79
29.51
22.08
Standard
27.23
22.13
56,44
25.68
21.08
In-house T2T
Diffusion-DPO
84.61 105.31
27.62
22.24
60.73
26.16
21.21
CHATS
29.84
22.54
89.28
29.53
21.66
has collected over 137,000 expert comparisons.
PickScore. PickScore (Kirstain al., 2023b) is another preference evaluator, finetuned the Pick-a-Pic dataset, large open dataset comprising text-to-image prompts and real user preferences for generated images.
Among these three evaluators, HPS v2 demonstrates the highest correspondence with human annotators. Therefore, we use as the primary evaluator for aesthetic evaluation the main text. In the following section, we provide the complete results from all three evaluators.
C. Full Aesthetic Results
We report the full aesthetic results on Table 7, 8 and 9. Consistent with the findings presented in the main text, our CHATS outperforms two baseline methods across all aesthetic evaluators and all groups of prompts. This further highlights the effectiveness of CHATS.
D. More Ablation on a
In Fig. 5, we present the complete results of the ablation studies a, exploring the impact of varying its values. CHATS consistently outperforms CFG (a 0) across almost all cases, it better incorporates preference information during sampling. Any value of within the range [0.5, 0.7] performs well. Therefore, to demonstrate the generalizability of CHATS, we set a 0.5 as the default choice for the main experiments, although = 0.7 achieves the best performance.
E. More Qualitative Results
We present additional qualitative results in Fig. 6 to 10. As shown in these figures, our CHATS significantly enhances the quality of generated images compared to the standard version of models. Furthermore, in comparison Diffusion-DPo approach shows superior capability in capturing and accurately rendering the intricate details described in the prompts.
17

---

CHATS: Combining Human-Aligned Optimization and Test-Time Sampling for Imxae Generation
SDXL
In-house T21
SDXL
In-house T21
33.02
30.47
30.95
29.95
HPS v2
HPS v2
32.97
30.42
30.92
29.90
32.92
30.37
30.89
29.85
32.87
30.32
30.86
29.80
a
a
(a) Anime
(b) Concept-Art
SDXL
In-house T2I
SDXL
In-house T2I
31.14
30.03
29.8
29.35
31.09
29.7.
29.30
29.98
HPS
31.04
29.93
29.6
29.25
30.99
29.5
29.20
30.94
29.88
29.4
29.15
30.89
29.83
29.3
29.10
a
a
a.
Paintings
(d) Photo
Figure 5. Full ablation results on We report the HPS v2 results Anime, Concept-Art, Paintings and Photo.
18

---

CHATS: Combining Human-Aligned Optimization and Test-Time Sampling for Text-to-Image Generation
Standard
Diffusion-DPo
CHATS
HUG ME!!
Hug Me.
HUG MMe.
Figure 6. More qualitative comparisons. Prompts: 1) Batman is shown working as Bagger at a grocery store. 2) A cat sitting besides rocket planet with a lot of cactuses. 3) West Highland white terrier holding a "Hug me!" sign. 4) A beaver in formal attire stands beside books in a library.
19

---

CHATS: Combining Human-Aligned Optimization and Test-Time Sampling for Text-to-Image Generation
Standard
Diffusion-DPO
CHATS
Figure 7. More qualitative comparisons. Prompts: 1) An anthropomorphic and surreal depiction of artificial intelligence 's seff-image. 2) Portrait of a male furry anthro Blue wolf farsona wearing black cyberpunk clothes city at night while rains. 3) An elderly woman poses for high fashion photoshoot in colorful, patterned clothes with a cyberpunk 2077 vibe. 4) Colorfid scift shanty town with metal roofoops and wooden and concrete walls the style of Studio Ghibli and other anime influences.
20

---

CHATS: Combining Human-Aligned Optimization and Test-Time Sampling for Text-lo-Image Generation
Standard
Diffusion-De -PO
CHATS
Figure 8. More qualitative comparisons. Prompts: 1) Colorful illustration of a forest tunnel illuminated by sunlight and filled with wildflowers. 2) spring landscape painting featuring a treeless mountain village with melting lake ice, winding stone steps, and fog. 3) A digital painting of Teemo from League of Legends, wearing cyborg parts and new skin, in fantasy MMORPG style. 4) A digital painting of a fantasy kitchen environment elements of cartoons, comicx, and manga.
21

---

CHATS: Combining Human-Aligned Optimization and Test-Time Sampling for Text-to-Image Generation
Standard
Diffusion-DPP
CHATS
Figure 9. More qualitative comparisons. Prompts: 1) Solar punk vehicle a bustling city. 2) A neofituristic island city depicted in photo-reate realistic illustration by five artists. 3) Architecture render with pleasing aesthetics. stylized digital image of cherry tree overlooking a valley with a waterfall during sunset.
22

---

CHATS: Combining Human-Aligned Optimization and Test-Time Sampling for Text-to-Image Generation
Standard
Diffusion-DPO
CHATS
Figure 10. More qualitative comparisons. Prompts: 1) The image is a digital art headshot of an owifolk character with high detail and
dramatic lighting. 2) An elderly man is sitting couch. 3) Woman walking down the side walk of busy night city. 4) A view of table with a bunch of cakes and tea on it.
23


Reviews:
[{"review_id": "SRyqm9gPFV", "reviewer": "Reviewer_Nr72", "summary": "This paper aims to improve the performance of text-to-image diffusion models by using a human preference dataset. To make better use of DPO and CFG, they propose a training objective that trains a preferred model and a dispreferred model. During the sampling step, they introduce a new guidance method that incorporates both models. Experimental results show performance improvements on several benchmark datasets.\n\n## Update after rebuttal\nThank you for the authors' response and the additional responses to further comments. After considering the authors' rebuttal and the reviews from other reviewers, I have revised my rating to a borderline accept. This work is meaningful in that it explores DPO applied to diffusion models sampled with CFG. However, this approach requires twice the memory unless a distillation method is applied. The authors have properly presented and analyzed the strengths of the proposed method, but the need for further hyperparameter tuning and the need for a distillation method are still apparent. Therefore, while I lean towards a positive evaluation, my support is not particularly strong.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful feedback and questions! Due to space limitations, we provide responses to your main comments. Further questions can be discussed in subsequent responses.\n### **1. Theoretical foundations  & convergence properties of CHATS**\nGiven that DPO is invariant to affine transformations of the reward, for reward $R'(z_{0:T}, c) = a \\cdot R(z_{0:T}, c) + b$, the optimal policy becomes (cf. Eq.29):\n$$\np^*(z_{0:T}\\mid c) = \\frac{p_{\\mathrm{ref}}(z_{0:T}\\mid c) e^{a \\cdot R(z_{0:T}, c) + b}}{Z'(c)}.\n$$\nwith $\\beta$ omitted for simplicity.\n\nCHATS decomposes reward of traditional DPO (Eq.30) into two parts:\n$$\nR^+(\\theta^+) = \\log\\frac{p_{\\theta^+}(z_{0:T}^+\\mid c)}{p_{\\mathrm{ref}}(z_{0:T}^+\\mid c)}, \\quad\nR^-(\\theta^-) = \\log\\frac{p_{\\theta^-}(z_{0:T}^-\\mid c)}{p_{\\mathrm{ref}}(z_{0:T}^-\\mid c)},\n$$\nwith $\\beta$ and $\\log Z(c)$ omitted since they are constants for optimization. Defining the **effective reward** as: $R_{\\mathrm{CHATS}} = R^+(\\theta^+) + R^-(\\theta^-)$, the **optimal distribution** becomes:\n$$\np^*(z_{0:T}\\mid c) = \\frac{p_{\\mathrm{ref}}(z_{0:T}\\mid c) e^{R_{\\mathrm{CHATS}}}}{Z^{\\text{CHATS}}(c)}.\n$$\n\nUnder the assumption of $L$-smoothness and using standard gradient descent, we obtain the inequality:\n\n$$\n\\mathcal{L}_{k+1} \\le \\mathcal{L}_k - \\frac{\\eta}{2} \\|\\nabla \\mathcal{L}(\\theta^+_k, \\theta^-_k)\\|^2,\n$$\nwhich ensures the CHATS loss (Eq. 9) decreases and converges. Since their combination recovers the same optimal joint distribution as traditional DPO methods [1], CHATS preserves theoretical foundations of DPO.\n\n### **2. Dual training signals**\n\nIn our training procedure, we do not split the data into separate (dis)preferred subsets. Instead, when minimizing losses such as those in Eq. 13 and 14, we select a ranked preference pair $(z_0^+, z_0^-)$ from the entire dataset. This single pair is then used to simultaneously update both the preferred and dispreferred models within a unified dual-objective framework. This joint training approach ensures that both models receive complementary signals derived from the ranked pair.\n\n### **3. Clarification on Eq. 15**\nAs indicated in Line 220–226 (right column), when $0 < \\alpha < \\frac{1+s}{s}$, the dispreferred distribution $p_{\\theta^-}$ is partially incorporated so that it contributes useful patterns while remaining less influential than $p_{\\theta^+}$. In other words, the dispreferred model's noise prediction is toned down, partially suppressing undesirable patterns while preserving beneficial information. Conversely, when $\\alpha < 0$, the terms $p_{\\theta^-}(z_t | c)^{\\alpha s}$ and $p_{\\theta^-}(z_t)^{-(1+\\alpha)s}$ actively push samples away from undesired modes, effectively suppressing the entire output of the dispreferred model, similar to using a null prompt $\\varnothing$.\n\n### **4. Justification on proxy prompt**   \nGenerative models represent prompts in a continuous embedding space where linear operations reflect meaningful semantic changes. For example, word embedding arithmetic [2] (e.g., “queen” ≈ “king” – “man” + “woman”) shows that semantic attributes can be linearly combined. Thus, interpolating between a prompt $c$ and the null prompt $\\varnothing$ (i.e., forming $\\hat{c} = -\\alpha c + (1+\\alpha) \\varnothing$) not only reliably captures the semantic content but also reduces the number of forward passes—from 3 in Eq.16 to 2 in Eq.17, thereby cutting inference cost by about 1/3.\n\n### **5. Memory efficiency**\n\nWhile the dual architectures in CHATS introduce additional cost (see Table 6), as noted in Line 436–439 (right column), this extra inference cost can be **completely eliminated** via distillation. As shown in the table below, by simultaneously distilling the guidance scale (i.e., $s$ in Eq. 17) and the two models into a single one, CHATS achieves both high efficiency and high-quality generation (Model: SDXL).\n|Method|Memory($\\downarrow$) |Throughput ($\\uparrow$)|HPS v2 on Photo($\\uparrow$)|\n|-|-|-|-|\n|Standard|1$\\times$|1$\\times$|26.88\n|CHATS|2$\\times$|0.97$\\times$|29.62\n|CHATS-distill|1$\\times$|2$\\times$| 29.53\n\n### **6. Justification on Table 4**\nEven though a single model trained on the full dataset outperforms the baseline, our CHATS method consistently delivers further improvements as shown in Table 4. Similar trends are observed with SDXL:\n\n|Config|HPS v2 on Photo ($\\uparrow$)|\n|-|-|\n| single model (full data) + $s$=5 | 28.20\n| two models + $s$=5,$\\alpha$=0.5 | 29.62\n\n### **7. More analysis on $\\alpha$**\nWe show more ablation on $\\alpha$ on table below (SDXL, $s$=5):\n|$\\alpha$|HPS v2 on Photo($\\uparrow$)|\n|-|-|\n|0.5 (default)|29.62 |\n|0.0| 29.36|\n|-0.1| 29.34|\n| -0.3| 29.25|\n\nWe observe the best choice of $\\alpha$ occurs around 0.5, consistent with our analysis in Fig.5 and Line 765-769 in appendix.\n\n**References**   \n[1] Diffusion Model Alignment Using Direct Preference Optimization, CVPR'24   \n[2] Distributed Representations of Words and Phrases and their Compositionality, NeurIPS'13", "claims_and_evidence": "The primary claim of the paper is that training and sampling with separate models for preferred and dispreferred outputs improves text-to-image diffusion models aligned with human preferences. The proposed method modifies the DPO objective to accommodate this dual-model setup and introduces a corresponding guidance-based sampling strategy. While the experimental results partially support this claim, I find that the advantages of this approach over existing methods are not clear.\n\n* The modification of the DPO objective to fit the dual-model framework requires further justification. The traditional DPO is designed from reinforcement learning setups where a single policy model represents the reward function. Splitting it into two models raises the question of whether the theoretical foundations of DPO still hold in this new formulation.\n\n* It is also unclear whether the (dis)preferred model only receives training signals from its corresponding (dis)preferred dataset and, if so, whether this setup can truly be considered a dual-objective framework.\n\n* A key concern is to understand the convergence properties of the diffusion model under the proposed objective, i.e., where does the optimal policy of the new formulation converge?\n\n* In the sampling step, the derivation of Eq. (15) requires further clarification. For example, in Eq. (16), what does it mean for $\\epsilon_{\\theta^-}(z_t, ,c)$ to receive a signal in the positive direction? A more intuitive or theoretical explanation of the behavior of this term would strengthen the argument.\n\n* The paper introduces proxy prompts but does not provide clear evidence of their effectiveness. Specifically, it is unclear why linear interpolation remains effective in this setting. The lack of a strong theoretical justification for this design choice weakens the argument for its necessity.\n\n* Training and sampling with two models raises concerns about memory efficiency.", "methods_and_evaluation_criteria": "Similar to my concerns above, it is uncertain whether the proposed dual-model training scheme preserves the theoretical foundations of DPO and whether it leads to a well-defined optimal policy.", "theoretical_claims": "I have reviewed the derivations of RLHF and the training objective of CHATS. However, I am particularly interested in whether the convergence point of the loss function from Eq. (9) is consistent with that of existing RLHF or DPO methods. A clearer discussion of how the loss formulation ensures consistency with established preference learning frameworks would strengthen the paper's claims.", "experimental_designs_or_analyses": "I have reviewed Experiments section of the paper and have the following concerns:\n\n* In Table 4, the ablation study shows that a single model trained on the full dataset already outperforms the baseline. This raises the question of whether the proposed two-model approach provides a significant advantage over a simpler alternative. A clearer rationale is needed to demonstrate the need to train separate preferred and dispreferred models.\n\n* In Figure 3, the paper provides a sensitivity analysis for $\\alpha$, but while the main text discusses cases where $\\alpha$ is negative, the experiments do not include results for negative $\\alpha$. Including this analysis would strengthen the empirical evaluation by providing a more complete picture of the behavior of the method.", "relation_to_broader_scientific_literature": "The paper proposes a new preference optimization approach for diffusion models, building on methods widely used in large language models (LLMs) and other domains. By extending preference optimization to the diffusion model framework, the work introduces a potentially impactful direction for aligning generative models with human preferences. \n\nIf the proposed method demonstrates significant advantages over existing approaches, it could have broader applicability beyond diffusion models, potentially influencing preference optimization strategies in LLMs and other generative models. A stronger discussion on the generalizability of this approach to different architectures would further highlight its relevance to the broader scientific community.", "essential_references_not_discussed": "As far as I know, the essential papers seem to be well-referenced.", "other_strengths_and_weaknesses": "I have discussed in above sections.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "I would like the response to primarily address the issues and questions raised in the Claims and Evidence section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cMJOhANaYu", "reviewer": "Reviewer_yJZL", "summary": "This paper presents CHATS, a framework for text-to-image generation (T2I) that enhances both text-image alignment and generation quality. Unlike traditional approaches that separately apply human preference alignment and classifier-free guidance, CHATS integrates both components to optimize text-to-image diffusion models. The proposed method models both preferred and dispreferred distributions and employs a proxy-prompt-based sampling strategy to leverage useful information from both. CHATS demonstrates data efficiency, achieving good performance with minimal fine-tuning data. Experimental results show that CHATS outperforms existing preference alignment techniques on some evaluation metrics.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments.   \n### **1. Novelty**\n\nWe respectfully disagree with the assertion that CHATS merely *combines existing technologies*. In our work, we explicitly differentiate our approach from prior DPO methods for diffusion models in related work [1–3] in Line 135-146 (left column). In fact, the 3 papers you cited are **evidences that support** our novelty rather than undermining it. The key insight of our CHATS is to integrate human preference optimization finetuning with the sampling process, leveraging their inherent synergy to refine image generation. However, none of DPO methods you mentioned explore it.\n\nMoreover, Null-text Inversion [4] concentrates on image editing by optimizing the null-text embeddings **using gradient updates** to achieve a superior DDIM inversion. In contrast, the proxy-prompt-based sampling strategy employed by CHATS is primarily designed to enhance sampling efficiency by reducing the number of forward passes from 3 in Eq.16 to 2 in Eq.17, while all prompt features remain **frozen and untrained**. Additionally, [4] neither utilizes human preference data nor trains separate models to explicitly capture the distributions of preferred and dispreferred images, and the dual architectures used by CHATS naturally separate these distributions into two distinct parts. As a result, our method is **not simply a combination of existing techniques but rather a novel integration** that leverages human preference data to improve the generative process. To the best of our knowledge, such an approach has **not** been previously explored in the context of text-to-image generation.\n\n\n\n### **2. Clarification on performance improvement**\n\nWe acknowledge that the numerical improvements, especially when comparing SDXL + Diffusion-DPO with CHATS, may appear modest on individual benchmarks. However, we emphasize that the improvements are **consistent** across multiple benchmark evaluations and are observed in both diffusion models (SDXL) and flow matching models (In-house T2I). Our extensive evaluations across aesthetic scores, GenEval, and DPG-Bench consistently demonstrate that CHATS improves aesthetic alignment and generation quality. The consistent gains across diverse datasets and model architectures validate the effectiveness of our approach. Moreover, these improvements are achieved with only a small high-quality fine-tuning dataset, highlighting the data efficiency of CHATS.\n\n### **3. Clarification of deployment cost**\n\nWhile CHATS introduces a dual-model architecture that, if used naively, doubles the mode size and slightly decreases the throughputs compared to a single diffusion model (as shown in Table 6), this cost can be **completely eliminated** through distillation. As noted in Line 436–439 (right column), by simultaneously distilling the guidance scale (i.e., $s$ in Eq. 17) and the two models into a single model, we achieve both high efficiency and high-quality generation. For example, our distillation variant, “CHATS-distill,” attains a memory footprint and throughput comparable to or even exceeding that of the standard model, while retaining the improved HPS v2 score, as demonstrated in the following table:\n\n\n| Method         | Memory ($\\downarrow$) | Throughput ($\\uparrow$) | HPS v2 on Photo ($\\uparrow$) |\n|----------------|--------------------------|---------------------------|-------------------------------|\n| Standard       | 1×                       | 1×                        | 26.88                         |\n| CHATS          | 2×                       | 0.97×                     | 29.62                         |\n| CHATS-distill  | 1×                       | 2×                        | 29.53                         |\n\nThus, while the raw dual-model approach incurs additional computational cost, our results show that a distilled version can match or surpass the efficiency of a single model without compromising the quality gains provided by CHATS.\n\n**References**\n\n[1] Diffusion Model Alignment Using Direct Preference Optimization, CVPR’24.  \n[2] Training Diffusion Models with Reinforcement Learning, ICLR’24.  \n[3] DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models, NeurIPS’23.  \n[4] Null-text Inversion for Editing Real Images using Guided Diffusion Models, CVPR’23.", "claims_and_evidence": "NA", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "NA", "experimental_designs_or_analyses": "- The performance improvement, especially over SDXL + Diffusion-DPO, on Tab 1 - Tab 3 is marginal. The effectiveness of the proposed method may not be significantly verified.", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "[d] Null-text Inversion for Editing Real Images using Guided Diffusion Models. CVPR’23", "other_strengths_and_weaknesses": "Strengths: \n+ A new framework combining RL and guidance. The paper introduces an approach that jointly optimizes human preference alignment and test-time sampling, addressing some limitations in existing text-to-image models.\n+ Data-Efficient Fine-Tuning. CHATS achieves good performance with a small, high-quality fine-tuning dataset, making it more practical and resource-efficient for real-world applications. \n\n\nWeaknesses: \n- Lack of novelty. Human alignment methods for diffusion models [a, b, c] and learning negative/disprefer concepts for text-time sampling [d] have been already proposed. This method seems to simply combine exisiting technoligies. \n\n- The performance improvement, especially over SDXL + Diffusion-DPO, on Tab 1 - Tab 3 is marginal. The effectiveness of the proposed method may not be significantly verified. \n\n- The deployment cost may be double compared with a single diffusion model, due to the introduction of the minus model. Besides, the efficiency can also be influenced as shown in Tab 6. \n\n\n[a] Diffusion Model Alignment Using Direct Preference Optimization. CVPR’24\n[b] Training Diffusion Models with Reinforcement Learning. ICLR’22\n[c] DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models. NeurIPS’23\n[d] Null-text Inversion for Editing Real Images using Guided Diffusion Models. CVPR’23", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vwYvsObDoa", "reviewer": "Reviewer_KEuT", "summary": "The paper introduces CHATS, a text-to-image generation framework that integrates human preference alignment with test-time sampling. It employs two distinct models to capture preferred and dispreferred distributions, trained with a new objective based on Direct Preference Optimization (DPO), and uses a proxy-prompt sampling strategy. The main experimental results claim CHATS outperforms traditional methods like Diffusion-DPO across benchmarks。\n\n## Update after rebuttal\nThanks to the authors for their efforts in their rebuttal. I decided to keep my initial positive rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable and insightful feedback！\n\n### **1. Clarification on approximating an intractable expectation using Jensen’s inequality**\nWe acknowledge the reviewer's concern regarding the use of Jensen’s inequality in our derivation of Eq. 12. Specifically, we approximate the intractable expectation as follows:\n$$\n\\mathcal{L}(z\\_0^+, z\\_0^-) = -\\log \\sigma\\left( \\mathbb{E}[X(z\\_{0:T})] \\right) \\leq -\\mathbb{E} \\left[\\log \\sigma\\left( X(z\\_{0:T}) \\right)\\right],\n$$\nplease refer to Eq.39 and Eq.40 in appendix for more details.\n\nAlthough applying Jensen’s inequality in this manner provides an upper bound rather than the exact loss, our extensive empirical evaluations across multiple benchmarks (e.g., Tables 1–3) indicate that the resulting training dynamics lead to consistent improvements in generation quality and robust convergence. In practice, the bias introduced by this approximation is effectively absorbed during optimization, and the desired preferred/dispreferred split is maintained.\n\nWhile a tighter approximation might further reduce any potential bias, the experimental results confirm that our current approximation does not lead to suboptimal convergence or a misrepresentation of the two distributions. Quantifying the exact impact of this inequality remains challenging due to the inherent complexity of the diffusion process. However, the empirical performance gains observed across diverse datasets and model architectures strongly suggest that the approximation error is minimal.\n\n### **2. Validation on different dataset**\nAlthough the 7,459-pair dataset (OIP) demonstrates our method's high data efficiency, we have also evaluated CHATS on a larger and noisier dataset. Our experiments on the PaP v2 dataset, which comprises 851,293 preference pairs, are reported in Table 5 and confirm that CHATS consistently outperforms baseline methods on both datasets. Furthermore, our results indicate that training with the higher-quality OIP dataset yields better performance, as discussed in \"A small high-quality preference dataset is enough\" (Line 379, right column). Detailed information about the datasets can be found in the appendix, Line 681-687.\n\n### **3. More analysis on sensitivity of $\\alpha$ and $s$**\n\nWe conduct additional experiments to further examine the impact of hyperparameters. The results on SDXL are summarized below.\n\nFor the guidance scale $s$, we obtained the following:\n\n|$s$| HPS v2 on Photo ($\\uparrow$) |\n|-|-|\n|2|28.83|\n|3| 29.36|\n|4| 29.81|\n| 5 (default) | 29.62 |\n| 6 | 29.61|\n| 7 | 29.39|\n| 8 | 28.72|\n\nThese results indicate that while our default value $s=5$ works well, a value of $s=4$ yields a slightly higher score. It is important to note that $s$ is a user-specified hyperparameter that governs the trade-off between concentrating generation on high semantic-density areas and maintaining output diversity. As such, it is not unique to CHATS and was not extensively tuned in our framework.\n\nRegarding $\\alpha$, our experiments yield the following:\n\n| $\\alpha$ | HPS v2 on Photo ($\\uparrow$) |\n|-|-|\n| 0.5 (default) | 29.62 |\n| 0.0        | 29.36 |\n| -0.1       | 29.34 |\n| -0.3       | 29.25 |\n\nThese results confirm that the best performance is achieved around $\\alpha = 0.5$, which is consistent with our analysis presented in Figure 5 and Line 765-769 of the appendix.\n\nIn summary, our ablation studies demonstrate that CHATS is robust to moderate variations in these hyperparameters. The default settings of $s=5$ and $\\alpha=0.5$ represent a high-quality generation, with minor adjustments yielding comparable performance.\n\n### **4. Computational cost**\nWe would like to point out that Table 6 already reports computational cost metrics in terms of images generated per second, comparing CHATS with single-model methods like Diffusion-DPO. In addition, we have performed supplementary experiments to address the increased cost introduced by the dual-model architecture. As explained in Line 436-439 (right column), by simultaneously distilling both the guidance scale (i.e., $s$ in Eq. 17) and the two models into a single model, the extra inference cost can be **completely** eliminated while still achieving high-quality generation. For example, the following table shows that our distillation variant (CHATS-distill) not only reduces memory usage and increases throughput relative to CHATS but also maintains the improved HPS v2 score:\n\n|Method|Memory($\\downarrow$) |Throughput ($\\uparrow$)|HPS v2 on Photo($\\uparrow$)|\n|-|-|-|-|\n|Standard|1$\\times$|1$\\times$|26.88\n|CHATS|2$\\times$|0.97$\\times$|29.62\n|CHATS-distill|1$\\times$|2$\\times$| 29.53\n\nThus, while the dual-model approach in CHATS does introduce additional cost in its raw form, our distillation strategy fully recovers efficiency without compromising the quality of the generated images.", "claims_and_evidence": "The claims of improved performance and data efficiency are supported by experiments on SD1.5, SDXL, and an in-house model, with results in Tables 1-3 showing CHATS outperforming baselines.", "methods_and_evaluation_criteria": "The proposed CHATS method, combining human preference alignment and test-time sampling, makes sense for improving text-to-image generation quality and alignment.", "theoretical_claims": "I reviewed the correctness of the training objective derivation for CHATS in Section A.2 (Appendix A). The logic is sound.", "experimental_designs_or_analyses": "I checked the experimental design in Section 5, focusing on Tables 1-3. The comparison with Standard and Diffusion-DPO baselines across SD1.5, SDXL, and In-house T2I models is valid, and the metrics (HPS v2, ImageReward, PickScore) are well-established.", "relation_to_broader_scientific_literature": "CHATS extends DPO (Rafailov et al., 2023) and CFG (Ho & Salimans, 2022) by integrating preference alignment and sampling.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Weakness:\n* The training objective (Section 4.1, Eq. 12) uses Jensen’s inequality to approximate an intractable expectation, but the paper doesn’t quantify the impact of this simplification. Could this lead to suboptimal convergence or bias in the preferred/dispreferred split?\n* The small dataset size (7,459 pairs) is a strength for efficiency but a weakness for validation. The paper doesn’t test CHATS on larger, messier datasets.\n* The default $\\alpha$=0.5 and $s$=5 work well, but the ablation (Table 4, Fig. 3) suggests that sensitivity isn’t deeply explored.\n* Using two models increases the computing cost compared to single-model methods like Diffusion-DPO. The paper doesn’t report runtime or memory metrics.", "other_comments_or_suggestions": "N/A.", "questions_for_authors": "Please refer to the weakness.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces CHATS, a framework integrating human preference optimization and sampling guidance for t2i models. Reviewers acknowledged the novel integration approach, data efficiency, and performance improvements across several benchmarks. 

Initial concerns regarding theoretical justification were resolved by the authors' rebuttal. The weakness of the increased computational cost (memory, inference time) due to the dual-model architecture was addressed by the author with CHATS-distill. After carefully reading the paper and all reviews, I generally agree with reviewers comments and the discussion outcome. Overall, I think the paper has values in the image diffusion domain. I’d recommend the authors to address the reviewers’ comments in the revision.
Decision: Accept (poster)