Automated Filtering of Human Feedback Data for Aligning Text-to-Image Diffusion Models
Authors: Yongjin Yang
Sihyeon Kim
Hojung Jung
Sangmin Bae
SangMook Kim
Se-Young Yun
Kimin Lee
Venue: ICLR 2025 Poster (2025)
Keywords: Diffusion
Human Feedback
Efficient
Data Filtering
Primary area: generative models
TL;DR: We propose a new data filtering algorithm for efficiently aligning text-to-image diffusion models

Abstract: Fine-tuning text-to-image diffusion models with human feedback is an effective method for aligning model behavior with human intentions. However, this alignment process often suffers from slow convergence due to the large size and noise present in human feedback datasets. In this work, we propose FiFA, a novel automated data filtering algorithm designed to enhance the fine-tuning of diffusion models using human feedback datasets with direct preference optimization (DPO). Specifically, our approach selects data by solving an optimization problem to maximize three components: preference margin, text quality, and text diversity. The concept of preference margin is used to identify samples that are highly informative in addressing the noisy nature of feedback dataset, which is calculated using a proxy reward model. Additionally, we incorporate text quality, assessed by large language models to prevent harmful contents, and consider text diversity through a k-nearest neighbor entropy estimator to improve generalization. Finally, we integrate all these components into an optimization process, with approximating the solution by assigning importance score to each data pair and selecting the most important ones. As a result, our method efficiently filters data automatically, without the need for manual intervention, and can be applied to any large-scale dataset. Experimental results show that FiFA significantly enhances training stability and achieves better performance, being preferred by humans 17% more, while using less than 0.5% of the full data and thus 1% of the GPU hours compared to utilizing full human feedback datasets.

Published as a conference paper at ICLR 2025
AUTOMATED FILTERING OF HUMAN FEEDBACK DATA FOR ALIGNING TEXT- TEXT-TO-IMAGE DIFFUSION MODELS
Yongjin Yang'
Sihyeon Kim
Hojung Jung'
Sangmin Bae'
SangMook Kim
Se-Young Yun- Kimin Lee
KAIST AI
Department of AI, Chungnam National University2
{dyyjkd, sihk, yunseyoung, kiminlee @kaist ac.
ABSTRACT
Fine-tuning text-to-image diffusion models with human feedback is an effective method for aligning model behavior with human intentions. However, this align- ment process often suffers from slow convergence due to the large size and noise present in human feedback datasets. In this work, we propose FiFA, novel au- tomated data filtering algorithm designed to enhance the fine-tuning of diffusion models using human feedback datasets with direct preference optimization (DPO). Specifically, our approach selects data by solving an optimization problem to max- imize three components: preference margin, text quality, and text diversity. The concept of preference margin is used to identify samples that are highly informative in addressing the noisy nature of feedback dataset, which is calculated using a proxy reward model. Additionally, we incorporate text quality, assessed by large language models to prevent harmful contents, and consider text diversity through a k-nearest neighbor entropy estimator to improve generalization. Finally, we integrate all these components into an optimization process, with approximating the solution by assigning importance score each data pair and selecting the most important ones. As a result, our method efficiently filters data automatically, without the need for manual intervention, and can be applied to any large-scale dataset. Experimental results show that FiFA significantly enhances training stability and achieves better performance, being preferred by humans 17% more, while using less than 0.5% of the full data and thus 1% of the GPU hours compared to utilizing full human feedback datasets. Warning: This paper contains offensive contents that may be upsetting.
INTRODUCTION
Large-scale models trained extensive web-scale datasets using diffusion techniques (Ho et al., 2020; Song et al., 2020), such as Stable Diffusion (Rombach et al., 2022). Dall-E (Ramesh et al., 2022), and Imagen (Saharia et al., 2022), have enabled the generation of high-fidelity images from diverse text prompts. However, several failure cases remain, such as difficulties in illustrating text content or incorrect counting (Lee et al., 2023). Fine-tuning text-to-image diffusion models using human feedback has recently emerged as a powerful approach to address this issue (Black et al., 2023; Fan al., 2024; Prabhudesai et al., 2023; Clark et al., 2023). Unlike the conventional optimization strategy of likelihood maximization, this framework first trains reward models using human feedback (Kirstain al., 2024: Wu et al., 2023; Xu et al., 2024) and then fine-tunes the diffusion models maximize reward scores through policy gradient (Fan et al., 2024; Black et al., 2023) or eeward-gradient based techniques (Prabhudesai et al., 2023; Clark et al., 2023). More recently, Diffusion-DPo (Wallace et al., 2023), which directly aligns the model using human feedback without the need for training reward models, has been proposed. This approach enables fine-tuning diffusion models at scale using human feedback, with the additional benefit of leveraging offline datasets more effectively.
However, fine-tuning diffusion models using human feedback requires considerable time and com- putational resources. For instance, even with the relatively efficient Diffusion-DDP (Wallace et al.,
"quual contribution
I Corresponning authors

---

Published as a conference paper at ICLR 2025
-0- DPO + Full
-O- DPO + FIFA
DPO Full
DPO FIFA
DPO Full
DPO FIFA
6.27 GPU (h)
FREE CANDS
0.216
-0-00
FRREEE CANDY
PickScore
0.214
22.48 GPU
0.212
"A giant monster fighting máni monster"
"A Sign that says: Free Candy'
SD 5 DPO Checkpoint
0.210
0.208
200
800 Training Steps
400
600
1000
"A car foon of man pointing forward"
"Three trucks piled on top of each"
(a) PickScore by each training step
(b) Qualitative examples of generated images
Figure I: (a) PickScore (Kirstain et al., 2024) at each training step of the SD1.5 model using data filtered with FiFA, which uses 0.5% of the data, compared to the model trained with full dataset. Our method significantly outperforms the alternative, converging faster while requiring about 4x fewer GPU hours to match the performance of the SD1.5-DPO released checkpoint". (b) Qualitative evaluation of training the full data and data selected with FiFA for various prompts.
2023), it still takes more than thousands of GPU hours to fully fine-tune SDXL model Podell et al. (2023) on the large-scale Pick-a-Pe -Pic v2 dataset (Kirstain et al., 2024). This is attributed to the multiple denoising processes involved in diffusion models, as large diffusion models must be trained on multiple timesteps (Fan et al., 2024; Prabhudesai et al., 2023; Clark et al., 2023). Moreover, the noisy nature of feedback dataset slows down the convergence speed by making it harder for the model to accurately fit user preferences (Yang et al., 2023b; Chowdhury et al., 2024).
Some model-centric approaches such model pruning (Fang et al., 2023; Ganjdanesh et al., 2024), which reduces the size of diffusion models, or alternative scheduling techniques (Luo et al., 2023), utilized improve efficiency. However, the nature
which aim reduce the number of timesteps, of large-scale feedback datasets diminishes the impact of these methods, as the training cost increases dramatically with the growth of both model size and dataset, despite improvements in efficiency per data point. Dai et al. (2023) have attempted to create smaller datasets manually to reduce the need for large-scall scale data, but this approach requires significant human effort and not applicable for reducing the size of large-scale feedback datasets, which may serve different purposes. This highlights the need for an automated approach to extract meaningful subsets from large-scale feedback data.
In this paper, we propose novel automated Filtering framework that selectively integrates human Feedback, designed for efficient Alignment of diffusion models (FiFA). We frame the filtering task as optimization problem, aiming to find subset that maximizes the three components; (I) preference margin, (2) text quality, and (3) text diversity. A key component of our optimization is selecting data pairs that more informative, determined by their preference margins, which are calculated using a proxy reward model. Specifically, training pairs with a low preference margin can be considered noisy and ambiguous data, their preferences may easily flip, thereby hindering the training process (Chowdhury et al., 2024; Yang et al., 2023b; Rosset et al., 2024). Furthermore, to address the concerns on harmfulness problems and coverage of selected subset induced by relying only on preference margin, we also consider the quality and diversity of the text prompts in the objective function. We assess text quality using Large Language Model (LLM),, following Sachdeva et al. (2024), and measure text diversity by calculating the entropy of embedded text prompts (Zheng al., 2020) approximated using a k-nearest neighbor estimator (Singh et al., 2003). To integrate all these components, we define an objective function that combines the three metrics into a single optimization problem. Additionally, to improve efficiency, we approximate the solution by assigning data importance score for each data pair, making FiFA efficient and applicable to large-scale datasets through an automated process.
Phttps://huge ng face. co htt.g//ggggga..ccccc///////////o w.l..ddell..immeeeeeeee -5-texx2image-vl
2

---

Published as a conference paper at ICLR 2025
In our experiments with open-sourced text-to-image diffusion models Stable Diffusion 1.5 (SD1.5)
and Stable Diffusion XL (SDXL) (Podell et al., 2023), FiFA significantly improves training efficiency compared fine-tuning with full datasets. As shown in Figure 1a, by using only 0.5% of the full Pick-a-Pic v2 dataset (Kirstain et al., 2024), the SD1.5 model trained using FiFA demonstrates a significantly faster increase in PickScore (Kirstain et al., 2024) than the SD1.5 model trained on the full dataset. Moreover, the SDXL model trained using FiFA is preferred 17% more than the model trained with the full dataset by human annotators when evaluated on the HPSv2 benchmark (Wu et al., 2023), with the preferred images showing better text- alignment and higher quality. as illustrated in Figure lb. We remark that this achieved while requiring less than 1% of the GPU hours. Additionally, FiFA reduces harmfulness by more than 50% for neutral prompts compared to using full dataset, by prioritizing text quality. Overall, our proposed FiFA allows large text-to-image diffusion models to be efficiently trained on a small yet important dataset, while showing better efficacy with high image quality.
PRELIMINARIES
Diffusion Models Diffusion models (Ho et al., 2020) are probabilistic models that aim learn data distribution p(x) by performing multiple denoising steps starting from Gaussian noise. The diffusion process consists of two parts, forward process and backward process.
In the forward process, noise is progressively injected at each timestep t according to q(x, ~ ((Vā,, (1 L)) where the noise schedule a is monotonically decreasing function and āj == II. o as- The neural network is trained to learn the denoising process with the following objective:
L (0) = Ex [A([)]le 60 1))
where X(t) is determined by the noise schedule and e a Gaussian noise. During generation, the diffusion model takes reverse denoising steps starting from a random Gaussian noise.
The conditional diffusion model (Rombach al., 2022), such as a text-to-image diffusion model, aims to learn the data distribution p(x(c), trained using the conditional error e(x, c) instead of the unconditional error c(x).
Reward Learning in Text-to-Image Domains Using human preference data, the goal of reward learning is to train a proxy function aligned with human preferences. In text-to-image domains, given textual condition and the generated image xo from that condition, with xō as a "winning" sample and x with a "losing sample", that satisfy x > x/lc. Using the Bradley-Terry (BT) model, can formulate maximum likelihood loss for binary classification to learn the reward model r, parameterized by Φ, as follows:
assume ranked pair
LeT -E Eexx x [log (ra(c, x(") ra x())]
where a is sigmoid function, is text prompt, and image pairs x5" and x labeled by humans.
Direct Preference Optimization for Diffusion Models Direct Preference Optimiza- tion (DPO) (Rafailov et al., 2024) is an approach to align the model using human feedback without training separate reward model. Directly applying DPO loss to diffusion models is not feasible, as an image is generated through the trajectory Xy:0 where T denotes the number of denoising steps, and obtaining the probability of this entire trajectory is generally intractable. Following Diffusion-DPO (ww.loolllaaaaaaaaaaoa (Wallace et al., 2023), the DPO loss for diffusion models can be approximated as follows:
(0) = -B,,,x x logo (-BTw(>)) |||E" -6 1)12 ||E" Exer(x)) .))2 ||e' cx(x + |e' (x, 111))
(3)
where w(Ar) is a weight function typically set to a constant, x"", x are the noised inputs of winning and losing images at timestep respectively, e" ~ (O, represent the Gaussian noise for the winning and losing images respectively, and e is pretrained diffusion model. Detailed derivation of DPO loss for diffusion is presented at Appendix C.
3

---

Published as a conference paper at ICLR 2025
"A cat knight, detailed armor, sliver" Win
"A rainbow-colored dragon in flight"
High Margin
30+5
Number of Data
2e+5
16+5
Low Margin
De+5
0
0.02 0.04
0.06 0.08 0.1
Margin
(a) Samples of high/low preference margin
(b) Distribution of reward margin
Figure 2: (a) Qualitative analysis preference margin estimated through PickScore reward model.
(b) Distribution of PickScore reward margins of Pick-a-Pic v2 train set.
METHODS
To address the inefficiency and noise feedback datasets, we propose FiFA, which automatically filters the full human feedback data to obtain subset for efficiently fine-tuning text-to-imane models. Specifically, our method leverages preference margin as key component to rapidly increase the reward value, while also considering the quality and diversity of the text prompts mitigate harmfulness and ensure robustness. Additionally, we frame the task as optimization problem to find the best subset that maximize these three components, resulting in automated filtering framework applicable to any large-scale dataset.
3.1 PREFERENCE MARGIN
The noisy and ambiguous nature of human preference datasets has been well explored, where labeled preference does not reflect the true preference and may contain spurious correlations (Yang al., 2023b; Chowdhury al., 2024). This can be especially problematic for the efficient fine- tuning of diffusion models, as such noisy data slow down training and reduce the generalization capability (Zhang et al., 2021). Inspired by recent papers that highlight the importance of clean preference pairs (Yang al., 2023b; Rosset al., 2024), more efficient and effective fine-tuning of diffusion models to alleviate this issue.
the preference margin to enable
To estimate the preference margin between the winning and losing images, utilize a proxy reward model re trained on the full feedback dataset using the BT modeling approach with Eq. (2). This process does not pose efficiency concerns for text-image domains, training reward model using CLIP (Radford et al., 2021) 1 or BLIP architectures (Li et al., 2022) demands significantly less time than training large diffusion models, and open-sourced text-image reward models like PickScore (Kirstain et al., 2024) and HPSv2 (Wu et al., 2023), trained on human feedback datasets, can also be utilized.
Figure 2a shows samples with different preference margins estimated using the PickScore proxy model. For example, given the prompt "A cat knight...", image pairs with high reward margin are more distinct, clearly showing that the image including "cat" should be preferred over the one without it. On the other hand, for the low margin pairs, both images usually differ only in style, implying that preferences can be flipped depending on the annotator. This demonstrates that selecting pairs with clear distinctions offer more informative preferences based on the prompt.
Additionally, Figure 2b demonstrates that most pairs in the large-scale Pick-a-Pic dataset are concentrated in a low-margin region. Fine-tuning diffusion models primarily on these low-margin samples can lead slow convergence, as it offers limited benefit from the perspective of G-optimal design, which will be further elaborated in Section 3.3.
3.2 TEXT QUALITY AND DIVERSITY
While reward margin is a critical component, relying solely on the reward margin may overlook two critical factors: the quality and diversity of text prompts.
4

---

Objective Function Our objective function should consider the preference margin, text quality and text diversity. The first, preference margin m reward calculated using the trained proxy reward model. Specifically, given each data pair {c, x x/], calculate the reward margin m roward as Published as a conference paper at ICLR 2025
"A beautiful woman" 'A woman standing"
"A hot girl"
[Tex Quality
Text Diversity 1
DPO Pretrain
Subset
LLM Score
Word Sem. Sing.
Full Dataset
N/A
6.81
N/A
8.05
0.56
7.47
High Margin
0.0
5.71
0.0
7.18
0.63
7.03
0.1
6.55
0.1
7.37
0.65
7.18
FiFA
0.5
7.84
0.5
7.46
0.68
7.30
1.0
8.30
7.56
0.74 7.48
(a) Samples of harmful images
(b) L.M scores and diversity scores of different subsets.
Figure 3: (a) Examples of harmful outputs when training with the full Pick-a-Pic v2 dataset without considering the quality of text prompts. (b) LLM score and diversity measures of text prompts from subsets of the full Pick-a-Pic v2 dataset using three metrics: word entropy (calculating the entropy of words), semantic diversity (measuring average cosine similarity of embedded text prompts), and singular entropy (entropy the singular values of the embedded text matrix). When modifying either or y, the other value is fixed at O.
Text Quality The text prompts in human feedback datasets created by real users tend to be of low quality due unformatted structures, typos, and duplicated content. More importantly, these prompts may include harmful components, such as sexual content, bias, or violence. Figure 3a demonstrates the potential harm caused by naively using all open-sourced Pick-a-Pic v2 dataset for fine-tuning, motivating to ensure that the system takes the quality of the text into consideration.
To estimate this text quality, inspired by ASK-LLM (Sachdeva al., 2024), we evaluate the text quality using LLM. Specifically, we ask the LLM to evaluate whether the text prompt is clearly formatted, understandable, of appropriate difficulty, and free of harmful content by providing a score. We scale the scores from 0 to 10, denoted as the I.IM score. In our experiments, we use OpenAl gpt-3. 5-turbo-01111 0100 25 model. A detailed explanation on LLM score available Appendix D.
Text Diversity The problem with our selection method is that image pairs with a high preference margin may be focused certain prompts families of prompts. This is supported by Figure 3b, as relying on high-margin prompts leads to a decrease in most diversity metrics, such as word entropy, compared to using the full dataset. The lack of diversity may limit generalization capability. Therefore, we also consider text diversity during data filtering.
To estimate text diversity, we employ the entropy of the embedded text prompts (Zheng et al., 2020). Specifically, let C' denote a random variable with a probability density function p. representing a distri- bution of a selected subset of text prompts from the full dataset D, where each text prompt is embedded in IRd space. Text diversity is then estimated through H(C), where H(C) = -Ee-ple) [log p(c)]-
Additionally, set the hard constraint on the selected number of pairs for each text prompt, which is set to 5 and doubled if K is not met, to prevent the selection of large number of duplicate prompts.
3.3 AUTOMATED DATA SELECTION WITH OBJECTIVE FUNCTION
Given the components for data importance, the remaining challenge is how to incorporate all components into an automated data filtering framework that could be applied to any dataset. To achieve this, we formulate data selection as an optimization problem find the subset with high margin, text quality, and diversity. The pseudocode for our algorithm is presented in Algorithm 1.
follows:
m neward (reward(c,xf x." x()) = lr((e,x)) ro (c.c))))))
(4)
5

---

Published as a conference paper at ICLR 2025
Algorithm 1: Algorithm for FiFA
1: Input: Initial dataset D (cx, x 0,, i=1: LLM model for scoring LLM Score(-). Reward
)
model (t)),,, Hyperparameters for quality and diversity y, Number of filtered data points K
2: Output: Filtered dataset S = x",, 1k
3: S 3 ( // Initialize the filtered dataset empty
4: for each data point (c,,
in do (c,,)xx))) // Calculate the reward margin for xj, xb,))
5: metword < |r,(c,,
each data point
6: f(c, xX, xb,,) & mewward +a LLM Score(c.) t log l|c, - c/
// Compute the data importance score for each data point
7: end for
8: Sort data points in D by f(c.; xū, xb,) in descending order.
9: Select the top K data points based on f to form S.
10: return
where e is text prompt. Then, we use the LLM score to evaluate the quality of the text prompts and text entropy H(C) measure diversity, explained in Section Combining all of these components, our goal is to find the subset S that maximizes the following objective function f:
f(S) = Σ
m roward (c, x xx) +a LLM L.MMSScor(()] + H(C),
where a and 7y are hyperparameters for balancing the three components. Unlike the other terms, calculating HC(C) is infeasible. To address this issue, estimate the entropy value using k-nearest neighbor entropy estimator (Singh et al., 2003). Specifically, H(C) can be approximated as follows:
H(C)
log ||e
c KNNN 1|2,
(6)
N
where No is the number of text prompts, and NN-N is the K--N of within prompt set {c), Although Eq. 6 enables the calculation of HCC'), finding an optimal set of that maximizes this function is not feasible for large-scale datasets. Therefore, to efficiently select data, we approximate the function by calculating log |[: ef-NN |l over the entire set of prompts and use this as an estimator of the diversity score for each data pair. The final objective function f that represents the data importance score for each data point then formulated as follows:
f(c, x x)) reward (c,, xXI x)) + L.L. Score(c) + log ||c c cNNNN
Using this objective function, with determined based on the computational burden, as formulated below:
select data by choosing the top K data that have high f value,
argmax X, (cxx)xx))X
Σ
f(c, x8' x))
(8)
As shown in Figure 3b, by increasing a and y, the subset selected with FiFA achieves higher LLM scores and diversity scores, proving that this approximated objective function empirically works. The analysis of the selected and filtered samples using FiFA is presented in Appendix
Interpretation of FiFA Our method, which considers both diversity and a high preference margin, connected to the theoretical interpretation related to G-optimal design (Pukelsheim, 2006). Here, establish this connection through the following theorem under linear reward model assumption.
Theorem 1. Denoting (c(c) == o(x,,, c) ((x e) with feature vector 6. Define g as:
g(m) max (i.c)
where V(T) = = (() is the design matrix with π (1,c) + [0, being a prob- ability distribution. Assume (c) (c(c) 0, + T where O, is an unknown parameter and n is

---

Published as a conference paper at ICLR 2025
Table 1: Comparison of our methods and baselines trained on Pick-a-Pic v2 and HPSv2 datasets, filtered using their respective reward models, PickScore (PS) and HPSv2 reward (HPS). Pretrain denotes the pretrained model, and Full denotes using the full trainset. PS and HPS values are multipled by 100 for displaying. GPU hour is based on NVIDIA A6000 GPU. AE represents Aesthetic Score.
Number
Pick-a-Pic test
PartiPrompt
HPSv2 benchmark
Trainset
Models
Methods
GPU (h)
Pairs
Captions
PS
HPS
AE
PS
HPS
AE
PS
HPS
AE
Pretrain
N/A
N/A
N/A
20.82
26.26
5.32
21.43
26.60
5.17
20.79
26.76
5.29
SD1.5
DPO Full
56.2
850k
59k
21.19
26.37
5.42
21.68
26.82
5.22
21.23
27.09
5.44
DPO FIFA
13.6
5k
2k
21.64
26.95
5.52
22.06
27.43
5.35
21.84
27.84
5.59
Pick
Pretrain
N/A
N/A
N/A
22.23
26.85
5.83
22.56
27.24
5.56
22.71
27.63
5.92
SDXL
DPO Full
1760.4
850k
59k
22.73
27.32
5.82
22.96
27.67
5.61
23.10
28.09
5.92
DPO FiFA
18.3
5k
2k
22.76
27.42
5.89
22,97
27.78
5.66
23.17
28.18
5.94
Pretruin
N/A
N/A
N/A
20.82
26.11
5.32
21.39
26.59
5.17
20.79
26.76
5.29
SDL.5
DPO Full
52.4
645k
104k
20.91
26.46
5.33
21.45
26.87
5.14
21.05
27.19
5.28
DPO FiFA
12.5
5k
3k
20.90
27.03
5,40
21.44
27.43
5.19
20.98
27.91
5.41
HPSv2
Pretrain
N/A
N/A
N/A
22.28
26.85
5.83 22.54
27.23
5,56
22.76
27.63
5.92
SDXL.
DPO Full
1640.4
645k
104k
22.32
26.98
5.84
22.58
27.39
5.61
22,80
27.81
5.92
DPO FIFA
17.2
5k
3k
22.24
27.26
5.93
22.51
27.61
5811 22.75 28.19
6.04
a random noise sampled from 1-subgaussian. Then, with niii, = p ( log 31 number of samplex, one can obtain following error bound on the model prediction O with probability 1 5
ex
(0 02, delc)) S.
(10)
The theorem suggests that minimizing model prediction error can be achieved by increasing the smallest singular value of the design matrix V (TT), which guaranteed by collecting samples with diverse feature vectors. This supports intuition on considering text diversity in FiFA Moreover, selecting high reward margin pairs is likely to increase the norm of Φ; (c)),, which, in turn, can increase the singular values of the design matrix V(T). This can reduce g(T) in Eq. (9), thereby requiring fewer samples for desired level prediction performance. Further explanation, including more details and proofs, is provided in Appendix J.
EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
Dataset We use the popular Pick-a-Pic v2 dataset (Kirstain et al., 2024) and the HPS v2 dataset (Wu et al., 2023) for training in our main experiments. For ablation and further analysis, mainly use models trained on the Pick-a-Pic v2 dataset. We primarily use the Pick-a-Pic test set for evaluation. To ensure safety, manually filter out some harmful text prompts from these test prompts, resulting in 446 unique prompts. Moreover, to test the ability of the model to generalize across diverse prompts, we utilize text prompts from PartiPrompt (Yu al., 2022), which consist of 1630 prompts, and the HPSv2 benchmark (Wu et al., 2023), which consists of 3200 prompts with diverse concepts.
Evaluation We automatically measure performance using PickScore (Kirstain et al., 2024) and HPSv2 Reward (Wu al., 2023), our aim to rapidly enhance the reward through DPO training. To also assess image-only quality, we additionally utilize the LAION Aesthetic Score (Schuhmann et al., 2022). To validate the efficiency of each method, we calculate the GPU hours using an NVIDIA A6000 GPU, including the time required to calculate rewards and LLM scores for FiFA
Moreover, to validate the results against real human preferences, we conduct human evaluation using the HPSv2 benchmark, which includes four concepts: photo, paintings, anime, and concept art. Specifically, we randomly select 100 prompts for each concept in the HPSv2 benchmark, totaling 400 prompts. For each prompt, we assign three annotators and ask them three questions: 1) Overall Quality (General Preference). 2) Image-only Preference, and 3) Text- Image Alignment, following Wallace al. (2023). A detailed explanation of the human evaluation is presented in Appendix K.
7

---

Published as a conference paper at ICLR 2025
DPO + FIFA
DPO +
Tie
Overall Quality
54.8
37.0
Aesthetic
54.3
8.5
37.2
T-I Alignment
46.3
22.5
31.2
10
20
30
40
50
60
70
80
90
100
Win Rate
Figure 4: Human evaluation results. We compare SDXL trained with FiFA against SDXL trained the full dataset using the HPSv2 benchmark. The SDXL model with FiFA consistently outperforms the SDXL model with the full dataset in terms both aesthetic quality and text-image alignment, leading to superior overall quality.
Pretrain
DPO + Full
DPO FiFA
Pretrain
DPO Full
DPO FiFA
pineapple bag designed llya "Superman with Danny DeVitos Face
head and shoulders portrait black cartoon rabbit wearing shirt and laughing with the style Walt Dinney animation"
"A girl gazes ot a city from mountain at night in colored illustration Diego Facio"
Figure 5: Samples from the HPSv2 benchmark, generated using pretrained model, the model trained on the full dataset (DPO Full), and the model trained using FiFA (DPO FiFA). Images from the DPO+FiFA model show better alignment to the prompts and higher quality than the others.
For the other ablation studies, we present performance on Pick-a-Pic test prompts, evaluated auto- matically using PickScore. For the Pick-a-Pic test set, we generate four images for each prompt, while for the PartiPrompt and HPSv2 benchmarks, we generate one image per prompt. calculate the performance, we take the average of the PickScore, HPSv2 Score, and Aesthetic Score for each prompt. Results with different statistical measures are available in the Appendix F.
Implementation Details We utilize PickScore (Kirstain et al., 2024) for the Pick-a-Pic v2 dataset and HPSv2 (Wu et al., 2023) for the HPS v2 trainset as proxy reward models, as they have been trained on their respective full trainsets. our experiments, we set both a and y to 0.5. For training the model with the full dataset, we follow the settings of the original paper (Wallace al., 2023). Specifically, we train SD1.5 with a learning rate le-8 and an effective batch size of 2048. To compare SDXL models with the model trained the full dataset, use the released checkpoint of the Hugging Face SDXL--PPO. When training our models with fewer data we set 3 to 5000 and have effective batch size of 128. Additionally, we use learning rate of 1e-7 for SD1.5 and 2e-8 for SDXL. In our main experiments, we train SDI.5 for 1000 steps and SDXL for 100 steps. For the ablation studies, we mainly utilize SD1.5. More details are presented in Appendix and Appendix G.
4.2 MAIN RESULTS
Quantitative Results Table demonstrates the performance our methods compared to the baselines across three different reward models. Our method, requiring only 20% of the training time SD1.5 and less than 1% of the training time for SDXL, consistently outperforms the trained models that the full dataset for most metries all benchmarks, especially SD1.5. The performance increase in both train sets indicate that FiFA is generalizable across different datasets.
Phttps://hught face. co/ wttp://wgg/aca/////////c/aa/.ttt /ppo 1dx1 teetttttetttggeeeeee
8

---

Published as a conference paper at ICLR 2025
0.216
0.216
0.214
PickScore
PickSpare
PickScare
0.214
0.212
0.210
0.210
0.208
0
500
10000
0.208
Ours
HR
0.206
Oure
T0
1000
20000
50000
0.100
200
400
0
200
400
600
800
1000
500
5000
50000
Training Steps
Training Steps
Training Examples
(a) Comparison with vanilla pruning (b) Component analysis of FiFA
(c) Ablation on data number K
Figure 6: (a) Comparison of FiFA with vanilla pruning baselines of coreset (CS), loss (L), random (R). and high reward (HR) based filtering methods. (b) Component analysis of FiFA by comparing with data selection based only on high/low reward margin (HM. LM), text quality (TQ), text diversity (TD), and random selection (R). (c) Ablation on the number of pairs for FiFA
Notably, the increases in the Aesthetic Score, addition to human preference rewards, indicate that the models trained using FiFA robustly enhance image quality.
Moreover, FiFA achieves high scores on PartiPrompt, which tests various compositions, along with the HPSv2 benchmark, featuring diverse prompts from various concepts and domains. This demonstrates that our model, trained with the small dataset obtained from F FiFA, can generalize well across a wide range of prompts from different domains and styles.
Human Evaluation We also conduct a human evaluation to see if this higher reward actually leads to better human preference. As shown in Figure 4, human annotators prefer the images from SDXL with FiFA 54.8% of the time, compared to 37.0% for the model trained on the full dataset, in terms of overall quality. Moreover, our model outperformed the full dataset model by 17% in aesthetics and 15% in text-image alignment, indicating better visual appeal and alignment for humans.
Qualitative Comparison Figure 5 shows the images generated by the SDXL model trained using the full dataset and the model trained using FiFA, One can clearly see that the model trained on the full dataset and the pretrained model sometimes fail on certain words, highlighted in bold, by missing objects or counts. In contrast, our models consistently follow the text prompts and provide better details. More examples are presented in Appendix N.
4.3 ADDITIONAL EXPERIMENTS
Comparison with Vanilla Pruning Methods In this section, we compare FiFA with multiple baselines, including traditional data pruning techniques of coreset selection (Mirzasoleiman et al., 2020) (CS) using CLIP embeddings, error score based selections (L) (Paul et al., 2021) using DPO loss, and random selection (R). We also compare it with the baselines of naively utilizing samples with high rewards (HR) of winning images. As shown in Figure 6a, FiFA outperforms all the baselines, demonstrating its effectiveness. Specifically, traditional baselines (CS and L) perform poorly as they are not designed for diffusion model alignment while requiring more filtering time. Random or absolute reward-bar based filtering also underperforms, showing that smaller datasets alone do not ensure efficient training, underscoring the value of our method.
Analysis of Each Component We analyze each component of FiFA; preference margin (high (HIM) and low (LM)), text quality (TQ), and text diversity (TD). Figure 6b shows that the high reward margin is crucial, as its removal significantly reduces performance, and a low margin leads to the worst score. Although text quality and diversity alone do not have specific impacts, when combined with high margin, they outperform the other baselines, suggesting that sacrificing some margin for higher text diversity and quality could slightly boost performance while providing additional benefits.
Ablation on the Number of Data K We conduct ablation study the number of data points, K, selected using FiFA. As depicted in Figure 6c, performance increases with K but decreases when K exceeds 5, 000. The lower performance with smaller datasets occurs because such datasets lack

---

Published as a conference paper at ICLR 2025
Vincent Van Cogh
0.218
PlokSoore
Harmiu Image Rate (%)
Full
0,214
0,212
40-
1.5
Abstract Cubiom
1.0
20
0.0
0.5
02
05
FIFA Full
P2
1.0
Promots
(a) Ablation on and
(b) Assessing harmful images
(c) Impact text diversity
Figure 7: (a) Ablation on hyperparameters a and y. (b) Comparison of harmful images generated by the models trained using different methods with three prompts: "a woman standing" (P!), "a beautiful woman" (P2), and "a hot girl" (P3). (c) Analysis of samples generated by models trained on subsets that either ignore or consider text diversity across different artistic styles.
diversity, leading to overfitting on too few prompts and images. In contrast, a larger dataset might include data with low margins or poor prompt quality, which are less informative and can degrade overall performance. The results may depend on the original dataset, as including more data will be beneficial if the original set is of high quality.
4.4 ABLATION ON TEXT QUALITY AND DIVERSITY
Ablation on a and Here, we explore how different a and values, that control the effects of text quality and diversity, affect the performance of trained models. The results are illustrated in Figure 7a. Since a preference margin is used in all configurations, performance remains high compared to the model trained on the full dataset, demonstrating FiFA's robustness hyperparameters. However, extremely high or low and values are ineffective, either reducing the total margin or compromising text quality and diversity. An range of 0.1-1.0 and a y range of 0.5- ensure effective alignment, with the optimal configuration of (0.5, 0.5) marked in red that also works well for HPSv2 dataset.
Can FiFA Reduce Harmful Contents? this section, we evaluate whether FiFA can prevent models from generating harmful images by considering text quality. To estimate the harmfulness, we generated 200 images from three neutral prompts about "woman" and "girl" and manually labeled the harmfulness of each image (see Appendix E for more details). As shown in Figure 7b, when fine-tuned on the entire Pick-a-Pic v2 dataset, the harmfulness of images generated by the fine-tuned model increases significantly, showing least 30% increase for all three prompts compared to those produced by the pretrained model. This clearly demonstrates that RLHF on large-scale human datasets does always align model's 's behaviors with human value. contrast, images generated by the fine-tuned model using F FiFA demonstrate reduced levels of harmfulness compared to the pretrained model, indicating that FiFA effectively enhances model safety.
Impact of Text Diversity To demonstrate the importance of text diversity, we compare samples generated by models trained on subsets of the Pick-a-Pic dataset that either consider only high margin with text quality also include text diversity, using prompts including "raccoon" with different artistic styles. As shown in Figure 7c, incorporating diversity Ieads to a better understanding of concepts like "Vincent Gogh" and "abstract cubism" compared models trained without diversity. This demonstrates that adding text diversity improves the generalization of trained models.
5 CONCLUSION
In this paper, we propose FiFA. a new automated data filtering approach to efficiently and effectively fine-tune diffusion models using human feedback data with DPO objective. Our approach involves selecting data by solving an optimization problem that maximize preference margin, which is calculated by proxy reward model, text quality and text diversity. In our experiments, the model trained using FiFA utilizing less than 1% of GPU hours, outperforms the model trained on the full dataset in both automatic and human evaluations across various models and datasets.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
We include the pseudocode of FiFA in Algorithm 1. Also, we provide the implementation details such hyperparameters, models, and datasets in Section 4.1 and Appendix B. We share the source code through supplementary material.
ETHICS STATEMENT
Although text-to-image diffusion models are showing remarkable performance in creating high- fidelity images, they may generate harmful content, both intentionally and unintentionally, as the models do not always align well with the text prompts. Therefore, using extttt--image diffusion models requires extra caution.
This concern also applies to our approach. Despite the impressive performance the model when using FiFA, text-to-image models can still generate harmful, hateful, sexual images. Although mitigate this risk by filtering for text quality, as evidenced by improvements over models trained on the full dataset, the inherent issues of pretrained models can still arise in our models. We strongly recommend that users exercise caution when using models trained with our methods, considering the potential risks involved. Moreover, we will open-source the model, along with the safety filtering tool and guidelines for using our model.
ACKNOWLEDGEMENTS
This work was supported by Institute of Information & communications Technology Planning &
Evaluation (IITP) grant funded by the Korea government (MSIT) IRS-2019-II190775. Artificial Intel- ligence Graduate School Trogram(KAISTT Planning Evaluation (IITP) grant funded by the Korea (MV)) (No. R$ (8-2022--04477822. AI Research Hub Project), and Institute of Information & communications Technology Planning & (IIT)) grant funded by the Korea MSV)) (No. RS-2024-00509279, Global AI Frontier Lab). We thank Joonkee Kim and Gihun Lee for providing extensive feedback our paper.
Institute of Information & communications Technology
REFERENCES
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305. 13301, 2023.
Ondrej Bohdal, Yongxin Yang, and Timothy Hospedales. Flexible dataset distillation: Learn labels instead of images. arXiv preprint arXiv:2006. 08.500 2020.
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei A Efros, and Jun-Yan Zhu. Dataset distillation by matching training trajectories. In IEEE/VV Conference on Computer Vision and Pattern Recognition, 2022.
Chengliang Chai, Jiabin Liu, Nan Tang, Ju Fan, Dongjing Miao, Jiayi Wang. Yuyu Luo, and Guoliang Li. Goodcore: Data-effective and data-efficient machine learning through coreset selection over incomplete data. ACM Management Data, 2023.
Sayak Ray Chowdhury, Anush Kini, and Nagarajan Natarajan. Provably robust dpo: Aligning language models with noisy feedback. arXiv preprint arXiv:2403.00409 2024.
Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. Directly fine-tuning diffusion models on differentiable rewards. arXiv preprint arXiv:2309.17400 2023.
Xiaoliang Dai, Ji Hou, Chih- Yao Ma. Sam Tsai, Jialiang Wang. Rui Wang, Peizhao Zhang, Simon Vandenhende, Xiaofang Wang, Abhimanyu Dubey, et al. Emu: Enhancing image generation models using photogenic needles in haystack. arXiv preprint 200. 2023.
Fei Deng, Qifei Wang, Wei Wei, Matthias Grundmann, and Tingbo Hou. Prdp: Proximal re- ward difference prediction for large-scale reward finetuning of diffusion models. arXiv preprint AXxi::4022.087144 14, 2024.
11

---

Published as a conference paper at ICLR 2025
Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767 2023.
Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Reinforcement learning for fine- tuning text-to-image diffusion models. Advances Neural Information Processing Systems, 36, 2024.
Gongfan Fang. Xinyin Ma, and Xinchao Wang. Structural pruning for diffusion models. Advances in Neural Information Processing Systems, 2023.
Alireza Ganjdanesh, Reza Shirkavand, Shangqian Gao, and Heng Huang. Not all prompts made equal: Prompt-based pruning of text-to-image diffusion models. arXiv preprint arXiv:2406. 12042, 2024.
Animesh Gupta, Irtiza Hasan, Dilip K Prasad, and Deepak K Gupta. Data-efficient training of cnns and transformers with coresets: A stability perspective. arXiv preprint St.iv:23333020999 2023.
Muyang He, Shuo Yang, Tiejun Huang, and Zhao. Large-scale dataset pruning with dynamic uncertainty, arXiv preprint 23ti::2000005775. 2023.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 2020.
Mayg-Hyun uu Kim, Jinuk Kim, Seong Joon Oh, Sangdoo Yun, Hwanjun Song, Joonhyun Jeong, Jung- Woo Ha, and Hyun Oh Song. Dataset condensation via efficient synthetic-data parameterization. In International Conference on Machine Learning, 2022.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-pic: An open dataset of user preferences for text-to-image generation. Advances in Neural Information Processing Systems, 2024.
Jongwoo Ko, Saket Dingliwal, Bhavana Ganesh, Sailik Sengupta, Sravan Babu Bodapati, and Aram Galstyan. SeRA: Self-reviewing and alignment of LLMs using implicit reward margins. In The Thirteenth International Conference on Learning Representations, 2025. URL https: / //openeeviww.co net forum? ftt
Tor Lattimore and Csaba Szepesvári. Bandit algorithms. Cambridge University Press, 2020.
Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv: 2023.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-imane pre- training for unified vision-language understanding and generation. In International Conference on Machine Learning. PMLR, 2022.
Songhua Liu, Jingwen Ye, Runpeng Yu, and Xinchao Wang. Slimmable dataset condensation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
Simian Luo, Yiqin Tan, Longbo Huang. Jian Li, and Hang Zhao. Latent consistency models:
Synthesizing high-resolution images with few-step inference. arXiv preprint arX0v:2310000337. 2023.
Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models. In International Conference on Machine Learning, 2020.
E. Mitchell. A note on dpo with noisy preferences and relationship to ipo. https:// er- iemitchell.ai/cdoo. 2023.
Timothy Nguyen, Zhourong Chen, and Jachoon Lee. Dataset meta-learning from kernel ridge-
regression. arXiv preprint arXiv:2011.000 2020.
12

---

Published as a conference paper at ICLR 2025
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning data diet: Finding important examples early in training. Advances in Neural Information Processing Systems, 2021.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprini arXiv:2307.01952 2023.
Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning text-to-image diffusion models with reward backpropagation. arXiv preprini arXiv:2310.05739, 2023.
Friedrich Pukelsheim. Optimal design of experiments. SIAM, 2006.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. 2021.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 2024.
Ravi Raju, Kyle Daruwalla, and Mikko Lipasti. Accelerating deep learning with dynamic data pruning, arXiv preprint arXiv:2111.1200 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint 4rXiv:2204006125, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. IFEF/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Corby Rosset, Ching-An Cheng. Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. Direct nash optimization: Teaching language models to self oeff-improve with general preferences. arXiv preprint arXiv:2404.033 037115, 2024.
Noveen Sachdeva, Benjamin Coleman, Wang-Chers Kang, Jianmo Ni, Lichan Hong, Ed H Chi, James Caverlee, Julian McAuley, and Derek Zhiyuan Cheng. How to train data-efficient llms. arXiv preprint arXiv:2402.0966 2024.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 2022.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 2022.
Harshinder Singh, Neeraj Misra, Vladimir Hnizdo, Adam Fedorowicz, and Eugene Demchuk. Nearest neighbor estimates of entropy. American journal of mathematical and management sciences, 23 (3-4):301-321, 2003.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 2020.
Jiao Sun, Deqing Fu, Yushi Hu, Su Wang, Royi Rassin, Da-Cheng Juan, Dana Alon, Charles Herrmann, Sjoerd van Steenkiste, Ranjay Krishna, al. Dreamsyne: Aligning text-to-image generation with image understanding feedback. In Synthetic Data for Computer Vision Workshop@ CVPR 2024, 2023.
Haoru Tan, Sitong Wu. Fei Du, Yukang Chen, Zhibin Wang, Fan Wang, and Xiaojuan Qi. Data pruning via moving-one-sample-ont. Advances in Neural Information Processing Systems, 2024.
13

---

Published as a conference paper at ICLR 2025
Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. arXiv preprint trXiv:2311.1290S 2023.
Kai Wang, Bo Zhao, Xiangyu Peng. Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. Cafe: Learning to condense dataset by aligning features, In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis, arXiv preprint arXiv:2306.00 09341, 2023.
Xiaobo Xia, Jiale Liu, Jun Yu, Xu Shen, Bo Han, and Tongliang Liu. Moderate coreset: A universal method of data selection for real-world data-efficient deep learning. In International Conference on Learning Representations, 2022.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
Junwen Yang and Vincent Tan. Minimax optimal fixed-budget best arm identification in linear bandits. Advances in Neural Information Processing Systems, 35: 12:53-11- 2022.
Kai Yang, Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Qimai Li, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human feedback fine-tune diffusion models without any reward model. arXiv 2023a.
preprint arXiv:2311.132.
Kevin Yang, Dan Klein, Asli Celikyilmaz, Nanyun Peng, and Yuandong Tian. Rled: Reinforcement learning from contrast distillation for language model alignment. arXiv preprint arXiv:2307. 12950, 2023b.
Shuo Yang, Zeke Xie, Hanyu Peng, Min Xu, Mingming Sun, and Ping Li. Dataset pruning: Reducing training data by examining generalization influence. arXiv preprint arXiv:2205.09322 2022.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Bureu Karagol Ayan, al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprint arXiv:2206. 10789, 2(3):5, 2022.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 2021.
Yinan Zhang, Eric Tzeng, Yilun Du, and Dmitry Kislyuk. Large-scale reinforcement learning for diffusion models. arXiv preprint arXiv:2401. 12244, 2024.
Kangfeng Zheng, Xiujuan Wang, Bin Wu, and Tong Wu. Feature subset selection combining maximal information entropy and maximal information coefficient. Applied intelligence, 50:487-501, 2020.
Lianmin Zheng, Wii Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Erie Xing, et Judging Ilm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 2024.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao. Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 2024.
14

---

Published as a conference paper at ICLR 2025
A LIMITATIONS
Although our proposed FiFA demonstrates its effectiveness and efficiency, we validate our method primarily using the DPO objective among various aligmment methods such as policy gradient ap- proaches other DPO variants. It would also be meaningful to extend our algorithm to fit diverse algorithms, such as online DPO, its variants, or other RLHF methods. Specifically, for DPO variants that utilize preference datasets, FiFA integrates seamlessly. For online DPO, FiFA can be applied iteratively after generating online samples with current models to continuously refine the data.
Furthermore, we suggest applying FiFA to volicy-gradien based optimization by strategically selecting text prompts for training. This can be achieved by measuring the margins of online samples and evaluating LLM scores. We believe that such extension would not only be intriguing but also significantly enhance the value of our FiFA framework.
B IMPLEMENTATION DETAILS
Here, we explain the precise implementation details FiFA. An overview of our hyperparameters is presented in Table 2. For the final models of the main experiments, we update the SD1.5 models for 1000 steps and the SDXL models for 100 steps. We apply warmup steps of 10 for SD1.5 and 5 for SDXL. We adopt the Adam optimizer for SD1.5 and Adafactor for SDXL to save memory consumption. We set the learning rate to le-7 for SD1.5 and 2e-8 for SDXL. We use an effective batch size of 128 for both models by differentiating the batch size and accumulation steps. A piecewise constant learning rate scheduler is applied, which reduces the learning rate at certain steps.
For the full training of SDI.5, calculate the GPU hours for SDXL, we calculate the GPU hours for the first 20 steps, where the time spent for each step becomes constant, and then multiply that number to match the 1000 steps which the released version of SDXL is trained.
warmup steps of 500 and then
constant scheduler. To
Table Hyperparameters for SD1.5 and SDXL
Hyperparameters
SD1.5
SDXL
Update Steps
1000
100
Warmup Steps
10
5
Optimizer
Adam
Adafactor
Learning Rate
1c-7
2c-8
Learning Rate Scheduler
piecewise constant 1:200,0.25:400.00
piecewise constant
Learning Rate Scheduler Rule
1:200,0.1
Batch Size
8
Accumulation Steps
64
Effective Batch Size
128
128
C DPO FOR DIFFUSION MODELS
Direct Preference Optimization Given reward model trained with BT modelling of Eq. (2), the typical approach to increasing the reward is to utilize reinforcement learning maximize the reward. This often incorporates KL regularization term with reference distribution Prets which can be formulated follows:
max 1xpo ( c)) Prer(xo /c)]-
(11)
Direct Preference Optimization (DPO) (Rafailov et al., 2024) a method that directly aligns the model without reward training by integrating the reward model training and RL training stages into one. This approach leverages the insight that the optimal policy of the model in Eq. (11) can be represented by reward function. Hence, the optimal solution pa can be written
15

---

Published as a conference paper at ICLR 2025
(x(xolc) = Prer(Xolc) exp((((xo,c))) ())8)/((c)))//()))
(12)
where ((c) a partition function. Since (cc) usually intractable, one cannot directly obtain the optimal policy from this closed-form solution. However, after reformulating the reward in Eq. (12)
and incorporating it into the objective function of the BT model (Eq. (2)), intractable part cancel out and the result becomes tractable function parameterized by the model. The resulting loss becomes:
Pa (xD (c) 3 log Pier c)
po(xolc) Peer (xe(xxg
( -.Bexx logor (Blog
(13)
Diffusion DPO Objective For text-to-image generation, the diffusion model outputs (xo: c). where с is a text prompt and Xo is clean image. During inference, the DDIM sampler (Song et al., 2020) first obtains point estimate as follows:
X
VI (xE, c)
X
(14)
att
In order to obtain DPO loss of Eq. 13, one can first observe that estimation error between ground truth e. and our model makes conditional posterior as:
1-6e 2010
(x,,
(x c) (2
(15)
Here, one can use notation (1) as defined DDIM, and is the dimension of the data. Substitut- ing this into Eq. (13), the resulting loss can be reformulated follows:
Po (x,) xx c) Piot Prot xx", c)
Po xf, c)
[(0) -E, Er,,,x0xxx logo
[Boog
Bllgg
(16)
Pee xolxxt c)
1 ag
-col:
/e ccol 6r
e
20,0f
-BB,e,xy.xt log
B log
B log
(17)
as 20:o!
l|e
||
20.07
(BB(1 - aa))
E xb log
[[||e" cx(xx" 1)112 - ||e"" (xc] 1)
(18)
||e' ca (x/, 1) ||e| Cae (xf,
With the approximation 1-3e 1 ol+1² and by placing the expectation over time inside the log term, one can finally recover the Diffusion-DPO loss of Eq. (3):
at of of
(0) = - F2, F,,x, x logo (-BTw(A:) [[[le" cx(""" 1) - ||e" (x,"" , ||e' co(x,, t)l + ||e' ta(xx(+)))13 (19)
D LLM SCORE FOR TEXT PROMPTS
To calculate the LLM score for each text prompt in the Pick-a-Pic v2 training dataset, we use the gpt-3. 5-turbo-0123 model via the OpenAl API. Specifically, we instruct the LLM to assign low scores for excessive duplications, typos. and grammar errors. Additionally, low scores given to prompts that are too simplistic, trivial, or challenging, while learnable prompts are favored. Furthermore, allocate score of 0 filter out NSFW content.
16

---

Published as a conference paper at ICLR 2025
45000
20000
35000
15000
Prompt
# Prompt
25000
10000
15000
5000
2
0
10
Score
Score
(a) LLM scores in Pick-a-Pic v2 dataset
(b) LLM scores in HPSv2 dataset
Figure 8: LLM score distribution of text prompts in the (a) Pick-a-Pic v2 and (b) HPSv2 dataset.
Figure 8a and 8b show the resulting distribution of the LLM scores for prompts in the Pick-a-Pic v2 and HPSv2 datasets. Following Zheng et al. (2024) and Sachdeva et al. (2024), we use the following prompt templates for calculating text quality using LLM:
Prompt Template for LLM Score
Please act an impartial judge and assess the quality of the given text prompt for fine- tuning a text-to-ime text-to-image diffusion model. Use the following criteria to determine if a pair scores highly:
1. The text prompt should demonstrate high learnability. In other words, by learning this prompt, the model should be able to understand various concepts.
2. The prompt should contain no NSFW content; prompts with NSFW content should receive a score of
3. The prompt should have moderate difficulty, as prompts that are too difficult or too easy are not helpful.
4. Prompts with excessive duplicate words, typos, or grammar errors should receive lower scores.
Begin your evaluation by providing short and brief explanation. Be as objective as pas- sible. After providing your explanation, please rate the response on scale of 1 to 10 by strictly following this format: "[[ratin]]]", for example: "Rating: [[5]]'.
[Text Prompt] [text prompt)
As demonstrated in Table selected prompts with high LLM Scores generally contain more detailed and diverse expressions, enabling the model to learn a wide range of concepts, while filtered prompts contain simple and short contexts. Moreover, prompts with scores of O are extremely violent and crude, which could make models very harmful.
E EVALUATING HARMFUL CONTENTS
To assess the harmfulness of the DPO-trained model on the Pick-a-Pic v2 dataset, we evaluate harmfulness using three neutral prompts: simple prompt with action "a woman standing", simple prompt with an adjective "a beautiful woman", and a prompt that is not toxie but neutral, yet has higher probability of creating sexual images, "a hot girl". The motivation for using "woman" and "girl" is because user-generated prompts in the Pick-a-Pic v2 dataset frequently contain these keywords.
For each prompt, we generate 200 images with different seeds from 0 to 199 with the model trained using FiFA and the model trained using the full dataset. For each generated image, adopt human annotations, with three authors manually labeling each content with a binary label of harmful or not, considering the scale and safety issues when conducted other humans. We label image as
17

---

Published as a conference paper at ICLR 2025
Table 3: Examples of selected and filtered text prompts for Pick-a-Pic v2 by LLM score for FiFA
Status
Prompts
LLM score
close up of the demonic bison cyborg inside an iron maiden robot wearing royal robe, large view, surrealist painting by Jean Fouquet and alan bean and Philippe Druillet, volumetric lighting, detailed shadows
Select
8
cute halfting woman riding a friendly fuzzy spider while adventure, dnd, ttrpg, fantasy
Select
9
photo of teddybear and sunken steamtrain in the jungle river, flooded train, furry teddy misty mud rocks, panorama, headlights Chrome Detailing, teddybear eyes, open door
Select
8
Filter
A man with hat
3
Filter
text that says smile
4
Filter
BATMAN
3
Filter
Pixar-style cartoon of Ted Bundy.
0
Filter
Selfie of dead family, crude selfie
0
Pretrain
DPO + Full
DPO FiFA
Pretrain
DPO + Full
DPO + FiFA
"A woman standing"
"A beautiful woman"
Figure 9: Examples of generated images by the pretrained SDXL model, the model trained using FiFA, and the model trained on the full dataset with prompts related to women.
harmful if it contains NSFW content such nudity, vulgarity, or any other harmful elements. We employ majority voting to set the final label of each image. Then we calculate the harmfulness rate, the rate of images labeled as harmful out of all images for each prompt.
Figure shows some generated samples for these prompts from different models. As explained in Section 3.2. using the full dataset exposes the model to harmful content, which can lead to generating extremely harmful images. the other hand, when trained using FiFA, by considering text quality, can prevent the models from learning harmful behaviors, thereby ensuring safety.
F RESULTS WITH DIFFERENT STATISTICAL MEASURES
In our experiments on the Pick-a-Pic test set (Kirstain et al., 2024), including Figure Ia, generated four images for each prompt and then averaged the rewards across images and prompts. Here, we will show different statistical measures for aggregating multiple rewards for each prompt: the maximum reward each prompt (max), the minimum reward of each prompt (min), and the median reward of each prompt (median). Then, average the reward values across all prompts.
Figure 10 demonstrates the results when using different measures for aggregating rewards. For all measures, the model trained with FiFA significantly increases the PickScore compared to the model with full training, as when aggregated with the mean value. This shows that FiFA increases the quality of images regardless of specific seeds, indicating the robustness and generalizability of FiFA
18

---

Published as a conference paper at ICLR 2025
0- DPO Full
DPO FIFA
DPO Full
DPO FIFA
-0- DPO Full
-0- DPO FIFA
O
0.216
0.220
0.210
PickScore
PickScore
PickScore
0.214
0.218
0.208
0.206
0.204
0.210
0.214
0.202
0.208
200
400
600
1000
200
400
600
800
1000
200
400
600
800
1000
Training Steps
Training Steps
Training Steps
(a) Max PickScore of cach step
(b) Min PickScore of cach step
(c) Median PickScore each step
Figure 10: Results on the SD1.5 model with different statistical measures for aggregating PickScore values of each prompt using (a) max, (b) min, and (c) median instead of the mean value. These measures are then averaged across all prompts.
1,0
0.216
0.6
Accuracy
0.6
0.8
PickScare
Loss
0.6
0.208
Full FIFA
0.208
500
1000
500
1000
0
100
200
300
400
500
Training Steps
Training Steps
Training Steps
(a) Training loss
(b) Training accuracy
(c) Results weighted DPO
Figure 11: (a) Results of applying the reward- weighted DPO loss instead of the standard DPO loss. Our proposed FiFA still works well in this case. (b) Training loss of each step. FiFA enables a fast decrease in loss, while training the full dataset shows difficulty in converging. accuracy with implicit reward of each step. Our model shows a stable increase in implicit reward, demonstrating effectiveness and efficiency.
(c) Training
G TRAINING LOSS AND IMPLICIT ACCURACY
Figure lla shows the training loss comparing the training using FiFA and training using the full dataset. Consistent with the main result, FiFA enables much faster training with better convergence, as it decreases the loss rapidly. In contrast, due to the noisy nature, the loss of full training seems hard converge and eventually increases some points. Moreover, as shown in Figure lb,, the implicit reward model is much better trained when trained on the dataset pruned with FiFA. These results demonstrate that F FiFA makes training more stable and achieves faster convergence.
H REWARD-WEIGHTT DPO LOSS
We interpreted labels with low reward margins as noisy preferences. To make the training robust in this situation, Mitchell (2023) proposes a conservative DPO loss (cDPO), with the assumption that for the probability w E (0, 0.5), labels may be flipped. Using Eq. equation 3, cDPO loss is formulated
as follows:
Copo (0, xx xx)) (1 w) L0 xx)) LDro x// xxx
(20)
Motivated from cDPO, since have the proxy reward, we fully rely the proxy reward by setting the weight w based on the reward value, making the new rewiht-weighted DPO loss. Specifically, calculate the follows:
19

---

Published as a conference paper at ICLR 2025
Reward Margin de
LLM Score
Prompt Diversity
Selected
surrealism Disney pixar glowing green ghost owl
"the text "GifCo" written in sea shells the beach
"pikachu in hoodie"
Reward Margin
LLM Score
Prompt Diversity
Filtered
"wideangle panorama photo triceratops next to landrover defender...
diner on highway in desert at sundown..."
"Image_2717 2217"
Similar to selected prompt: "panorama photo triceratops ..."
Figure 12: Examples of selected and filtered samples when using FiFA.
exp(ro(x()/T) wxx0; x() exp(ro(x))) exp(ro(x))/7)
(21)
where T is the temperature, set to 0.01 in experiments. We interpret this loss incorporating the reward margin in the loss, so that it trains the model to learn based on how much one image preferred over the other. Figure lc shows the result of applying reward-weighted DPO loss. Even with weighted loss, FiFA shows superior performance with rapid increase in reward values compared the other basclines. Although choosing the data based on high reward margin is also effective, evidenced by the performance of HM, FiFA shows a larger gap training progresses with HM, implying the significance of the other two components even when incorporating the reward into the loss. The result demonstrates that FiFA aids training even when we modify the loss to fully depend on rewards, showing its applicability to DPO-based loss.
I SAMPLE ANALYSIS OF SELECTED DATA USING FIFA
Here, we analyze some selected or filtered samples by FiFA on either HPSv2 or Pick-a-pic v2 dataset. As explained in Section 3.3, we select samples based on high margin, text quality, and text diversity. As shown in Figure 12, the selected samples contain images with clear distinctions, meaningful prompts of appropriate difficulty, and minimal overlap with other prompts. On the other hand, the filtered prompts include images with either high or low text-image alignment, meaningless random prompts, prompts highly similar to the selected ones. These examples demonstrate that our objective function in Eq. (7) effectively selects samples that consider high preference margin and ensure high-quality, diverse text prompt sets, intended.
J PROOFS OF THEORETICAL ANALYSIS
In this section, we formally state Theorem with additional details. We first start with assuming linear model assumption for the reward feedback which is stated by following assumption.
Assumption 1 (Linear model with noisy feedback). For any feature vector ø(x, c) E Rd of image x and text there exists unknown parameter for the reward model 0,, .Rd such that reward r(x,c) c)
20

---

Published as a conference paper at ICLR 2025
is given by following equation:
r(x, c) = o(xx c) 0. +7.
(22)
Here, 1 is 1-subgaussian random noise from the reward model, and I C- RR df the given feature map which sends text prompt с & C and generated image I into d-dimensional latent space.
OLS estimator Suppose we have feature vectors (c) := o(x,, c) and corresponding observations r. r,(x, c) from Eq. 22 for i = 1, ..2..... n. Then, we can estimate the true parameter 6, by following estimator which is known OLS estimator:
-=V L
mol(i c)2
(23)
where design matrix V ;= Ei= (:(c)dx(C) For this Ó, we can obtain following confidence bound for any Φs E Rd, E (0, as follows (Lattimore and Szepesvári (2020), Chapter 20):
P (0 00-0, 0(x c)) 2 > /2110(x, -)1l- log (
8.
(24)
Above equation implies, we can get better estimate by minimizing @(xx c)llv and this problem called as a G-optimal design:
G-optimal design Let be distribution on the collection of (:(c) and A + [0, 1] be distribution on т. The goal G-optimal design is to find an optimal x which solves to find x that minimizes the following objective:
g(T)) = max (l. (f.c)
where, V(T) Eie (c) is design matrix constructed by the distribution x. Next, introducte Kiefer- Wolfowitz theorem which is stated as follows.
Theorem 2 Kiiefer-Wlffoitz) Following statements equivalent:
(i) n² is a minimizer of
(ii) x² is a maximizer of f(x) = llog det (TT)
(iii) g(T*) = dd.
The proof of the theorem can be found in Lattimore and Szepesvári (2020). We can combine Theorem 2 with Eq. 24 show that G-optimal design identifies optimal data configuration that minimizes model prediction error under a limited budget. G-optimal design maximizes det VTT) by selecting o; 's with diverse directions, implicitly incorporating diversity an objective. Moreover, we can further explain the benefit of using high reward margin pairs by observing that while Eq. 24 tries to bound the all of the feature vectors uniformly, model model prediction error most critical in the direction of 0.. By collecting higher reward margin pairs, we can increase singular values of the design matrix ((T)) to the direction of 0., One can expect this will reduce the model prediction error for good feature vector, which in turn, results in better performance of the model. The further support of the claim can be linked to the literature of best arm identification (Yang and Tan, 2022).
K HUMAN EVALUATION
In this section, we provide detailed information about our human evaluation, briefly explained in Section 4.1. We randomly select 100 prompts for each concept in the HPSv2 benchmark, where the concepts are photo, paintings, anime, and concept art. For the selected 100 prompts, we use 50 prompts create survey form and the remaining 50 prompts for another, totaling 8 survey forms with 400 prompts. We assign three different annotators to each survey form with a total of 24 annotators.
21

---

Published as a conference paper at ICLR 2025
In the case of human evaluations, take extreme care to ensure that no harmful content is included on the survey form. All the authors cross-check the content to prevent any harmful material from being exposed to others. Moreover, we do not collect any personal information from participants, including their name, email, or any other identifying details. Instead, all results are gathered from anonymous users. To ensure this, we cross-check all options, remove any questions related identity, and select options that do not collect any data (c.g., email) from the survey form. Additionally, among the eight test sets, each participant is assigned only one, making unnecessary collect private information such as an ID.
We have informed the participants about the purpose of our research, potential risks, and their right to withdraw their data. For the benefit of the participants, we provided payment of 158 for each participant, where the evaluation took less than an hour. The instruction we provided to the participants is written as below:
Instruction for Human Evaluation
We are conducting a study to evaluate the performance of the advanced text-to-image dif- fusion models. In this survey, you will be presented with pairs of images that were generated based on specific captions. Your task compare each pair of images and select the that you think best matches the caption or appears more visually appealing and coherent.
Before you begin, please be aware of the following important information:
Purpose of the Study: The aim of this study is to gather human feedback for the evalua- tion of certain text-to-image diffusion models.
Data Collection: We will collect data your image selections you provide during the survey. This data will be used solely for research purposes of evaluating certain text-to-image generation models.
While the study poses minimal risk, it is important to understand that your participation is not mandatory. Therefore, you have the right to withdraw from the study at any time, and you may request that your data be removed
Potential Risk:
By continuing with this survey, you acknowledge that you have read and understood the information provided above and consent to participate under these conditions. You have the right to ask any questions and receive clear answers before proceeding.
An example of the survey form illustrated in Figure 13.
L MORE RESULTS
Quantitative Impact of Text Diversity Diversity may play more significant role when the original dataset contains a large number of duplicated text prompts. Therefore, we provide further quantitative results to demonstrate the importance of incorporating text diversity through pilot experiment. Specifically, we select text prompts that are duplicated more than 20 times or share common keywords construct new dataset with multiple duplicated and similar prompts. We then apply FiFA with varying values to analyze the impact of text diversity. As shown in Figure 14a, neglecting diversity reduces performance, as the model predominantly trains on similar prompts. In contrast, incorporating diversity by selecting subset of distinct prompts from the original dataset achieves performance comparable the full set, highlighting the importance of text diversity.
Training without the Highest Margin Set To determine whether the highest margin set is necessary or if some level of high margin sufficient, we compare the results of applying FiFA to the original dataset and dataset excluding the top 10% margin pairs. As illustrated in Figure 14b, removing the top 10% significantly reduces performance, demonstrating that confident and clean pairs are essential for rapidly improving the model, particularly in the early stages of training, as they provide more informative signals.
Results Using Rewae To further demonstrate the generalizability of FiFA, employ an alternative preference reward function, ImageReward (Xu et al., 2024), for calculating reward gaps and evaluation. As shown in Figure 14c, compared random data selection using the full dataset,
22

---

Published as a conference paper at ICLR 2025
Please answer the following three questions, given two images separated by black gap. You must choose image that answers each question. the decision is too close, please also select the "tie" option. However, you still need to choose "left" or "right".
A moai wearing headphones.
Tie (Please also choose "Left" "Right")
Left
Right
Which image do you prefer given the caption?
Which image is more visually appealing?
Which image better fits the text description (caption)?
Example survey form for the human evaluation. We ask three questions about overall quality (general preference), image-only assessment, and text-image alignment. The images from FiFA and the full dataset are randomly assigned to the left or right, The annotators should choose either left or right, but they can also choose the tie option.
Figure 13:
23

---

Published as a conference paper at ICLR 2025
y=0.0 -0- y=0.5 -0-
v=1.0
original
excluding
Ours -0- Full
Random
0.216
0.216
0.8
PickScore
0.214
PickScore
0.214
ImageReward
0.6
0.4
0.212
0.212
0.2
0.210
0.210
100
300
500
100
300
500
100
300
500
Training Steps
Training Steps
Training Steps
(a) Impact Text Diversity (b) Impact of the Highest Margin Set (c) Results with ImageReward
Figure 14: (a) Impact of text diversity under specific settings. We created subset of the Pick-a-Pic v2 dataset by selecting prompts with similar or identical keywords and compared FiFA across different levels of diversity by varying yy (b) Results with and without filtering the top 10% margins of the training set before applying FiFA. (c) Comparison of random sampling (R), the full dataset (full,, and FiFA with ImageReward (Xu et al., 2024) on the Pick-a-Pic v2 dataset.
FiFA rapidly improves the reward, demonstrating both its effectiveness and efficiency in terms of ImageReward values. These results indicate that our method is not reliant on specific preference
model.
M RELATED WORK
M.1 ALIGNING TEXT-TO-IMAGE DIFFUSION MODELS
Fine-tuning text-to-image diffusion models using human feedback datasets has proven to be an effective way to improve large pretrained models. Early approaches are based on supervised fine- tuning, involving
weighted loss (Lee al., 2023), or rejection-sampling based methods Dong et al. (2023); Sun et al. (2023). Policy gradient approaches are also proposed Black et al. (2023); Fan et al. (2024) to further improve the models with online learning. However, these methods are highly ineffective and severely limited to small scale (Deng et al., 2024; Wallace et al., 2023). To address these issues, some methods directly use reward backpropagation (Clark et al., 2023; Prabhudesai et al., 2023), but these approaches unstable and often lead to reward overoptimization (Zhang et al., 2024). Diffusion-DPo (Wallace et al., 2023), which directly utilizes human preferences, enables large-scale training without reward hacking, demonstrating its effectiveness for alignment along with its variants (Deng et al., 2024; Yang et al., 2023a). In this paper, we focus on Diffusion-DP& as it represents one of the most effective state-of-the-art methods for alignment using human feedback.
M.2 DATA FILTERING FOR DEEP NEURAL NETWORKS
There have been attempts to improve the efficiency of training by selecting core subset from the entire training data. Data pruning (Raju et al., 2021; Yang et al., 2022; He et al., 2023; Tan et al., 2024) involves removing unnecessary less important data from dataset to reduce its size and enhance efficiency. Coreset selection (Xia et al., 2022; Mirzasoleiman et al., 2020; Gupta et al., 2023: Chai al., 2023) aims select a small subset that retains the original dataset's statistical properties. Data distillation (Cazenavette et al., 2022; Nguyen et al., 2020; Bohdal et al., 2020) and data condensation (Liu et al., 2023; Kim et al., 2022; Wang et al., 2022) generate small synthetic training data from the entire dataset that can achieve the learning effect of the entire data. However, these approaches usually estimate uncertainty based on gradients or loss, often do not align well with diffusion models, as calculating the loss or gradient for diffusion models is time-consuming, In the LLM domain, Zhou et al. (2024) introduced a small yet effective dataset for instruction tuning, while Ko et al. (2025), similar to our work, selected samples with large implicit reward margins during DPO training. In diffusion domains, Emu (Dai et al., 2023) demonstrated that training on a high- quality, curated dataset significantly improves image quality and text-image alignment. However, its reliance on extensive human curation makes it impractical for large-scale applications. Unlike these approaches, we propose automated data filtering framework that minimizes human involvement.
24

---

Published as a conference paper at ICLR 2025
N
MORE EXAMPLES
Figure 15-18 demonstrate more randomly selected examples generated using the Pick-a-Pic test set, PartiPrompt, and HPSv2 benchmark.
25

---

Published as a conference paper at ICLR 2025
Pretrain
DPO + Full
DPO + FiFA
"An image of a fantastical city floating in the clouds."
"A man drinking cup of cosmic energy in surreal anime style artwork."
"Symmetrical portrait of fantasy sorceress created by renownedartists including Yoshitaka Amano, Ruan Jia, Kentaro Miura" "
"An illustration featuring characters from Dungeons and Dragons game."
Figure 15: More example images generated using the different models.
26

---

Published as a conference paper at ICLR 2025
Pretrain
DPO Full
DPO + FiFA
"A bread cat."
"A frog wearing an anime-inspired onesie."
"A puppy is driving a car in film still."
"A low poly lemon in high quality rendering.
Figure 16: More example images generated using the different models.
27

---

Published as a conference paper at ICLR 2025
Pretrain
DPO + Full
DPO + FiFA
"Portrait of conquistador with a pet tiger in a jungle."
"A train crosses a trestle bridge in the mountains optimistic and vibrant illustration."
"Flying crocodile." "
EO
"The dirt bike has seen many hill climbs in its history."
Figure 17: More example images generated using the different models.
28

---

Published as a conference paper at ICLR 2025
Pretrain
DPO + Full
DPO + FiFA
"A photo of alien devices from alien spaceship."
"An oil painting of a child king ruling kingdom made entirely of cheese in a surreal and comic book style." "
"A woman eating vegetables in front of a stove." "
"A brown bear pushes a shopping cart in a grocery store."
Figure 18: More example images generated using the different models.
29


Reviews:
[{"review_id": "MZ9IKk4JTd", "reviewer": "Reviewer_uZ1n", "summary": "This paper introduces FiFA, an automated data filtering algorithm designed to optimize fine-tuning of text-to-image diffusion models, aligning model behavior more effectively with human intent. While human feedback datasets are valuable for model alignment, their large size and high noise levels often hinder convergence. FiFA enhances fine-tuning by automatically filtering data based on an optimization problem that maximizes three key components: preference margin, text quality, and text diversity. Experimental results show that FiFA enhances training speed and achieves better performance.", "questions": "Please refer to Weaknesses.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper propose the FiFA algorithm, which leverages three core metrics—preference margin, text quality, and text diversity—to optimize data filtering automatically. This approach effectively addresses noise in human feedback datasets and improves the fine-tuning of diffusion models, particularly for large-scale datasets.\n2. The paper is of high quality, with well-designed and comprehensive experiments, including several ablation and comparative studies that strongly support the effectiveness of FiFA in enhancing training efficiency and image quality.\n3. The structure of the paper is clear and well-organized. Key concepts, such as preference margin, text quality, and text diversity, are clearly defined, making the methodology accessible.", "weaknesses": "1. In the paper, some equations lack corresponding equation numbers.\n2. In the introduction, the phrasing around \"difficulty of convergence\" is inconsistent with the discussion of the iterative training required for diffusion models. It is recommended that the authors clarify the logical flow.\n3. In Equation 2, there is an extra left parenthesis \"(\".\n4. When using pre-trained models (e.g., CLIP, BLIP) to calculate the preference margin, how is the validity of the results ensured? Given that pre-trained models are trained on noisy and ambiguous datasets, they may also yield incorrect results.\n5. From the ablation study results, the effects of Text quality and Text diversity are not very significant. The authors state, \"when combined with a high margin, they outperform the model trained solely on margin, highlighting the importance of both components,\" but where are the results? Is the improvement due solely to the higher margin?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "baeHwyVx1r", "reviewer": "Reviewer_f1jQ", "summary": "This work presents a novel approach to fine-tuning text-to-image diffusion models using human feedback data filtered through an automated algorithm. The proposed methodology optimizes the fine-tuning process by selecting a subset of the available human feedback based on a preference margin criterion, enhancing the reward value while considering both prompt quality and diversity to maintain robustness and mitigate potential harmful content.", "questions": "1. The proposed filtering algorithm systematically narrows down the human feedback dataset to a subset that is optimal for model fine-tuning. As a general approach, further discussion on the generalizability of this filtering approach could enrich the analysis, such as how it may integrate with other alignment frameworks like RLHF and DPO-based methods. In addition, expanding the range of comparative methods would strengthen the evaluation.\n\n2. To clarify the novelty of this approach, the specific roles of preference margin and the quality/diversity metrics for text prompts could be further justified. Detailing the design motivations behind these components and their interdependencies would clarify their contributions to the model’s overall performance.\n\n3. DPO requires extensive high-quality preference data, which can be costly and difficult to obtain. The accuracy of preference data is essential, as low-quality feedback may lead to biased or suboptimal model behavior. There appears to be some ambiguity in the statement regarding dataset preparation: \"To ensure safety, we manually filter out some harmful text prompts from these test prompts, resulting in 446 unique prompts.\" It seems that an additional manual filtering step was applied before evaluating the proposed algorithm’s ability to handle harmful prompts. Clarifying this step’s rationale and how it affects the filtering method’s efficacy would add clarity.\n\n4. DPO-based approaches can sometimes narrow the scope of outputs, potentially limiting diversity. To validate the claimed advantage of this filtering method in maintaining diverse outputs, more empirical evidence should be presented. Additionally, a comparison with online DPO and recent DPO variants would help contextualize the findings.\n\n5. More qualitative evidence on how the proposed approach reduces training costs would be valuable, particularly with concrete examples or case studies showing the efficiency gains obtained through this filtering algorithm.\n\n6. Human evaluation is conducted in this work, however, it appears that there is no evidence of human ethics approval, despite it potentially being a low-risk case.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Please see questions", "weaknesses": "Please see questions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2B3lFaCtTd", "reviewer": "Reviewer_irH1", "summary": "In this paper, the authors aim to improve aligning text-to-image diffusion models from the perspective of filtering human feedback data. Specifically, they select the data pairs by maximizing three components: preference margin, text quality, and text diversity. For each component, they design an optimization objective. Finally, several experiments have been conducted to verify the contribution of each component", "questions": "How about the effectiveness of the proposed approaches on other datasets with different preference models?\n\nI also want to the win rate of the proposed approaches compared with only the base model or DPO.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tThe motivation of filtering the human feedback data is reasonable. It is well-known that the training of diffusion is very cost. High quality data would contribution both the effectiveness and efficiency of the model.\n\n2.\tThe paper writing is great.", "weaknesses": "1.\tThe technical contribution is relatively small. In my opinion, the proposed approaches for filtering data are travel and nature. In addition, as for preference margin, I believe that it is better to maximize preference margin in a limited range, while a very large margin would provide difficulty for optimization. \n\n2.\tOnly the pick-a-pic dataset is used in the experiment, which is highly related to Pick Score. Some other datasets should be involved to verify the generalization. For example, the authors can use the HPS score to compute the preference margin, even on the same pick-a-pic dataset.\n\n3.\tIt is also significant to show the pair-wise human evaluation in Figure 4.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rv4VJLplAI", "reviewer": "Reviewer_CrC4", "summary": "This paper proposes FiFA (Filtering for Feedback Alignment), a novel automated data filtering approach for efficiently fine-tuning text-to-image diffusion models using human feedback data. \n\nThe main contributions are:\n* An automated data selection method that maximizes: preference margin, text quality and text diversity.\n* Formulation of data selection as an optimization problem to find a subset that maximizes these components.\n* Empirical evidence showing FiFA's effectiveness across various models and datasets.", "questions": "The authors highlighted that the method can achieve good results with just 0.5% of the data. Do you have results showing how well FiFA filtering works on say 0.1%, 1%, 5%, 10% of the dataset? It could help us understand how tunable FiFA is.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents a novel approach to data filtering for fine-tuning text-to-image diffusion models. While data pruning and coreset selection are not new concepts in the domain of text-to-image diffusion models (first documented by Meta’s EMU paper), this work focuses on the automation of coreset selection. The combination of preference margin, text quality, and text diversity in a single optimization framework is an effective and reasonable solution in this problem space.\n\nThe paper demonstrates effective results across different models (SD1.5 and SDXL) and datasets (Pick-a-Pic v2 and HPSv2), providing robust evidence for their claims. The inclusion of both automatic metrics and human eval provide a complete picture in terms of metrics. There is also some  theoretical analysis provided in the author’s paper.\n\nits most impressive for the authors to achieve high quality alignment with just 0.5% of the data and 1% of the GPU hours. FiFA also demonstrated reduction in harmful content generation, which is critical for these automatic coreset selection method.", "weaknesses": "I think the biggest issue with this work is that it did not experiment with strong diffusion models like SD3-2B or FLUX models or the Playground models. Those models are much better to start with. It would be very helpful to know if the proposed model can further improve strong models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)