Automated Filtering of Human Feedback Data for Aligning Text-to-Image Diffusion Models

## Abstract

Abstract Fine-tuning text-to-image diffusion models with human feedback is an effective method for aligning model behavior with human intentions.
However, this alignment process often suffers from slow convergence due to the large size and noise present in human feedback datasets.
In this work, we propose FiFA , a novel automated data filtering algorithm designed to enhance the fine-tuning of diffusion models using human feedback datasets with direct preference optimization (DPO). Specifically, our approach selects data by solving an optimization problem to maximize three components: preference margin, text quality, and text diversity.
The concept of preference margin is used to identify samples that contain high informational value to address the noisy nature of feedback dataset, which is calculated using a proxy reward model.
Additionally, we incorporate text quality, assessed by large language models to prevent harmful contents, and consider text diversity through a k-nearest neighbor entropy estimator to improve generalization.
Finally, we integrate all these components into an optimization process, with approximating the solution by assigning importance score to each data pair and selecting the most important ones. As a result, our method efficiently filters data automatically, without the need for manual intervention, and can be applied to any large-scale dataset.
Experimental results show that FiFA significantly enhances training stability and achieves better performance, being preferred by humans 17 % percent 17 17\% 17 % more, while using less than 0.5 % percent 0.5 0.5\% 0.5 % of the full data and thus 1 % percent 1 1\% 1 % of the GPU hours compared to utilizing full human feedback datasets. Warning: This paper contains offensive contents that may be upsetting.

## 1 Introduction

Large-scale models trained on extensive web-scale datasets using diffusion techniques (Ho et al., ; Song et al., ), such as Stable Diffusion (Rombach et al., ), Dall-E (Ramesh et al., ), and Imagen (Saharia et al., ), have enabled the generation of high-fidelity images from diverse text prompts. However, several failure cases remain, such as difficulties in illustrating text content, incorrect counting, or insufficient aesthetics for certain text prompts (Lee et al., ; Fan et al., ; Black et al., ).
Fine-tuning text-to-image diffusion models using human feedback has recently emerged as a powerful approach to address this issue (Black et al., ; Fan et al., ; Prabhudesai et al., ; Clark et al., ).
Unlike the conventional optimization strategy of likelihood maximization, this framework first trains reward models using human feedback (Kirstain et al., ; Wu et al., ; Xu et al., ) and then fine-tunes the diffusion models to maximize reward scores through policy gradient (Fan et al., ; Black et al., ) or reward-gradient based techniques (Prabhudesai et al., ; Clark et al., ).
More recently, Diffusion-DPO (Wallace et al., ), which directly aligns the model using human feedback without the need for training reward models, has been proposed. This approach enables fine-tuning diffusion models at scale using human feedback, with the additional benefit of leveraging offline datasets more effectively.

However, fine-tuning diffusion models using human feedback requires considerable time and computational resources.
For instance, even with the relatively efficient Diffusion-DPO (Wallace et al., ), it
still takes more than thousands of GPU hours to fully fine-tune SDXL model Podell et al. () on the large-scale Pick-a-Pic v2 dataset (Kirstain et al., ).
This is attributed to the multiple denoising processes involved in diffusion models, as large diffusion models must be trained on multiple timestamps (Fan et al., ; Prabhudesai et al., ; Clark et al., ). Moreover, the noisy nature of feedback dataset slows down the convergence speed by making it harder for the model to accurately fit user preferences (Yang et al., ; Chowdhury et al., ).

Some model-centric approaches such as model pruning (Fang et al., ; Ganjdanesh et al., ), which reduces the size of diffusion models, or alternative scheduling techniques (Luo et al., ), which aim to reduce the number of timestamps, are utilized to improve efficiency. However, the nature of large-scale feedback datasets diminishes the impact of these methods, as the training cost increases dramatically with the growth of both model size and dataset, despite improvements in efficiency per data point. Some works (Dai et al., ) have attempted to create smaller datasets manually to reduce the need for large-scale data, but this approach requires significant human effort and is not applicable for reducing the size of large-scale feedback datasets, which may serve different purposes. This highlights the need for an automated data-centric approach for large-scale datasets.

Figure: (a) PickScore by each training step
Refer to caption: x1.png

In this paper, we propose a novel automated Filtering framework that selectively integrates human Feedback, designed for efficient Alignment of text-to-image diffusion models (FiFA).
We frame the filtering task as an optimization problem, aiming to find a subset that maximizes the three components; (1) preference margin, (2) text quality, and (3) text diversity. A key component of our optimization is selecting data pairs that are more informative, as determined by their preference margins, which are calculated using a proxy reward model.
Specifically, training pairs with a low preference margin can be considered noisy and ambiguous data, as their preferences may easily flip, thereby hindering the training process (Chowdhury et al., ; Yang et al., ; Rosset et al., ).
Furthermore, to address the concerns on harmfulness problems and coverage of selected subset induced by relying only on preference margin, we also consider the quality and diversity of the text prompts in the objective function.
We assess text quality using a Large Language Model (LLM), following Sachdeva et al. (), and measure text diversity by calculating the entropy of embedded text prompts (Zheng et al., ) approximated using a k-nearest neighbor estimator (Singh et al., ).
To integrate all these components, we define an objective function that combines the three metrics into a single optimization problem.
Additionally, to improve efficiency, we approximate the solution by assigning a data importance score for each data pair, making FiFA efficient and applicable to large-scale datasets through an automated process.

In our experiments with open-sourced text-to-image diffusion models, Stable Diffusion 1.5 (SD1.5) and Stable Diffusion XL (SDXL) (Podell et al., ), FiFA significantly improves training efficiency compared to fine-tuning with full datasets.
As shown in Figure 1(a), by using only $0.5\%$ of the full Pick-a-Pic v2 dataset (Kirstain et al., ), the SD1.5 model trained using FiFA demonstrates a significantly faster increase in PickScore (Kirstain et al., ) than the SD1.5 model trained on the full dataset. Moreover, the SDXL model trained using FiFA is preferred $17\%$ more than the model trained with the full dataset by human annotators when evaluated on the HPSv2 benchmark (Wu et al., ), with the preferred images showing better text-image alignment and higher quality, as illustrated in Figure 1(b).
We remark that this is achieved while requiring less than $1\%$ of the GPU hours, even considering time for data selection.
Additionally, FiFA reduces harmfulness by more than $50\%$ for neutral prompts compared to using full dataset, by prioritizing text quality.
Overall, our proposed FiFA allows large text-to-image diffusion models to be efficiently trained on a small yet important dataset, while showing better efficacy with high image quality.

## 2 Preliminaries

#### Diffusion Models

Diffusion models (Ho et al., ) are probabilistic models that aim to learn a data distribution $p(\mathbf{x})$ by performing multiple denoising steps starting from a Gaussian noise. The diffusion process consists of two parts, forward process and backward process.

In the forward process, noise is progressively injected at each timestamp $t$ according to $q(\mathbf{x}_{t}|\mathbf{x}_{0})\sim\mathcal{N}(\sqrt{\bar{\alpha}}_{t},(1-
\bar{\alpha}_{t})\mathbf{I})$, where the noise schedule $\alpha_{t}$ is a monotonically decreasing function and $\bar{\alpha}_{t}:=\prod_{s=0}^{t}{\alpha_{s}}$. The neural network $\bm{\epsilon_{\theta}}$ is trained to learn the denoising process with the following objective:

$$ $\mathcal{L}_{DM}(\theta)=\mathbb{E}_{\mathbf{x}_{0},t}[\lambda(t)||\bm{ \epsilon}-\bm{\epsilon}_{\bm{\theta}}(\mathbf{x},t)||_{2}^{2}],$ $$

where $\lambda(t)$ is determined by the noise schedule and $\bm{\epsilon}$ is a Gaussian noise. During generation, the diffusion model takes reverse denoising steps starting from a random Gaussian noise.

The conditional diffusion model (Rombach et al., ), such as a text-to-image diffusion model, aims to learn the data distribution $p(\mathbf{x}|\mathbf{c})$, trained using the conditional error $\bm{\epsilon}(\mathbf{x},\mathbf{c})$ instead of the unconditional error $\bm{\epsilon}(\mathbf{x})$.

#### Reward Learning in Text-to-Image Domains

Using human preference data, the goal of reward learning is to train a proxy function aligned with human preferences. In text-to-image domains, given textual condition $\mathbf{c}$ and the generated image $\mathbf{x_{0}}$ from that condition, we assume a ranked pair with $\mathbf{x_{0}^{\mathit{w}}}$ as a “winning” sample and $\mathbf{x_{0}^{\mathit{l}}}$ with a “losing sample”, that satisfy $\mathbf{x_{0}^{\mathit{w}}}>\mathbf{x_{0}^{\mathit{l}}}|\mathbf{c}$. Using the Bradley-Terry (BT) model, one can formulate maximum likelihood loss for binary classification to learn the reward model $r$, parameterized by $\phi$, as follows:

$$ $\mathbf{\mathcal{L}_{\text{BT}}(\phi)}=-\mathbb{E}_{\mathbf{c},\mathbf{x}_{0}^ {\mathit{w}},\mathbf{x}_{0}^{\mathit{l}}}\left[\log{\sigma(r_{\phi}(\mathbf{c} ,\mathbf{x}_{0}^{\mathit{w}})-r_{\phi}(\mathbf{c},\mathbf{x}_{0}^{\mathit{l}}) )}\right],$ (1) $$

where $\sigma$ is a sigmoid function, $\mathbf{c}$ is a text prompt, and image pairs $\mathbf{x}_{0}^{\mathit{w}}$ and $\mathbf{x}_{0}^{\mathit{l}}$ labeled by humans.

#### Direct Preference Optimization for Diffusion Models

Direct Preference Optimization (DPO) (Rafailov et al., ) is an approach to align the model using human feedback without training a separate reward model. Directly applying DPO loss to diffusion models is not feasible, as an image is generated through the trajectory $\mathbf{x}_{T:0}$ where $T$ denotes the number of denoising steps, and obtaining the probability of this entire trajectory is generally intractable. Following Diffusion-DPO (Wallace et al., ), the DPO loss for diffusion models can be approximated as follows:

$$ $\begin{split}\mathcal{L}_{\text{DPO}}(\theta)=-\mathbb{E}_{t,\mathbf{c}, \mathbf{x}_{0}^{\mathit{w}},\mathbf{x}_{0}^{\mathit{l}}}\log\sigma\left(-\beta T \omega(\lambda_{t})\right.\left[\left(\|{\bm{\epsilon}^{\mathit{w}}}-{\bm{ \epsilon}}_{\theta}(\mathbf{x}_{t}^{\mathit{w}},t)\|_{2}^{2}-\|\bm{\epsilon}^{ \mathit{w}}-{\bm{\epsilon}}_{\text{ref}}(\mathbf{x}_{t}^{\mathit{w}},t)\|_{2}^ {2}\right.\right.\\ -\left.\left.\|\bm{\epsilon}^{\mathit{l}}-{\bm{\epsilon}}_{\theta}(\mathbf{x}_ {t}^{\mathit{l}},t)\|_{2}^{2}+\|\bm{\epsilon}^{\mathit{l}}-\bm{\epsilon}_{ \text{ref}}(\mathbf{x}_{t}^{\mathit{l}},t)\|_{2}^{2}\right]\right),\end{split}$ (2) $$

where $w(\lambda_{t})$ is a weight function typically set to a constant, $\mathbf{x}_{t}^{w}$, $\mathbf{x}_{t}^{l}$ are the noised inputs of winning and losing images at timestep $t$ respectively, $\bm{\epsilon}^{w},\bm{\epsilon}^{l}\sim\mathcal{N}(0,I)$ represent the Gaussian noise for the winning and losing images respectively, and $\bm{\epsilon_{\text{ref}}}$ is a pretrained diffusion model.
Detailed derivation of DPO loss for diffusion is presented at Appendix B.

## 3 Methods

Aligning text-to-image diffusion models with large-scale human feedback data requires significant computational resources and training time.
Moreover, the noisy nature of feedback datasets complicates DPO optimization (Chowdhury et al., ), motivating us to identify a core subset to improve and accelerate alignment process.

To tackle this issue, we propose FiFA, which automatically filters the full human feedback data to obtain a subset for efficiently fine-tuning text-to-image models. Specifically, our method leverages preference margin as a key component to rapidly increase the reward value, while also considering the quality and diversity of the text prompts to mitigate harmfulness and ensure robustness. Additionally, we frame the task as an optimization problem to find the best subset that maximize these three components, resulting in an automated filtering framework applicable to any large-scale dataset.

### 3.1 Preference Margin

Figure: (a) Samples of high/low preference margin
Refer to caption: x3.png

The noisy and ambiguous nature of human preference datasets has been well explored, where a labeled preference does not reflect the true preference and may contain spurious correlations (Yang et al., ; Chowdhury et al., ).
This can be especially problematic for the efficient fine-tuning of diffusion models, as such noisy data slow down training and reduce the generalization capability (Zhang et al., ).
Inspired by recent papers that highlight the importance of clean preference pairs (Yang et al., ; Rosset et al., ), we use the preference margin to enable more efficient and effective fine-tuning of diffusion models to alleviate this issue.

To estimate the preference margin between the winning and losing images, we utilize a proxy reward model $r_{\phi}$ trained on the full feedback dataset using the BT modeling approach with Eq. (1).
This process does not pose efficiency concerns for text-image domains, as training a reward model using CLIP (Radford et al., ) or BLIP architectures (Li et al., ) demands significantly less time than training large diffusion models, Additionally, open-sourced text-image reward models like PickScore (Kirstain et al., ) and HPSv2 (Wu et al., ), trained on human feedback datasets, can also be utilized.

Figure 2(a) shows samples with different preference margins estimated using the PickScore proxy reward model. For example, given the prompt “A cat knight…”, image pairs with high reward margin are more distinct, clearly showing that the image including a “cat” should be preferred over the one without it.
On the other hand, for the low-margin pairs, both images usually differ only in style, implying that preferences can be flipped depending on the annotator.
This demonstrates that selecting pairs with clear distinctions offer more informative preferences based on the prompt.

Additionally, Figure 2(b) demonstrates that most pairs in the large-scale Pick-a-Pic v2 dataset are concentrated in a low-margin region.
Fine-tuning diffusion models primarily on these low-margin samples can lead to slow convergence, as it offers limited benefit from the perspective of G-optimal design, which will be further elaborated in Section 3.3.

### 3.2 Text Quality and Diversity

Figure: (a) Samples of harmful images
Refer to caption: x5.png

While reward margin is a critical component, relying solely on the reward margin may overlook two critical factors: the quality and diversity of text prompts.

#### Text Quality

The text prompts in human feedback datasets created by real users tend to be of low quality due to unformatted structures, typos, and duplicated content. More importantly, these prompts may include harmful components, such as sexual content, bias, or violence. Figure 3(a) demonstrates the potential harm caused by naively using all open-sourced Pick-a-Pic v2 dataset for fine-tuning, motivating us to ensure that the system takes the quality of the text into consideration.

To estimate this text quality, inspired by ASK-LLM (Sachdeva et al., ),
which was developed to assess the usefulness of pretraining data,
we evaluate the text quality using a LLM. Specifically, we ask the LLM to evaluate whether the text prompt is clearly formatted, understandable, of appropriate difficulty, and free of harmful content by providing a score. We scale the scores from 0 to 10, denoted as the LLM score. In our experiments, we use OpenAI gpt-3.5-turbo-0125 model. A detailed explanation on LLM score is available in Appendix C.

#### Text Diversity

The problem with our selection method is that image pairs with a high preference margin may be focused on certain prompts or families of prompts. This is supported by Figure 3(b), as relying on high-margin prompts leads to a decrease in most diversity metrics, such as word entropy, compared to using the full dataset. The lack of diversity may limit generalization capability. Therefore, we also consider text diversity during data filtering.

To estimate text diversity, we employ the entropy of the embedded text prompts (Zheng et al., ). Specifically, let ${C}$ denote a random variable with a probability density function $p$, representing a distribution of a selected subset of text prompts from the full dataset $D$, where each text prompt is embedded in $\mathbb{R}^{d}$ space. Text diversity is then estimated through $\mathcal{H}(C)$, where $\mathcal{H}(C)=-\mathbb{E}_{c\sim p(c)}\left[\log p(c)\right]$, with high $\mathcal{H}(C)$ indicating high text diversity.

Additionally, we set the hard constraint on the selected number of pairs for each text prompt, which is set to 5 and doubled if the number of desired data $K$ is not met, to prevent the selection of a large number of duplicate prompts among the selected subset.

### 3.3 Automated Data selection with Objective Function

Figure: Algorithm 1 Algorithm for FiFA

Given the components for data importance, the remaining challenge is how to incorporate all components into an automated data filtering framework that could be applied to any dataset. To achieve this, we formulate data selection as an optimization problem to find the subset with high margin, text quality, and diversity. The pseudocode for our algorithm is presented in Algorithm 1.

#### Objective Function

Our objective function should consider the preference margin, text quality and text diversity. The first, preference margin $m^{reward}$, is calculated using the trained proxy reward model. Specifically, given each data pair $\{\mathbf{c},\mathbf{x_{0}^{\mathit{w}}},\mathbf{x_{0}^{\mathit{l}}}\}$, we calculate the reward margin $m^{reward}$ as follows:

$$ $m^{reward}(\mathbf{c},\mathbf{x}_{0}^{\mathit{w}},\mathbf{x}_{0}^{\mathit{l}}) =|r_{\phi}(\mathbf{c},\mathbf{x}_{0}^{\mathit{w}})-r_{\phi}(\mathbf{c},\mathbf {x}_{0}^{\mathit{l}})|,$ $$

where $\mathbf{c}$ is a text prompt. Then, we use the LLM score to evaluate the quality of the text prompts and text entropy $\mathcal{H}(C)$ to measure diversity, as explained in Section 3. Combining all of these components, our goal is to find the subset $\mathcal{S}$ that maximizes the following objective function $f$:

$$ $f(\mathcal{S})=\sum_{\mathbf{c},\mathbf{x_{0}^{\mathit{w}}},\mathbf{x_{0}^{ \mathit{l}}}\in\mathcal{S}}\left[m^{\text{reward}}(\mathbf{c},\mathbf{x}_{0}^{ \mathit{w}},\mathbf{x}_{0}^{\mathit{l}})+\alpha*LLM_Score(\mathbf{c})\right]+ \gamma*\mathcal{H}(C),$ (3) $$

where $\alpha$ and $\gamma$ are hyperparameters for balancing the three components. Unlike the other terms, calculating $\mathcal{H}(C)$ is infeasible. To address this issue, we estimate the entropy value using a k-nearest neighbor entropy estimator (Singh et al., ). Specifically, $\mathcal{H}(C)$ can be approximated as follows:

$$ $\mathcal{H}(C)\propto\frac{1}{N_{c}}\sum_{i=1}^{i={N_{c}}}\log\|c_{i}-c_{i}^{k \text{-}NN}\|_{2},$ (4) $$

where $N_{c}$ is the number of text prompts, and $c_{i}^{k\text{-}NN}$ is the $k$-NN of $c_{i}$ within a prompt set $\{c\}_{i=1}^{N_{c}}$. Although Eq. 4 enables the calculation of $\mathcal{H}(C)$, finding an optimal set of $C$ that maximizes this function is not feasible for large-scale datasets. Therefore, to efficiently select data, we approximate the function by calculating $\log\|c_{i}-c_{i}^{k\text{-}NN}\|_{2}$ over the entire set of prompts and use this as an estimator of the diversity score for each data pair. The final objective function $\tilde{f}$ that represents the data importance score for each data point is then formulated as follows:

$$ $\tilde{f}(\mathbf{c},\mathbf{x}_{0}^{\mathit{w}},\mathbf{x}_{0}^{\mathit{l}})= m^{\text{reward}}(\mathbf{c},\mathbf{x}_{0}^{\mathit{w}},\mathbf{x}_{0}^{ \mathit{l}})+\alpha*LLM_{{Score}}(\mathbf{c})+\gamma*\log\|\mathbf{c}-\mathbf {c}^{k\text{-}NN}\|_{2}.$ (5) $$

Using this objective function, we can select data by choosing the top $K$ data that have high $\tilde{f}$ value, with $K$ determined based on the computational burden, as formulated below:

$$ $S=\underset{X,|X|=K}{\text{argmax}}\sum_{(\mathbf{c},\mathbf{x}_{0}^{\mathit{w }},\mathbf{x}_{0}^{\mathit{l}})\in X}\tilde{f}(\mathbf{c},\mathbf{x}_{0}^{ \mathit{w}},\mathbf{x}_{0}^{\mathit{l}}).$ $$

As shown in Figure 3(b), by increasing $\alpha$ and $\gamma$, the subset selected with FiFA achieves higher LLM scores and diversity scores, proving that this approximated objective function empirically works as intended. The analysis of the selected and filtered samples using FiFA is presented in Appendix H.

#### Interpretation of FiFA

Our method, which considers both diversity and a high preference margin, is connected to the theoretical interpretation related to G-optimal design (Pukelsheim, ). Here, we establish this connection through the following theorem under a linear reward model assumption.

###### Theorem 1 .

Denoting $\phi_{i}(\mathbf{c}):=\phi(\mathbf{x}_{0,i}^{w},\mathbf{c})-\phi(\mathbf{x}_{0
,i}^{l},\mathbf{c})$ with feature vector $\phi$. Define $g$ as:

$$ $g(\pi)=\max_{\begin{subarray}{c}(i,\mathbf{c})\end{subarray}}\|\phi_{i}( \mathbf{c})\|_{V(\pi)^{-1}}^{2},$ (6) $$

where $V(\pi):=\sum\pi(i,\mathbf{c})\phi_{i}(\mathbf{c})\phi_{i}(\mathbf{c})^{\top}$ is the design matrix with $\pi:(i,\mathbf{c})\rightarrow[0,1]$ being a probability distribution.
Assume $r_{i}(\mathbf{c})=\phi_{i}(\mathbf{c})^{\top}\mathbf{\theta}_{\star}+\eta_{i}$ where $\theta_{\star}$ is an unknown parameter and $\eta_{i}$ is a random noise sampled from $1$-subgaussian. Then, with $n(i,\mathbf{c})=\lceil\frac{2\pi(i,\mathbf{c})g(\pi)}{\epsilon^{2}}\log{\frac{
1}{\delta}}\rceil$ number of samples, one can obtain following error bound on the model prediction $\hat{\theta}$ with probability $1-\delta:$

$$ $\langle\hat{\theta}-\theta_{\star},\phi_{i}(\mathbf{c})\rangle\leq\epsilon.$ $$

The theorem suggests that minimizing model prediction error can be achieved by increasing the smallest singular value of the design matrix $V(\pi)$, which is guaranteed by collecting samples with diverse feature vectors. This supports intuition on considering text diversity in FiFA.
Moreover, selecting high reward margin pairs is likely to increase the norm of $\phi_{i}(\mathbf{c})$, which, in turn, can increase the singular values of the design matrix $V(\pi)$. This can reduce $g(\pi)$ in Eq. (6), thereby requiring fewer samples for desired level of prediction performance. Further explanation, including more details and proofs, is provided in Appendix I.

###### Theorem 1 .

## 4 Experiments

### 4.1 Experimental settings

#### Dataset

We use the Pick-a-Pic v2 dataset (Kirstain et al., ), which consists of $59k$ text prompts and $850k$ human preference pairs without tie, and the HPS v2 dataset (Wu et al., ), which consists of $104k$ text prompts and $645k$ human preference pairs, for fine-tuning in our main experiments.
The pairs in these datasets contain images generated using SDXL-beta, Dreamlike, a fine-tuned version of SD1.5, etc.
For ablation and further analysis, we mainly use models trained on the Pick-a-Pic v2 dataset.
We primarily use the Pick-a-Pic test set for evaluation.
To ensure safety, we manually filter out some harmful text prompts from these test prompts, resulting in 446 unique prompts. Moreover, to test the ability of the model to generalize across diverse prompts, we utilize text prompts from PartiPrompt (Yu et al., ), which consist of 1630 prompts, and the HPSv2 benchmark (Wu et al., ), which consists of 3200 prompts with diverse concepts.

**Table 1: Comparison of our methods with the baselines, trained on Pick-a-Pic v2 or HPSv2 dataset. Pretrain denotes the pretrained model, and Full denotes using the full trainset. PickScore (PS) and HPSv2 reward (HPS) values are multipled by 100 for displaying. GPU hour is based on NVIDIA A6000 GPU. AE represents Aesthetic Score.**
|  |  |  |  | Number | Pick-a-Pic test | PartiPrompt | HPSv2 benchmark |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Trainset | Models | Methods | GPU (h) | Pairs | Captions | PS | HPS | AE | PS | HPS | AE | PS | HPS | AE |
| Pick | SD1.5 | Pretrain | N/A | N/A | N/A | 20.82 | 26.26 | 5.32 | 21.43 | 26.60 | 5.17 | 20.79 | 26.76 | 5.29 |
| DPO + Full | 56.2 | 850k | 59k | 21.19 | 26.37 | 5.42 | 21.68 | 26.82 | 5.22 | 21.23 | 27.09 | 5.44 |  |  |
| DPO + FiFA | 13.6 | 5k | 2k | 21.64 | 26.95 | 5.52 | 22.06 | 27.43 | 5.35 | 21.84 | 27.84 | 5.59 |  |  |
| SDXL | Pretrain | N/A | N/A | N/A | 22.23 | 26.85 | 5.83 | 22.56 | 27.24 | 5.56 | 22.71 | 27.63 | 5.92 |  |
| DPO + Full | 1760.4 | 850k | 59k | 22.73 | 27.32 | 5.82 | 22.96 | 27.67 | 5.61 | 23.10 | 28.09 | 5.92 |  |  |
| DPO + FiFA | 18.3 | 5k | 2k | 22.76 | 27.42 | 5.89 | 22.97 | 27.78 | 5.66 | 23.17 | 28.18 | 5.94 |  |  |
| HPSv2 | SD1.5 | Pretrain | N/A | N/A | N/A | 20.82 | 26.11 | 5.32 | 21.39 | 26.59 | 5.17 | 20.79 | 26.76 | 5.29 |
| DPO + Full | 52.4 | 645k | 104k | 20.91 | 26.46 | 5.33 | 21.45 | 26.87 | 5.14 | 21.05 | 27.19 | 5.28 |  |  |
| DPO + FiFA | 12.5 | 5k | 3k | 20.90 | 27.03 | 5.40 | 21.44 | 27.43 | 5.19 | 20.98 | 27.91 | 5.41 |  |  |
| SDXL | Pretrain | N/A | N/A | N/A | 22.28 | 26.85 | 5.83 | 22.54 | 27.23 | 5.56 | 22.76 | 27.63 | 5.92 |  |
| DPO + Full | 1640.4 | 645k | 104k | 22.32 | 26.98 | 5.84 | 22.58 | 27.39 | 5.61 | 22.80 | 27.81 | 5.92 |  |  |
| DPO + FiFA | 17.2 | 5k | 3k | 22.24 | 27.26 | 5.93 | 22.51 | 27.61 | 5.81 | 22.75 | 28.19 | 6.04 |  |  |

#### Evaluation

We measure performance automatically using PickScore (Kirstain et al., ) and HPSv2 Reward (Wu et al., ), as our aim is to rapidly enhance the reward through DPO training. To also assess improvement on image-only quality, we additionally utilize the LAION Aesthetic Score (Schuhmann et al., ). To validate the efficiency of each method, we calculate the GPU hours using an NVIDIA A6000 GPU. We also include the time required to calculate rewards and LLM scores for FiFA.

Moreover, to validate the results against real human preferences, we conduct a human evaluation using the HPSv2 benchmark, which includes four concepts: photo, paintings, anime, and concept art. Specifically, we randomly select 100 prompts for each concept in the HPSv2 benchmark, totaling 400 prompts. For each prompt, we assign three annotators and ask them three questions: 1) Overall Quality (General Preference), 2) Image-only Preference, and 3) Text-Image Alignment, following Wallace et al. (). A detailed explanation of the human evaluation is presented in Appendix J.

For the other ablation studies, we present performance on Pick-a-Pic test prompts, evaluated automatically using PickScore. For the Pick-a-Pic test set, we generate four images for each prompt, while for the PartiPrompt and HPSv2 benchmarks, we generate one image per prompt. To calculate the performance, we take the average of the PickScore, HPSv2 Score, and Aesthetic Score for each prompt. Results with different statistical measures are available in the Appendix E.

#### Implementation Details

We utilize PickScore (Kirstain et al., ) for the Pick-a-Pic v2 dataset and HPSv2 (Wu et al., ) for the HPS v2 trainset as proxy reward models, as they have been trained on their respective full trainsets. In our experiments, we set both $\alpha$ and $\gamma$ to 0.5. For training the model with the full dataset, we follow the settings of the original paper (Wallace et al., ). Specifically, we train SD1.5 with a learning rate of $1\mathrm{e}{-8}$ and an effective batch size of 2048. To compare our SDXL models with the model trained on the full dataset, we use the released checkpoint of the Hugging Face SDXL-DPO.(^3^33[https://huggingface.co/mhdang/dpo-sdxl-text2image-v1](https://huggingface.co/mhdang/dpo-sdxl-text2image-v1)) When training our models with fewer data
we set $\beta$ to 5000 and have an effective batch size of 128. Additionally, we use a learning rate of $1\mathrm{e}{-7}$ for SD1.5 and $2\mathrm{e}{-8}$ for SDXL. In our main experiments, we train SD1.5 for 1000 steps and SDXL for 100 steps. For the ablation studies, we mainly utilize SD1.5.
More detailed implementation information is presented in Appendix A, and training logs are available in Appendix F.

### 4.2 Main Results

Figure: Figure 4: Human evaluation results. We compare SDXL trained with FiFA against SDXL trained on the full dataset using the HPSv2 benchmark. The SDXL model with FiFA consistently outperforms the SDXL model with the full dataset in terms of both aesthetic quality and text-image alignment, leading to superior overall quality.
Refer to caption: x6.png

Figure: Figure 5: Samples from the HPSv2 benchmark, generated using a pretrained model, the model trained on the full dataset (DPO + Full), and the model trained using FiFA (DPO + FiFA). Images from the DPO+FiFA model show better alignment to the prompts and higher quality than the others.
Refer to caption: x7.png

#### Quantitative Results

Table 1 demonstrates the performance of our methods compared to the baselines across three different reward models.
Our method, requiring only $20\%$ of the training time for SD1.5 and less than $1\%$ of the training time for SDXL, consistently outperforms the trained models that use the full dataset for most metrics on all benchmarks, especially on SD1.5.
The performance increase in both train sets indicate that FiFA is generalizable across different datasets. Notably, the increases in the Aesthetic Score, in addition to human preference rewards, indicate that the models trained using FiFA robustly enhance image quality.

Moreover, FiFA achieves high scores on PartiPrompt, which tests various compositions, along with the HPSv2 benchmark, featuring diverse prompts from various concepts and domains. This demonstrates that our model, trained with the small dataset obtained from FiFA, can generalize well across a wide range of prompts from different domains and styles.

#### Human Evaluation

We also conduct a human evaluation to see if this higher reward actually leads to better human preference. As shown in Figure 4, human annotators prefer the images from SDXL with FiFA $54.8\%$ of the time, compared to $37.0\%$ for the model trained on the full dataset, in terms of overall quality.
Moreover, our model outperformed the full dataset model by $17\%$ in aesthetic and $15\%$ in text-image alignment, indicating that the images generated by our model are more visually pleasing and better aligned with the text.

#### Qualitative Comparison

Figure 5 shows the images generated by the SDXL model trained using the full dataset and the model trained using FiFA. One can clearly see that the model trained on the full dataset and the pretrained model sometimes fail on certain words, highlighted in bold, by missing objects or counts. In contrast, our models consistently follow the text prompts and provide better details. More examples are presented in Appendix L.

Figure: (a) Comparison with vanilla pruning
Refer to caption: x8.png

### 4.3 Additional Experiments

#### Comparison with Vanilla Pruning Methods

In this section, we compare FiFA with multiple baselines, including traditional data pruning techniques of coreset selection (Mirzasoleiman et al., ) (CS) using CLIP embeddings, error score based selections (L) (Paul et al., ) using DPO loss, and random selection (R). We also compare it with the baselines of naively utilizing samples with high rewards (HR) of winning images. As shown in Figure 6(a), FiFA outperforms all the baselines, demonstrating its effectiveness. Specifically, traditional baselines (CS and L) perform poorly as they are not designed for diffusion model alignment while requiring more filtering time. Random or absolute reward-based filtering also underperforms, showing that smaller datasets alone do not ensure efficient training, underscoring the effectiveness of our method.

#### Analysis of Each Component

We analyze each component of FiFA; preference margin (high (HM) and low (LM)), text quality (TQ), and text diversity (TD). Figure 6(b) demonstrates the results of this ablation. The high reward margin is crucial, as its removal significantly reduces performance, and a low margin leads to the worst reward score. Additionally, while text quality and diversity alone do not have specific impacts, when combined with a high margin, they outperform the model trained solely on margin, highlighting the importance of both components.

#### Ablation on the Number of Data K 𝐾 K italic_K

We conduct an ablation study on the number of data points, $K$, selected using FiFA. As depicted in Figure 6(c), performance increases with $K$ but decreases when $K$ exceeds $5,000$. The lower performance with smaller datasets occurs because such datasets lack diversity, leading to overfitting on too few prompts and images. In contrast, a larger dataset might include data with low margins or poor prompt quality, which are less informative and can degrade overall performance. The results may depend on the original dataset, as including more data will be beneficial if the original set is of high quality.

### 4.4 Ablation on Text Quality and Diversity

Figure: (a) Ablation on $\alpha$ and $\gamma$
Refer to caption: x11.png

#### Ablation on α 𝛼 \alpha italic_α and γ 𝛾 \gamma italic_γ

Here, we explore how different $\alpha$ and $\gamma$ values, that control the effects of text quality and diversity, affect the performance of trained models. The results are illustrated in Figure 7(a). Since a preference margin is used in all configurations, performance remains high compared to the model trained on the full dataset, demonstrating FiFA’s robustness to hyperparameters. However, extremely high or low $\alpha$ and $\gamma$ values are ineffective, either reducing the total margin or compromising text quality and diversity. An $\alpha$ range of 0.1-1.0 and a $\gamma$ range of 0.5-1.5 ensure effective alignment, with the optimal configuration of (0.5, 0.5) marked in red that also works well for HPSv2 dataset.

#### Can FiFA Reduce Harmful Contents?

In this section, we evaluate whether FiFA can prevent models from generating harmful images by considering text quality.
To estimate the harmfulness, we generated 200 images from three neutral prompts about “woman” and “girl” and manually labeled the harmfulness of each image (see Appendix D for more details).
As shown in Figure 7(b), when fine-tuned on the entire Pick-a-Pic v2 dataset, the harmfulness of images generated by the fine-tuned model increases significantly, showing at least a $30\%$ increase for all three prompts
compared to those produced by the pretrained model.
This clearly demonstrates that RLHF on large-scale human datasets does not always align model’s behaviors with human value.
In contrast, images generated by the fine-tuned model using FiFA demonstrate reduced levels of harmfulness compared to the pretrained model, indicating that FiFA effectively enhances model safety.

#### Impact of Text Diversity

To demonstrate the importance of text diversity, we compare samples generated by models trained on subsets of the Pick-a-Pic v2 dataset that either consider only high margin with text quality or also include text diversity, using prompts including “raccoon” with different artistic styles. As shown in Figure 7(c), incorporating diversity leads to a better understanding of concepts like “Vincent van Gogh” and “abstract cubism” compared to models trained without diversity. This demonstrates that adding text diversity improves the generalization of fine-tuned models.

## 5 Conclusion

In this paper, we propose FiFA, a new automated data filtering approach to efficiently and effectively fine-tune diffusion models using human feedback data with a DPO objective. Our approach involves selecting data by solving an optimization problem that maximize preference margin, which is calculated by a proxy reward model, text quality and text diversity. In our experiments, the model trained using FiFA, utilizing less than $1\%$ of GPU hours, outperforms the model trained on the full dataset in both automatic and human evaluations across various models and datasets.

## Limitations

Although our proposed FiFA demonstrates its effectiveness and efficiency, we validate our method primarily using the DPO objective among various alignment methods such as policy gradient approaches or those employing reward gradients. It would also be meaningful to extend our algorithm to fit diverse algorithms and other modalities.

## Ethics Statement

Although text-to-image diffusion models are showing remarkable performance in creating high-fidelity images, they may generate harmful content, both intentionally and unintentionally, as the models do not always align well with the text prompts. Therefore, using text-to-image diffusion models requires extra caution.

This concern also applies to our approach. Despite the impressive performance of the model when using FiFA, text-to-image models can still generate harmful, hateful, or sexual images. Although we mitigate this risk by filtering for text quality, as evidenced by improvements over models trained on the full dataset, the inherent issues of pretrained models can still arise in our models. We strongly recommend that users exercise caution when using models trained with our methods, considering the potential risks involved. Moreover, we will open-source the model, along with the safety filtering tool and guidelines for using our model.

## Acknowledgements

We thank Joonkee Kim for providing extensive feedback on our paper. We also express our gratitude to Gihun Lee for engaging in discussions during the design of our method.

## Appendix A Implementation details

Here, we explain the precise implementation details of FiFA. An overview of our hyperparameters is presented in Table 2. For the final models of the main experiments, we update the SD1.5 models for 1000 steps and the SDXL models for 100 steps. We apply warmup steps of 10 for SD1.5 and 5 for SDXL. We adopt the Adam optimizer for SD1.5 and Adafactor for SDXL to save memory consumption. We set the learning rate to $1\mathrm{e}{-7}$ for SD1.5 and $2\mathrm{e}{-8}$ for SDXL. We use an effective batch size of 128 for both models by differentiating the batch size and accumulation steps. A piecewise constant learning rate scheduler is applied, which reduces the learning rate at certain steps.

For the full training of SD1.5, we use warmup steps of 500 and then use a constant scheduler. To calculate the GPU hours for SDXL, we calculate the GPU hours for the first 20 steps, where the time spent for each step becomes constant, and then multiply that number to match the 1000 steps on which the released version of SDXL is trained.

**Table 2: Hyperparameters for SD1.5 and SDXL**
| Hyperparameters | SD1.5 | SDXL |
| --- | --- | --- |
| Update Steps | 1000 | 100 |
| Warmup Steps | 10 | 5 |
| Optimizer | Adam | Adafactor |
| Learning Rate | $1\mathrm{e}{-7}$ | $2\mathrm{e}{-8}$ |
| Learning Rate Scheduler | piecewise constant | piecewise constant |
| Learning Rate Scheduler Rule | 1:200,0.25:400,0.1 | 1:200,0.1 |
| Batch Size | 8 | 1 |
| Accumulation Steps | 8 | 64 |
| Effective Batch Size | 128 | 128 |

## Appendix B DPO for diffusion models

#### Direct Preference Optimization

Given a reward model trained with BT modelling of Eq. (1), the typical approach to increasing the reward is to utilize reinforcement learning to maximize the reward. This often incorporates a KL regularization term with reference distribution $p_{\text{ref}}$, which can be formulated as follows:

$$ $\underset{p_{\theta}}{\max}\mathbb{E}_{\mathbf{x_{0}}\sim p_{\theta}(\mathbf{x _{0}}|\mathbf{c}),\mathbf{c}\sim D_{c}}[r\mathbf{(}x_{0},c)-\beta\mathbb{D}_{ KL}(p_{\theta}(\mathbf{(x_{0}|c)}\parallel p_{\text{ref}}\mathbf{(x_{0}|c)}].$ (7) $$

Direct Preference Optimization (DPO) [Rafailov et al., ] is a method that directly aligns the model without reward training by integrating the reward model training and RL training stages into one. This approach leverages the insight that the optimal policy of the model in Eq. (7) can be represented by a reward function. Hence, the optimal solution $p_{\theta}^{\star}$ can be written as:

$$ $p_{\theta}^{\star}(\mathbf{x_{0}}|\mathbf{c})=p_{\text{ref}}\mathbf{(x_{0}|c)} \cdot\exp(r\mathbf{(x_{0},c)}/\beta)/Z(\mathbf{c}),$ (8) $$

where $Z(\mathbf{c})$ is a partition function. Since $Z(\mathbf{c})$ is usually intractable, one cannot directly obtain the optimal policy from this closed-form solution. However, after reformulating the reward in Eq. (8) and incorporating it into the objective function of the BT model (Eq. (1)), intractable part cancel out and the result becomes a tractable function parameterized by the model. The resulting loss becomes:

$$ $\mathcal{L}(\mathbf{\theta})=-\mathbb{E}_{\mathbf{c},\mathbf{x_{0}^{\mathit{w} }},\mathbf{x_{0}^{\mathit{l}}}}\left[\log\sigma\left(\beta\log\frac{p_{\theta} (\mathbf{x_{0}^{\mathit{w}}}|\mathbf{c})}{p_{\text{ref}}(\mathbf{x_{0}^{ \mathit{w}}}|\mathbf{c})}-\beta\log\frac{p_{\theta}(\mathbf{x_{0}^{\mathit{l}} }|\mathbf{c})}{p_{\text{ref}}(\mathbf{x_{0}^{\mathit{l}}}|\mathbf{c})}\right) \right].$ (9) $$

#### Diffusion DPO Objective

For text-to-image generation, the diffusion model outputs $\bm{\epsilon}_{\theta}(\mathbf{x_{0}},\mathbf{c})$, where $\mathbf{c}$ is a text prompt and $\mathbf{x_{0}}$ is a clean image. During inference, the DDIM sampler [Song et al., ] first obtains a point estimate as follows:

$$ $\hat{\mathbf{x}}_{0}=\frac{\mathbf{x}_{t}-\sqrt{1-\bar{\alpha}_{t}}\bm{ \epsilon}_{\theta}(\mathbf{x}_{t},\mathbf{c})}{\sqrt{\bar{\alpha}_{t}}}.$ (10) $$

In order to obtain DPO loss of Eq. 9, one can first observe that estimation error between ground truth $\bm{\epsilon}$ and our model $\bm{\epsilon}_{\theta}$ makes conditional posterior as:

$$ $p_{\theta}(\mathbf{x}_{0}|\mathbf{x}_{t},c)=\frac{1}{(2\pi\sigma_{t}^{2})^{d/2 }}e^{-\frac{1-\bar{\alpha}_{t}}{2\bar{\alpha}_{t}\sigma_{t}^{2}}\|\bm{\epsilon }-\bm{\epsilon}_{\theta}(\mathbf{x}_{t},\mathbf{c})\|_{2}^{2}}.$ (11) $$

Here, one can use the notation $\sigma(t)$ as defined in DDIM, and $d$ is the dimension of the data. Substituting this into Eq. (9), the resulting loss can be reformulated as follows:

$$ $\mathcal{L}(\mathbf{\theta})=-\mathbb{E}_{t,\mathbf{c},\mathbf{x_{0}^{\mathit{ w}}},\mathbf{x_{0}^{\mathit{l}}}}\left[\log\sigma\left(\beta\log\frac{p_{ \theta}(\mathbf{x_{0}^{\mathit{w}}}|\mathbf{x_{t}^{\mathit{w}},\mathbf{c})}}{p _{\text{ref}}(\mathbf{x_{0}^{\mathit{w}}}|\mathbf{x_{t}^{\mathit{w}},\mathbf{c })}}-\beta\log\frac{p_{\theta}(\mathbf{x_{0}^{\mathit{l}}}|\mathbf{x_{t}^{ \mathit{l}},\mathbf{c})}}{p_{\text{ref}}(\mathbf{x_{0}^{\mathit{l}}}|\mathbf{x _{t}^{\mathit{l}},\mathbf{c})}}\right)\right]\quad\quad\quad\quad\leavevmode \nobreak\ \leavevmode\nobreak\$ (12) $$

$$ $=-\mathbb{E}_{t,\mathbf{c},\mathbf{x_{0}^{\mathit{w}}},\mathbf{x_{0}^{\mathit{ l}}}}\left[\log\sigma\left(\beta\log\frac{e^{-\frac{1-\bar{\alpha}_{t}}{2\bar{ \alpha}_{t}\sigma_{t}^{2}}\|\bm{\epsilon}^{\mathit{w}}-\bm{\epsilon}_{\theta} \|_{2}^{2}}}{e^{-\frac{1-\bar{\alpha}_{t}}{2\bar{\alpha}_{t}\sigma_{t}^{2}}\| \bm{\epsilon}^{\mathit{w}}-\bm{\epsilon}_{\text{ref}}\|_{2}^{2}}}-\beta\log \frac{e^{-\frac{1-\bar{\alpha}_{t}}{2\bar{\alpha}_{t}\sigma_{t}^{2}}\|\bm{ \epsilon}^{\mathit{l}}-\bm{\epsilon}_{\theta}\|_{2}^{2}}}{e^{-\frac{1-\bar{ \alpha}_{t}}{2\bar{\alpha}_{t}\sigma_{t}^{2}}\|\bm{\epsilon}^{\mathit{l}}-\bm{ \epsilon}_{\text{ref}}\|_{2}^{2}}}\right)\right]\$ (13) $$

$$ $\begin{split}=-\mathbb{E}_{t,\mathbf{c},\mathbf{x}_{0}^{\mathit{w}},\mathbf{x} _{0}^{\mathit{l}}}\log\sigma\left(\frac{-\beta(1-\bar{\alpha}_{t})}{\bar{ \alpha}_{t}\sigma_{t}^{2}}\right.\left[\left(\|{\bm{\epsilon}^{\mathit{w}}}-{ \bm{\epsilon}}_{\theta}(\mathbf{x}_{t}^{\mathit{w}},t)\|_{2}^{2}-\|\bm{ \epsilon}^{\mathit{w}}-{\bm{\epsilon}}_{\text{ref}}(\mathbf{x}_{t}^{\mathit{w} },t)\|_{2}^{2}\right.\right.\\ -\left.\left.\|\bm{\epsilon}^{\mathit{l}}-{\bm{\epsilon}}_{\theta}(\mathbf{x}_ {t}^{\mathit{l}},t)\|_{2}^{2}+\|\bm{\epsilon}^{\mathit{l}}-\bm{\epsilon}_{ \text{ref}}(\mathbf{x}_{t}^{\mathit{l}},t)\|_{2}^{2}\right]\right).\end{split}$ (14) $$

With the approximation $\frac{1-\bar{\alpha}_{t}}{\bar{\alpha}t}\frac{1}{\sigma_{t}^{2}}=\frac{\sigma{
t+1}^{2}}{\sigma_{t}^{2}}\approx 1$, and by placing the expectation over time $t$ inside the $\log$ term, one can finally recover the Diffusion-DPO loss of Eq. (2):

$$ $\begin{split}\mathcal{L}_{\text{DPO}}(\theta)=-\mathbb{E}_{t,\mathbf{c}, \mathbf{x}_{0}^{\mathit{w}},\mathbf{x}_{0}^{\mathit{l}}}\log\sigma\left(-\beta T \omega(\lambda_{t})\right.\left[\left(\|{\bm{\epsilon}^{\mathit{w}}}-{\bm{ \epsilon}}_{\theta}(\mathbf{x}_{t}^{\mathit{w}},t)\|_{2}^{2}-\|\bm{\epsilon}^{ \mathit{w}}-{\bm{\epsilon}}_{\text{ref}}(\mathbf{x}_{t}^{\mathit{w}},t)\|_{2}^ {2}\right.\right.\\ -\left.\left.\|\bm{\epsilon}^{\mathit{l}}-{\bm{\epsilon}}_{\theta}(\mathbf{x}_ {t}^{\mathit{l}},t)\|_{2}^{2}+\|\bm{\epsilon}^{\mathit{l}}-\bm{\epsilon}_{ \text{ref}}(\mathbf{x}_{t}^{\mathit{l}},t)\|_{2}^{2}\right]\right).\end{split}$ (15) $$

## Appendix C LLM score for text prompts

Figure: (a) LLM scores in Pick-a-Pic v2 dataset
Refer to caption: x14.png

To calculate the LLM score for each text prompt in the Pick-a-Pic v2 training dataset, we use the gpt-3.5-turbo-0125 model via the OpenAI API. Specifically, we instruct the LLM to assign low scores for excessive duplications, typos, and grammar errors. Additionally, low scores are given to prompts that are too simplistic, trivial, or challenging, while learnable prompts are favored. Furthermore, we allocate a score of 0 to filter out NSFW content.

Figure 8(a) and 8(b) show the resulting distribution of the LLM scores for prompts in the Pick-a-Pic v2 and HPSv2 datasets. Following Zheng et al. [] and Sachdeva et al. [], we use the following prompt templates for calculating text quality using LLM:

As demonstrated in Table 3, selected prompts with high LLM Scores generally contain more detailed and diverse expressions, enabling the model to learn a wide range of concepts, while filtered prompts contain simple and short contexts. Moreover, prompts with scores of 0 are extremely violent and crude, which could make models very harmful.

## Appendix D Evaluating harmful contents

To assess the harmfulness of the DPO-trained model on the Pick-a-Pic v2 dataset, we evaluate harmfulness using three neutral prompts: a simple prompt with an action “a woman standing”, a simple prompt with an adjective “a beautiful woman”, and a prompt that is not toxic but neutral, yet has a higher probability of creating sexual images, “a hot girl”. The motivation for using “woman” and “girl” is because user-generated prompts in the Pick-a-Pic v2 dataset frequently contain these keywords.

For each prompt, we generate $200$ images with different seeds from 0 0 to $199$ with the model trained using FiFA and the model trained using the full dataset. For each generated image, we adopt human annotations, with three authors manually labeling each content with a binary label of harmful or not, considering the scale and safety issues when conducted on other humans. We label an image as harmful if it contains NSFW content such as nudity, vulgarity, or any other harmful elements. We employ majority voting to set the final label of each image. Then we calculate the harmfulness rate, the rate of images labeled as harmful out of all images for each prompt.

Figure 9 shows some generated samples for these prompts from different models. As explained in Section 3.2, , using the full dataset exposes the model to harmful content, which can lead to generating extremely harmful images. On the other hand, when trained using FiFA, by considering text quality, we can prevent the models from learning harmful behaviors, thereby ensuring safety.

**Table 3: Examples of selected and filtered text prompts for Pick-a-Pic v2 by LLM score for FiFA.**
| Status | Prompts | LLM score |
| --- | --- | --- |
| Select | a close up of the demonic bison cyborg inside an iron maiden robot wearing royal robe, large view, a surrealist painting by Jean Fouquet and alan bean and Philippe Druillet, volumetric lighting, detailed shadows | 8 |
| Select | a cute halfling woman riding a friendly fuzzy spider while on an adventure, dnd, ttrpg, fantasy | 9 |
| Select | a photo of teddybear and a sunken steamtrain in the jungle river, flooded train, furry teddy misty mud rocks, panorama, headlights Chrome Detailing, teddybear eyes, open door | 8 |
| Filter | A man with a hat | 3 |
| Filter | text that says smile | 4 |
| Filter | BATMAN | 3 |
| Filter | Pixar-style cartoon of Ted Bundy. | 0 |
| Filter | Selfie of a dead family, crude selfie | 0 |

Figure: Figure 9: Examples of generated images by the pretrained SDXL model, the model trained using FiFA, and the model trained on the full dataset with prompts related to women.
Refer to caption: x16.png

## Appendix E Results with different statistical measures

In our experiments on the Pick-a-Pic test set [Kirstain et al., ], including Figure 1(a), we generated four images for each prompt and then averaged the rewards across images and prompts. Here, we will show different statistical measures for aggregating multiple rewards for each prompt: the maximum reward of each prompt (max), the minimum reward of each prompt (min), and the median reward of each prompt (median). Then, we average the reward values across all prompts.

Figure 10 demonstrates the results when using different measures for aggregating rewards. For all measures, the model trained with FiFA significantly increases the PickScore compared to the model with full training, as when aggregated with the mean value. This shows that FiFA increases the quality of images regardless of specific seeds, indicating the robustness and generalizability of FiFA.

Figure: (a) Max PickScore of each step
Refer to caption: x17.png

## Appendix F Training loss and implicit accuracy

Figure 11(a) shows the training loss comparing the training using FiFA and training using the full dataset. Consistent with the main result, FiFA enables much faster training with better convergence, as it decreases the loss rapidly. In contrast, due to the noisy nature, the loss of full training seems hard to converge and eventually increases at some points. Moreover, as shown in Figure 11(b), the implicit reward model is much better trained when trained on the dataset pruned with FiFA. These results demonstrate that FiFA makes training more stable and achieves faster convergence.

## Appendix G Reward-weighted DPO loss

Figure: (a) Training loss
Refer to caption: x20.png

We interpreted labels with low reward margins as noisy preferences. To make the training robust in this situation, Mitchell [] proposes a conservative DPO loss (cDPO), with the assumption that for the probability $\omega\in(0,0.5)$, labels may be flipped. Using Eq. equation 2, cDPO loss is formulated as follows:

$$ $\mathcal{L}_{\text{DPO}}^{\mathcal{C}}(\theta,\mathbf{x}_{0}^{w},\mathbf{x}_{0 }^{l})=(1-\omega)*\mathcal{L}_{\text{DPO}}(\theta,\mathbf{x}_{0}^{w},\mathbf{x }_{0}^{l})+\omega*\mathcal{L}_{\text{DPO}}(\theta,\mathbf{x}_{0}^{l},\mathbf{x }_{0}^{w}).$ $$

Motivated from cDPO, since we have the proxy reward, we can fully rely on the proxy reward by setting the weight $\omega$ based on the reward value, making the new reward-weighted DPO loss. Specifically, we calculate the $\omega$ as follows:

$$ $\omega(\mathbf{x}_{0}^{w},\mathbf{x}_{0}^{l})=\frac{\exp(r_{\phi}(\mathbf{x}_{ 0}^{l})/\mathcal{T})}{\exp(r_{\phi}(\mathbf{x}_{0}^{w})/\mathcal{T})+\exp(r_{ \phi}(\mathbf{x}_{0}^{l})/\mathcal{T})},$ $$

where $\mathcal{T}$ is the temperature, set to $0.01$ in our experiments. We can interpret this loss as incorporating the reward margin in the loss, so that it trains the model to learn based on how much one image is preferred over the other. Figure 11(c) shows the result of applying reward-weighted DPO loss. Even with weighted loss, FiFA shows superior performance with a rapid increase in reward values compared to the other baselines. Although choosing the data based on high reward margin is also effective, as evidenced by the performance of HM, FiFA shows a larger gap as training progresses with HM, implying the significance of the other two components even when incorporating the reward into the loss. The result demonstrates that FiFA aids training even when we modify the loss to fully depend on the rewards, showing its applicability to DPO-based loss.

## Appendix H Sample Analysis of selected data using FiFA

Here, we analyze some selected or filtered samples by FiFA on either HPSv2 or Pick-a-pic v2 dataset.
As explained in Section 3.3, we select samples based on high margin, text quality, and text diversity. As shown in Figure 12, the selected samples contain images with clear distinctions, meaningful prompts of appropriate difficulty, and minimal overlap with other prompts. On the other hand, the filtered prompts include images with either high or low text-image alignment, meaningless or random prompts, or prompts highly similar to the selected ones. These examples demonstrate that our objective function in Eq. (5) effectively selects samples that consider a high preference margin and ensure high-quality, diverse text prompt sets, as intended.

Figure: Figure 12: Examples of selected and filtered samples when using FiFA.
Refer to caption: x23.png

## Appendix I Proofs of Theoretical Analysis

In this section, we formally state Theorem 1 with additional details. We first start with assuming linear model assumption for the reward feedback which is stated by following assumption.

###### Assumption 1 (Linear model with noisy feedback) .

For any feature vector $\phi(\mathbf{x},\mathbf{c})\in\mathbb{R}^{d}$ of image $\mathbf{x}$ and text $\mathbf{c}$, there exists unknown parameter for the reward model $\mathbf{\theta}_{\star}\in\mathbb{R}^{d}$ such that a reward $r(\mathbf{x},\mathbf{c})$ is given by following equation:

$$ $r(\mathbf{x},\mathbf{c})=\phi(\mathbf{x},\mathbf{c})^{T}\mathbf{\theta}_{\star }+\eta.$ (16) $$

Here, $\eta$ is a $1$-subgaussian random noise from the reward model, and $\phi:\mathcal{I}\times\mathcal{C}\rightarrow\mathbb{R}^{d}$ is the given feature map which sends text prompt $\mathbf{c}\in\mathcal{C}$ and generated image $\mathbf{x}\in\mathcal{I}$ into d-dimensional latent space.

###### Assumption 1 (Linear model with noisy feedback) .

#### OLS estimator

Suppose we have feature vectors $\phi_{i}(\mathbf{c}):=\phi(\mathbf{x}_{i},\mathbf{c})$ and corresponding observations $r_{i}(\mathbf{x},\mathbf{c})$ from Eq. 16 for $i=1,2,\dots,n$. Then, we can estimate the true parameter $\theta_{\star}$ by following estimator which is known as OLS estimator:

$$ $\hat{\theta}=V^{-1}\sum_{i=1}^{n}r_{i}\phi(\mathbf{x}_{i},\mathbf{c}),$ (17) $$

where design matrix $V:=\sum_{i=1}^{n}\phi_{i}(\mathbf{c})\phi_{i}(\mathbf{c})^{\top}$. For this $\hat{\theta}$, we can obtain following confidence bound for any $\phi_{i}\in\mathbb{R}^{d}$, $\delta\in(0,1)$ as follows (Lattimore and Szepesvári [], Chapter 20):

$$ $\mathbb{P}\left(\langle\hat{\theta}-\theta_{\star},\phi(\mathbf{x},\mathbf{c}) \rangle\geq\sqrt{2\|\phi(\mathbf{x},\mathbf{c})\|_{V^{-1}}^{2}\log\left(\frac{ 1}{\delta}\right)}\right)\leq\delta.$ (18) $$

Above equation implies, we can get better estimate $\hat{\theta}$ by minimizing $|\phi(\mathbf{x},\mathbf{c})\|_{V^{-1}}$ and this problem is called as a G-optimal design:

#### G-optimal design

Let ${\pi}$ be a distribution on the collection of $\phi_{i}(\mathbf{c})$ and $\pi:\mathcal{A}\rightarrow[0,1]$ be a distribution on $\pi$. The goal of G-optimal design is to find an optimal $\pi^{\star}$ which solves to find $\pi$ that minimizes the following objective:

$$ $g(\pi)=\max_{\begin{subarray}{c}(i,\mathbf{c})\end{subarray}}\|\phi_{i}( \mathbf{c})\|_{V(\pi)^{-1}}^{2},$ $$

where, $V(\pi)=\sum_{i,\mathbf{c}}\pi(\phi_{i})\phi_{i}(\mathbf{c})\phi_{i}(\mathbf{c}
)^{\top}$ is a design matrix constructed by the distribution $\pi$. Next, we introducte Kiefer-Wolfowitz theorem which is stated as follows.

###### Theorem 2 (Kiefer-Wolfowitz) .

Following statements are equivalent:

- (i)
$\pi^{\star}$ is a minimizer of $g$.
- (ii)
$\pi^{\star}$ is a maximizer of $f(\pi)=\log\det{V(\pi)}$
- (iii)
$g(\pi^{\star})=d$.

The proof of the theorem can be found in Lattimore and Szepesvári []. We can combine Theorem 2 with Eq. 18 to show that G-optimal design identifies an optimal data configuration that minimizes model prediction error under a limited budget. G-optimal design maximizes $\det V(\pi)$ by selecting $\phi_{i}$’s with diverse directions, implicitly incorporating diversity as an objective. Moreover, we can further explain the benefit of using high reward margin pairs by observing that while Eq. 18 tries to bound the all of the feature vectors uniformly, model model prediction error is most critical in the direction of $\theta_{\star}$. By collecting higher reward margin pairs, we can increase singular values of the design matrix $V(\pi)$ to the direction of $\theta_{\star}$. One can expect this will reduce the model prediction error for good feature vector, which in turn, results in better performance of the model. The further support of the claim can be linked to the literature of best arm identification [Yang and Tan, ].

###### Theorem 2 (Kiefer-Wolfowitz) .

## Appendix J Human evaluation

In this section, we provide detailed information about our human evaluation, briefly explained in Section 4.1. We randomly select 100 prompts for each concept in the HPSv2 benchmark, where the concepts are photo, paintings, anime, and concept art. For the selected 100 prompts, we use 50 prompts to create one survey form and the remaining 50 prompts for another, totaling 8 survey forms with 400 prompts. We assign three different annotators to each survey form with a total of 24 annotators.

In the case of human evaluations, we take extreme care to ensure that no harmful content is included on the survey form. All the authors cross-check the content to prevent any harmful material from being exposed to others. Moreover, we do not collect any personal information from participants, including their name, email, or any other identifying details. Instead, all results are gathered from anonymous users. To ensure this, we cross-check all options, remove any questions related to identity, and select options that do not collect any data (e.g., email) from the survey form. Additionally, among the eight test sets, each participant is assigned only one, making it unnecessary to collect private information such as an ID.

We have informed the participants about the purpose of our research, potential risks, and their right to withdraw their data. For the benefit of the participants, we provided a payment of $15\$$ for each participant, where the evaluation took less than an hour. The instruction we provided to the participants is written as below:

An example of the survey form is illustrated in Figure 13.

Figure: Figure 13: Example survey form for the human evaluation. We ask three questions about overall quality (general preference), image-only assessment, and text-image alignment. The images from FiFA and the full dataset are randomly assigned to the left or right. The annotators should choose either left or right, but they can also choose the tie option.
Refer to caption: x24.png

## Appendix K Related work

### K.1 Aligning text-to-image diffusion models

Fine-tuning text-to-image diffusion models using human feedback datasets has proven to be an effective way to improve large pretrained models. Early approaches are based on supervised fine-tuning, involving reward-weighted loss [Lee et al., ], or rejection-sampling based methods Dong et al. [], Sun et al. [].
Policy gradient approaches are also proposed Black et al. [], Fan et al. [] to further improve the models with online learning.
However, these methods are highly ineffective and severely limited to a small scale [Deng et al., , Wallace et al., ].
To address these issues, some methods directly use reward backpropagation [Clark et al., , Prabhudesai et al., ], but these approaches are unstable and often lead to reward overoptimization [Zhang et al., ].
Diffusion-DPO [Wallace et al., ], which directly utilizes human preferences, enables large-scale training without reward hacking, demonstrating its effectiveness for alignment along with its variants [Deng et al., , Yang et al., ].
In this paper, we focus on Diffusion-DPO as it represents one of the most effective state-of-the-art methods for alignment using human feedback.

### K.2 Data Filtering For Deep Neural Networks

There have been attempts to improve the efficiency of training by selecting a core subset from the entire training data.
Data pruning [Raju et al., , Yang et al., , He et al., , Tan et al., ] involves removing unnecessary or less important data from a dataset to reduce its size and enhance efficiency.
Coreset selection [Xia et al., , Mirzasoleiman et al., , Gupta et al., , Chai et al., ] aims to select a small subset that retains the original dataset’s statistical properties.
Data distillation [Cazenavette et al., , Nguyen et al., , Bohdal et al., ] and data condensation [Liu et al., , Kim et al., , Wang et al., ] generate small synthetic training data from the entire dataset that can achieve the learning effect of the entire data.
However, these approaches usually estimate uncertainty based on gradients or loss, often do not align well with diffusion models, as calculating the loss or gradient for diffusion models is time-consuming.
In the diffusion domain, Emu [Dai et al., ] proposes a high-quality, small dataset; training on this dataset can lead to much better image quality and text-image alignment.
However, this approach is not applicable for large-scale, as it requires significant human effort.
Unlike these approaches, we propose an automated data filtering framework that minimizes human involvement.

## Appendix L More examples

Figure 14-17 demonstrate more randomly selected examples generated using the Pick-a-Pic test set, PartiPrompt, and HPSv2 benchmark.

Figure: Figure 14: More example images generated using the different models.
Refer to caption: x25.png

Figure: Figure 15: More example images generated using the different models.
Refer to caption: x26.png

Figure: Figure 16: More example images generated using the different models.
Refer to caption: x27.png

Figure: Figure 17: More example images generated using the different models.
Refer to caption: x28.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please refer to Weaknesses.
soundness: 3
strengths: 1. The paper propose the FiFA algorithm, which leverages three core metrics—preference margin, text quality, and text diversity—to optimize data filtering automatically. This approach effectively addresses noise in human feedback datasets and improves the fine-tuning of diffusion models, particularly for large-scale datasets.
2. The paper is of high quality, with well-designed and comprehensive experiments, including several ablation and comparative studies that strongly support the effectiveness of FiFA in enhancing training efficiency and image quality.
3. The structure of the paper is clear and well-organized. Key concepts, such as preference margin, text quality, and text diversity, are clearly defined, making the methodology accessible.
summary: This paper introduces FiFA, an automated data filtering algorithm designed to optimize fine-tuning of text-to-image diffusion models, aligning model behavior more effectively with human intent. While human feedback datasets are valuable for model alignment, their large size and high noise levels often hinder convergence. FiFA enhances fine-tuning by automatically filtering data based on an optimization problem that maximizes three key components: preference margin, text quality, and text diversity. Experimental results show that FiFA enhances training speed and achieves better performance.
weaknesses: 1. In the paper, some equations lack corresponding equation numbers.
2. In the introduction, the phrasing around "difficulty of convergence" is inconsistent with the discussion of the iterative training required for diffusion models. It is recommended that the authors clarify the logical flow.
3. In Equation 2, there is an extra left parenthesis "(".
4. When using pre-trained models (e.g., CLIP, BLIP) to calculate the preference margin, how is the validity of the results ensured? Given that pre-trained models are trained on noisy and ambiguous datasets, they may also yield incorrect results.
5. From the ablation study results, the effects of Text quality and Text diversity are not very significant. The authors state, "when combined with a high margin, they outperform the model trained solely on margin, highlighting the importance of both components," but where are the results? Is the improvement due solely to the higher margin?

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. The proposed filtering algorithm systematically narrows down the human feedback dataset to a subset that is optimal for model fine-tuning. As a general approach, further discussion on the generalizability of this filtering approach could enrich the analysis, such as how it may integrate with other alignment frameworks like RLHF and DPO-based methods. In addition, expanding the range of comparative methods would strengthen the evaluation.

2. To clarify the novelty of this approach, the specific roles of preference margin and the quality/diversity metrics for text prompts could be further justified. Detailing the design motivations behind these components and their interdependencies would clarify their contributions to the model’s overall performance.

3. DPO requires extensive high-quality preference data, which can be costly and difficult to obtain. The accuracy of preference data is essential, as low-quality feedback may lead to biased or suboptimal model behavior. There appears to be some ambiguity in the statement regarding dataset preparation: "To ensure safety, we manually filter out some harmful text prompts from these test prompts, resulting in 446 unique prompts." It seems that an additional manual filtering step was applied before evaluating the proposed algorithm’s ability to handle harmful prompts. Clarifying this step’s rationale and how it affects the filtering method’s efficacy would add clarity.

4. DPO-based approaches can sometimes narrow the scope of outputs, potentially limiting diversity. To validate the claimed advantage of this filtering method in maintaining diverse outputs, more empirical evidence should be presented. Additionally, a comparison with online DPO and recent DPO variants would help contextualize the findings.

5. More qualitative evidence on how the proposed approach reduces training costs would be valuable, particularly with concrete examples or case studies showing the efficiency gains obtained through this filtering algorithm.

6. Human evaluation is conducted in this work, however, it appears that there is no evidence of human ethics approval, despite it potentially being a low-risk case.
soundness: 3
strengths: Please see questions
summary: This work presents a novel approach to fine-tuning text-to-image diffusion models using human feedback data filtered through an automated algorithm. The proposed methodology optimizes the fine-tuning process by selecting a subset of the available human feedback based on a preference margin criterion, enhancing the reward value while considering both prompt quality and diversity to maintain robustness and mitigate potential harmful content.
weaknesses: Please see questions

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: How about the effectiveness of the proposed approaches on other datasets with different preference models?

I also want to the win rate of the proposed approaches compared with only the base model or DPO.
soundness: 2
strengths: 1.	The motivation of filtering the human feedback data is reasonable. It is well-known that the training of diffusion is very cost. High quality data would contribution both the effectiveness and efficiency of the model.

2.	The paper writing is great.
summary: In this paper, the authors aim to improve aligning text-to-image diffusion models from the perspective of filtering human feedback data. Specifically, they select the data pairs by maximizing three components: preference margin, text quality, and text diversity. For each component, they design an optimization objective. Finally, several experiments have been conducted to verify the contribution of each component
weaknesses: 1.	The technical contribution is relatively small. In my opinion, the proposed approaches for filtering data are travel and nature. In addition, as for preference margin, I believe that it is better to maximize preference margin in a limited range, while a very large margin would provide difficulty for optimization. 

2.	Only the pick-a-pic dataset is used in the experiment, which is highly related to Pick Score. Some other datasets should be involved to verify the generalization. For example, the authors can use the HPS score to compute the preference margin, even on the same pick-a-pic dataset.

3.	It is also significant to show the pair-wise human evaluation in Figure 4.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: The authors highlighted that the method can achieve good results with just 0.5% of the data. Do you have results showing how well FiFA filtering works on say 0.1%, 1%, 5%, 10% of the dataset? It could help us understand how tunable FiFA is.
soundness: 3
strengths: The paper presents a novel approach to data filtering for fine-tuning text-to-image diffusion models. While data pruning and coreset selection are not new concepts in the domain of text-to-image diffusion models (first documented by Meta’s EMU paper), this work focuses on the automation of coreset selection. The combination of preference margin, text quality, and text diversity in a single optimization framework is an effective and reasonable solution in this problem space.

The paper demonstrates effective results across different models (SD1.5 and SDXL) and datasets (Pick-a-Pic v2 and HPSv2), providing robust evidence for their claims. The inclusion of both automatic metrics and human eval provide a complete picture in terms of metrics. There is also some  theoretical analysis provided in the author’s paper.

its most impressive for the authors to achieve high quality alignment with just 0.5% of the data and 1% of the GPU hours. FiFA also demonstrated reduction in harmful content generation, which is critical for these automatic coreset selection method.
summary: This paper proposes FiFA (Filtering for Feedback Alignment), a novel automated data filtering approach for efficiently fine-tuning text-to-image diffusion models using human feedback data. 

The main contributions are:
* An automated data selection method that maximizes: preference margin, text quality and text diversity.
* Formulation of data selection as an optimization problem to find a subset that maximizes these components.
* Empirical evidence showing FiFA's effectiveness across various models and datasets.
weaknesses: I think the biggest issue with this work is that it did not experiment with strong diffusion models like SD3-2B or FLUX models or the Playground models. Those models are much better to start with. It would be very helpful to know if the proposed model can further improve strong models.

Meta-review: This paper proposes a method to automatically select high-quality data for diffusion DPO process, which significantly accelerates the training and reduces the GPU hours. With solid experiments, the paper shows its significance in real-world practice. However, the idea of the paper is a common practice in data selection. Therefore, I recommend to accept this paper as a poster, and I also recommend the authors to add more discussions on data selection works.

Decision: Accept (Poster)