Human-Feedback Efficient Reinforcement Learning for Online Diffusion Model Finetuning

## Abstract

Abstract Controllable generation through Stable Diffusion (SD) fine-tuning aims to improve fidelity, safety, and alignment with human guidance.
Existing reinforcement learning from human feedback methods usually rely on predefined heuristic reward functions or pretrained reward models built on large-scale datasets, limiting their applicability to scenarios where collecting such data is costly or difficult.
To effectively and efficiently utilize human feedback, we develop a framework, HERO,
which leverages online human feedback collected on the fly during model learning. Specifically, HERO features two key mechanisms: (1) Feedback-Aligned Representation Learning , an online training method that captures human feedback and provides informative learning signals for fine-tuning,
and (2) Feedback-Guided Image Generation , which involve generating images from SD’s refined initialization samples, enabling faster convergence towards the evaluator’s intent.
We demonstrate that HERO is 4 × 4\times 4 × more efficient in online feedback for body part anomaly correction compared to the best existing method.
Additionally, experiments show that HERO can effectively handle tasks like reasoning, counting, personalization, and reducing NSFW content with only 0.5K online feedback.

### 1 Introduction

Controllable text-to-image (T2I) generation focuses on aligning model outputs with user intent, such as producing realistic images, e.g., undistorted human bodies, or accurately reflecting the count, semantics, and attributes specified by users.
To tackle this problem, a common paradigm involves fine-tuning latent diffusion models (DM) like Stable Diffusion (SD; Rombach et al., ) using supervised fine-tuning (SFT; Lee et al., ),
which mostly learn from pre-collected, offline datasets.
To further enhance the alignment,
online reinforcement learning (RL) fine-tuning methods
(Fan et al., ; Black et al., ) utilize online feedback that specifically evaluates the samples generated by the model during training.
With such dynamic guidance provided on the fly, these methods demonstrate superior performance on various T2I tasks, such as aesthetic quality improvement.
Yet, these approaches rely on either predefined heuristic reward functions or pretrained reward models learned from large-scale datasets, which could be challenging to obtain, especially for tasks involving personalized content generation (e.g., capturing cultural nuances) or concepts
like specific colors or compositions.

To address the above issue, Yang et al. () introduces D3PO, an alternative method that directly leverages online human feedback for fine-tuning diffusion models.
Instead of learning from heuristic reward functions or pretrained reward models, D3PO leverages the samples generated by the model as well as human annotations collected during training.
With online human feedback, D3PO addresses various tasks, such as distorted human body correction and NSFW content prevention, without requiring a pretrained reward model for each individual task.
However, it still necessitates approximately 5K instances of online human feedback during training (Yang et al., ; Uehara et al., ),
placing a significant burden on the human evaluator and restricting the use of customized fine-tuning to match individual preferences.

Figure: Figure 1: \raisebox{-.9pt} {\footnotesize0}⃝ Online Human Feedback on Generated Images: Each epoch, SD generates a batch of images, evaluated by a human as “good” or “bad”, with the “best” among the “good” selected. The corresponding SD noises and latents are saved. \raisebox{-.9pt} {\footnotesize1}⃝ Feedback-Aligned Representation Learning: Human-annotated images train an embedding map via contrastive learning, converting feedback into continuous representations. These are rated by cosine similarity to one of the “best” images and used to fine-tune SD via DDPO (Black et al., ). \raisebox{-.9pt} {\footnotesize2}⃝ Feedback-Guided Image Generation: New images are generated from a Gaussian mixture centered around the recorded noises of “good” images. This process is repeated until the feedback budget is exhausted.
Refer to caption: x1.png

To further improve the feedback efficiency of T2I alignment using online human feedback, this work proposes a Human-feedback Efficient Reinforcement learning for Online diffusion model fine-tuning framework, dubbed HERO, to efficiently and effectively utilize online human feedback to fine-tune a SD model, as illustrated in Figure 1.
Specifically, we propose two novel components: (1) *Feedback-Aligned Representation Learning*, an online-trained embedding map that creates a representation space that implicitly captures human preferences and provides continuous reward signals for RL fine-tuning,
and (2) *Feedback-Guided Image Generation*, which involve generating images from SD’s refined initialization samples aligned with human intent, for faster convergence to the evaluator’s preferences.

Figure: Figure 2: Result preview. Randomly sampled outputs generated by HERO and baselines given the prompt “photo of one blue rose in a vase” are presented. Successful samples are marked with , and unsuccessful samples are marked with , which fail to accurately capture the specified count (more than one roses), color (non-blue roses), and context (missing vase). HERO successfully captures these aspects, outperforming the baselines.
Refer to caption: extracted/5907427/figures/blue-rose-comparison-v3-min.png

Feedback-aligned representation learning (Figure 1’s \raisebox{-.9pt} {\footnotesize1}⃝) aims to create a representation space that implicitly reflects human preferences, offering continuous reward signals for RL fine-tining.
At each epoch, SD generates a batch of images,
and a human evaluator classifies the images as “good” or “bad”,
selecting one “best” image from the “good” set.
The latents of the human-annotated images are then employed to train an embedding map through contrastive learning (Chen et al., ), aiming to develop a feedback-aligned representation space.
By calculating the cosine similarity to the “best” representation vector in the learned representation space, we obtain a continuous evaluation for each latent.
Subsequently, we utilize the computed similarity as continuous reward signals to fine-tune SD via LoRA (Hu et al., ).

After fine-tuning the SD for the first iteration, our feedback-guided image generation (Figure 1’s \raisebox{-.9pt} {\footnotesize2}⃝) samples a new batch of images from a Gaussian mixture centered on the stored “good” and “best” initial noises from the previous iteration.
This process facilitates the generation of images that align with human intentions better than random initial noises, thereby enhancing the efficiency of fine-tuning.
HERO effectively achieves controllable T2I generation with minimal online human feedback through iterative feedback-guided image generation, feedback-aligned representation learning, and SD model finetuning.

We conduct extensive experiments on various T2I tasks to compare HERO with existing methods.
The experimental results show that HERO can effectively fine-tune SD to reliably follow given text prompts with $4\times$ fewer amount of human feedback compared to D3PO (Yang et al., ).
On the other hand, the results show that these tasks are difficult to solve through prompt enhancement (Winata et al., ) or fine-tuning approaches, e.g., DreamBooth (Ruiz et al., ), that rely on a few reference images (Gal et al., ).
Figure 2 presents a preview of the results.
Extensive ablation studies verify the effectiveness of our proposed feedback-aligned representation learning and the technique of generating images from refined noises.
Additionally, we show that the model fine-tuned by HERO demonstrates transferability to previously unseen inference prompts,
showcasing that the desired concepts were acquired by the model.

### 2 Related Works

Recent research has explored controllable generation with SD for tasks
like T2I alignment (Black et al., ; Prabhudesai et al., ), conceptual generation (Yang et al., ; Zhong et al., ), correcting generation flaws (Zhang et al., ), personalization (Gal et al., ; Ruiz et al., ) and removing NSFW content (Gandikota et al., ; Kumari et al., ; Lu et al., ).

Supervised fine-tuning.
DreamBooth (DB; Ruiz et al., ) and Textual Inversion (Gal et al., ) take images as input and fine-tunes SD via supervised learning to learn the specific subject present in the input images.
However, such methods require reference images, limiting their applicability to general T2I tasks, such as
conceptual generation,
e.g., emotional image content generation (Yang et al., ),
or accurately reflecting user-specified counts, semantics, and attributes (Lin et al., ).
On the other hand,
Prabhudesai et al. (); Gandikota et al. (); Xu et al. (); Clark et al. () use pretrained reward models to calculate differentiable gradients for SD fine-tuning.
However, such pretrained models are not always accessible for tasks of interest, and moreover, these methods cannot directly utilize human feedback, which is non-differentiable.

RL fine-tuning.
Various methods have explored incorporating non-differentiable signals, such as human feedback, as rewards to fine-tune SD using RL.
For example, DDPO (Black et al., ) uses predefined reward functions for tasks like compressibility, DPOK (Fan et al., ) leverages feedback from an AI model trained on a large-scale human dataset, and SEIKO (Uehara et al., ) obtain rewards from custom reward functions trained from extensive feedback datasets.
Yet, these methods require a predefined reward function or reward model, which can be difficult to obtain for tasks that involve generating personalized content (e.g., reflecting cultural nuances) or abstract concepts, such as specific colors or compositions (Amadeus et al., ; Kannen et al., ).

Direct preference optimization (DPO).
Diffusion-DPO (Wallace et al., ) applies DPO (Rafailov et al., ) to directly utilize preference data to fine-tune SD, eliminating the need for predefined rewards.
Despite encouraging their results, such a method requires a large-scale pre-collected human preference dataset
e.g., Diffusion-DPO uses the Pick-a-Pic dataset with 851K preference pairs,
making it costly to collect and limiting its applicability to various tasks, including personalization.
Instead of leveraging offline datasets, D3PO (Yang et al., ) uses online human feedback collected on-the-fly during model training for DPO-style finetuning of SD.
It demonstrates success in tasks such as body part deformation correction and content safety improvement while avoiding the demand for large-scale offline datasets.
However, the amount of human feedback required for D3PO is still high, requiring 5-10k feedback instances per task, which motivates us to develop a more human-feedback-efficient framework.

### 3 Preliminaries

###### Stable Diffusion (SD)

operates in two stages. First, an autoencoder
compresses images ${\mathbf{x}}$ from pixel space into latent representations ${\mathbf{z}}_{0}$, which can later be decoded back to pixel space. Second, a diffusion model (DM) is trained to model the distribution of these latent representations conditioned on text ${\mathbf{c}}$. The forward diffusion process is defined as $p({\mathbf{z}}_{t}|{\mathbf{z}}_{0}):=\mathcal{N}({\mathbf{z}}_{t};\alpha_{t}{
\mathbf{z}}_{0},\sigma_{t}^{2}\mathbf{I})$, where $\alpha_{t}$ and $\sigma_{t}$ are pre-defined time dependent constants for $t\in[0,T]$. Both the forward transition kernel $p({\mathbf{z}}_{t}|{\mathbf{z}}_{t-1},{\mathbf{c}})$ and the backward conditioned transition kernel $p({\mathbf{z}}_{t-1}|{\mathbf{z}}_{t},{\mathbf{c}},{\mathbf{z}}_{0})$ are Gaussian with closed-form expressions.
The DM is trained to predict the clean sample ${\mathbf{z}}_{0}$ using a neural network $\hat{\mathbf{z}}_{\phi}({\mathbf{z}}_{t},t,{\mathbf{c}})$, denoising the noisy sample ${\mathbf{z}}_{t}$ at time $t$:

$$ $\displaystyle p_{\phi}({\mathbf{z}}_{t-1}|{\mathbf{z}}_{t},{\mathbf{c}}):=p \big{(}{\mathbf{z}}_{t-1}|{\mathbf{z}}_{t},{\mathbf{c}},{\mathbf{z}}_{0}:=\hat {\mathbf{z}}_{\phi}({\mathbf{z}}_{t},t,{\mathbf{c}})\big{)}$ $$

by optimizing the following objective:

$$ $\displaystyle\min_{\phi}\mathbb{E}_{{\mathbf{z}}_{0},{\mathbf{c}},\bm{\epsilon },t}\Big{[}\left\lVert\hat{\mathbf{z}}_{\phi}(\alpha_{t}{\mathbf{z}}_{0}+ \sigma_{t}\bm{\epsilon},t,{\mathbf{c}})-{\mathbf{z}}\right\rVert_{2}^{2}\Big{] },\quad\bm{\epsilon}\sim\mathcal{N}(\bm{0},\mathbf{I}).$ $$

At inference, random noise ${\mathbf{z}}_{T}$ is sampled from a prior and iteratively denoised using samplers like DDPM (Ho et al., ) and DDIM (Song et al., ) to obtain a latent code ${\mathbf{z}}_{0}$, which is then decoded into an image. This denoising and decoding process forms a text-to-image generative model, with random noise ${\mathbf{z}}_{T}$ sampled from a prior and ${\mathbf{c}}$ as the user-provided prompt.

###### Denoising Diffusion Policy Optimization (DDPO)

formulates the denoising process of diffusion models as a multi-step Markov decision process. With this formulation, one can make direct Monte Carlo estimates of the reinforcement learning objective.
Given a denoising trajectory $\{{\mathbf{z}}_{T},{\mathbf{z}}_{T-1},...,{\mathbf{z}}_{0}\}$, the denoising diffusion RL update is defined as the following:

$$ $\nabla_{\phi}\mathcal{L}_{\text{DDRL}}(\phi)=\mathbb{E}\biggl{[}\sum^{T}_{t=0} \nabla_{\phi}\log p_{\phi}({\mathbf{z}}_{t-1}|{\mathbf{z}}_{t},{\mathbf{c}})r( {\mathbf{z}}_{0},{\mathbf{c}})\biggr{]},$ (1) $$

where $\phi$ is the diffusion model, and $r({\mathbf{x}}_{0},{\mathbf{c}})$ is the received reward computed according the output image ${\mathbf{x}}_{0}$ and the input prompt ${\mathbf{c}}$.
Based on the above update, DDPO further utilizes the importance sampling estimator (Kakade & Langford, ) and the trust region clipping from Proximal Policy Optimization (PPO; Schulman et al., ) to perform multiple steps of optimization while maintaining the diffusion model $\phi$ not deviating too far from the previous iteration $\phi_{\text{old}}$. The DDPO update is defined as the following:

$$ $\nabla_{\phi}\mathcal{L}_{\text{DDPO}}(\phi)=\mathbb{E}\biggl{[}\sum^{T}_{t=0} \frac{p_{\phi}({\mathbf{z}}_{t-1}|{\mathbf{z}}_{t},{\mathbf{c}})}{p_{\phi_{ \text{old}}}({\mathbf{z}}_{t-1}|{\mathbf{z}}_{t},{\mathbf{c}})}\nabla_{\phi} \log p_{\phi}({\mathbf{z}}_{t-1}|{\mathbf{z}}_{t},{\mathbf{c}})r({\mathbf{z}}_ {0},{\mathbf{c}})\biggr{]}.$ (2) $$

### 4 Problem Setup and the Proposed Method

Given a user-specified text prompt, our goal is to fine-tune SD to generate images that align with the prompt by learning from human feedback guidance.
In this paper, we focus on challenging T2I tasks that require spatial reasoning, counting, feasibility understanding, etc., as detailed in Table 1.
To efficiently and effectively utilize online human feedback, we propose a human-feedback efficient reinforcement learning for online diffusion model fine-tuning framework, dubbed HERO, as illustrated in Figure 1.
Feedback-Aligned Representation Learning (Figure 1 \raisebox{-.9pt} {\footnotesize1}⃝) makes efficient use of limited human feedback by converting discrete feedback to informative, continuous reward signals.
In addition, Feedback-Guided Image Generation (Figure 1 \raisebox{-.9pt} {\footnotesize2}⃝) leverages human-preferred noise latents from previous iterations and encourages SD outputs to align more quickly with human intention, further improving sample efficiency.

#### 4.1 Online Human Feedback

In the first iteration of HERO, we generate synthetic images $\mathcal{X}$ from a batch of random noises $\mathcal{Z}_{T}$ sampled from SD’s prior distribution $\pi_{\text{HERO}}({\mathbf{z}}_{T}):=\mathcal{N}({\mathbf{z}}_{T};\bm{0},
\mathbf{I})$ using DDIM (Song et al., ; Ho et al., ).
For each ${\mathbf{z}}_{T}\in\mathcal{Z}$, the sampling trajectories are denoted as $\{{\mathbf{z}}_{T},{\mathbf{z}}_{T-1},\cdots,{\mathbf{z}}_{0}\}$, and each ${\mathbf{z}}_{0}$ is decoded to an image for human evaluation.
A human evaluator reviews $\mathcal{X}$, selects the “good” images $\mathcal{X}^{+}$, and labels the remaining images as $\mathcal{X}^{-}$.
To obtain a gradation among all “good” images and all “bad” images by representation learning, we ask the evaluator to identify the “best” image in $\mathcal{X}^{+}$, denoted as ${\mathbf{x}}^{\text{best}}$.
The details of our feedback-aligned representation learning are discussed in the following section and we store the following for future use:
the sets of images $\mathcal{X}$, $\mathcal{X}^{+}$, $\mathcal{X}^{-}$, ${\mathbf{x}}^{\text{best}}$; their corresponding SD’s clean latents $\mathcal{Z}_{0}$, $\mathcal{Z}^{+}_{0}$, $\mathcal{Z}^{-}_{0}$, ${\mathbf{z}}^{\text{best}}_{0}$ from which they are decoded; and their initial noises (at time $T$) $\mathcal{Z}_{T}$, $\mathcal{Z}^{+}_{T}$, $\mathcal{Z}^{-}_{T}$, ${\mathbf{z}}^{\text{best}}_{T}$ used in SD’s sampling.

#### 4.2 Feedback-Aligned Representation Learning

HERO fine-tunes SD with minimal online human feedback by learning representations via a contrastive objective that captures discrepancies between the best SD’s clean latent ${\mathbf{z}}_{T}^{\text{best}}$, positive $\mathcal{Z}^{+}_{0}$, and negative $\mathcal{Z}^{-}_{0}$ SD’s clean latents (Section 4.2.1).
By calculating similarity to the best image’s representation, we use these similarity scores as continuous rewards for RL fine-tuning (Section 4.2.2).
This approach bypasses reward model training by directly converting human feedback into learning signals, avoiding the need for over 100k training samples typically required to train a reward model for unseen data (Wallace et al., ; Rafailov et al., ).

##### 4.2.1 Learning Representations

To learn a representation space of $\mathcal{Z}_{0}$ aligned with human feedback, we build on the contrastive learning framework of Chen et al. ().
We design an embedding network $E_{\theta}(\cdot)$ to map $\mathcal{Z}_{0}$ into the representation space, followed by a projection head $g_{\theta}(\cdot)$ for loss calculation.
Triplet margin loss is applied to the projection head’s output:

$$ $\displaystyle\mathcal{L}(\theta;{\mathbf{z}}^{\text{best}}_{0},\mathcal{Z}^{+} _{0},\mathcal{Z}^{-}_{0})=\mathbb{E}_{{\mathbf{z}}^{\text{good}}_{0}\sim \mathcal{Z}^{+}_{0},{\mathbf{z}}^{\text{bad}}_{0}\sim\mathcal{Z}^{-}_{0}}\max \biggl{\{}$ $\displaystyle S\Bigl{(}g_{\theta}\bigl{(}E_{\theta}({\mathbf{z}}^{\text{best}} _{0})\bigr{)},g_{\theta}\bigl{(}E_{\theta}({\mathbf{z}}^{\text{good}}_{0}) \bigr{)}\Bigr{)}$ (3) $\displaystyle-$ $\displaystyle S\Bigl{(}g_{\theta}\bigl{(}E_{\theta}({\mathbf{z}}^{\text{best}} _{0})\bigr{)},g_{\theta}\bigl{(}E_{\theta}({\mathbf{z}}^{\text{bad}}_{0})\bigr {)}\Bigl{)}+\alpha,0\biggr{\}}.$ $$

$E_{\theta}({\mathbf{z}}^{\text{best}}_{0})$ serves as the anchor in the contrastive loss, with $S(\cdot,\cdot)$ representing the similarity score (using cosine similarity) and $\alpha$ as the triplet margin set to $0.5$.
By using the best image in the triplet loss, we obtain a gradation within positive and negative categories based on the distance to the best sample. With the learned representation $E_{\theta}({\mathbf{z}}_{0})$ for ${\mathbf{z}}_{0}\in\mathcal{Z}_{0}$, we can compute continuous rewards for RL fine-tuning.

##### 4.2.2 Similarity-based Rewards Computation

After training the embedding $E_{\theta}(\cdot)$ on the current batch of human feedback,
reward values are computed as the cosine similarity in the learned representation space between each
$E_{\theta}({\mathbf{z}}_{0})$ for ${\mathbf{z}}_{0}\in\mathcal{Z}_{0}$ and $E_{\theta}({\mathbf{z}}^{\text{best}}_{0})$:

$$ $R({\mathbf{z}}_{0})=\frac{E_{\theta}({\mathbf{z}}_{0})\cdot E_{\theta}({ \mathbf{z}}^{\text{best}}_{0})}{\max\big{\{}\left\lVert E_{\theta}({\mathbf{z} }_{0})\right\rVert_{2}\left\lVert E_{\theta}({\mathbf{z}}^{\text{best}}_{0}) \right\rVert_{2},\delta\big{\}}}\quad\text{for each }{\mathbf{z}}_{0}\in \mathcal{Z}_{0},$ (4) $$

where $\delta=1\times 10^{-8}$ to avoid zero division. By using the learned representations to convert simple (discrete) human feedback into continuous reward signals, we avoid the need for a large pretrained reward model or costly training of such a model.

Besides the “similarity-to-best” design, we also consider a “similarity-to-positives” design,
which uses the similarity between an image and the average of all “good” images in the learned representation space.
We choose the “similarity-to-best” design for its superior performance. Further discussion is available in Section 5.3.1.

##### 4.2.3 Diffusion Model Finetuning

DDPO fine-tunes SD by reweighting the likelihood with reward values.
For a noise latent ${\mathbf{z}}_{T}\in\mathcal{Z}_{T}$ and its sampling trajectory $\{{\mathbf{z}}_{T},{\mathbf{z}}_{T-1},\cdots,{\mathbf{z}}_{0}\}$, we incorporate the reward $R({\mathbf{z}}_{0})$ from Eq. (4) into the DDPO update rule in Eq. (2) to fine-tune the SD model $\phi$.
To reduce costly gradient computations, we adopt LoRA (Hu et al., ) for fine-tuning.

#### 4.3 Feedback-Guided Image Generation

After the previous iteration of fine-tuning, we propose feedback-guided image generation to facilitate the fine-tuning process by generating images that reflect human intentions.
We sample the noise latents for a new batch of images from the Gaussian mixture with means centered around the human-selected “good” $\mathcal{Z}^{+}_{T}$ and “best” ${\mathbf{z}}_{T}^{\text{best}}$ SD noise latents from the previous iteration, with a small variance $\varepsilon_{0}$.
Specifically, we sample the noise latent ${\mathbf{z}}_{T}$ from the distribution $\pi_{\text{HERO}}({\mathbf{z}}_{T})$ defined as:

$$ $\pi_{\text{HERO}}({\mathbf{z}}_{T})=\begin{cases}\mathcal{N}({\mathbf{z}}_{T}; \bm{0},\mathbf{I}),&\text{first iteration}\\ \beta\mathcal{N}({\mathbf{z}}_{T};{\mathbf{z}}^{\text{best}}_{T},\varepsilon_{ 0}^{2}\mathbf{I})+\frac{(1-\beta)}{|\mathcal{Z}^{+}_{T}|}\sum_{{\mathbf{z}}^{ \text{good}}_{T}\in\mathcal{Z}^{+}_{T}}\mathcal{N}({\mathbf{z}}_{T};{\mathbf{z }}^{\text{good}}_{T},\varepsilon_{0}^{2}\mathbf{I})&\text{otherwise.}\end{cases}$ (5) $$

Here, we introduce a hyperparameter best image ratio $\beta$ to control the proportion of the next batch sampled from the “best” image noise latent.
We find that leveraging ${\mathbf{z}}^{\text{best}}_{T}$ with a larger $\beta$
can accelerate training convergence to evaluator preferences but may reduce the diversity or the converged accuracy.
The above tradeoff can be controlled by the best image ratio $\beta$.
We generally set $\beta=0.5$ to balance these effects.
Further discussion on the best image ratio parameter is in Section 5.3.2.

We remark that since the variance $\varepsilon_{0}$ is small, after a few iterations, samples from $\pi_{\text{HERO}}({\mathbf{z}}_{T})$ still concentrate near the prior $\mathcal{N}({\mathbf{z}}_{T};\bm{0},\mathbf{I})$ at high probability (see Proposition A.1). Also, ${\mathbf{z}}_{T}^{\text{good}}$ and ${\mathbf{z}}_{T}^{\text{best}}$ may retain semantic information about human alignment from ${\mathbf{z}}_{0}^{\text{good}}$ and ${\mathbf{z}}_{0}^{\text{best}}$, as they are connected through the finite-step discretization of the SD sampler (see Proposition A.2). Thus, these validate our proposed $\pi_{\text{HERO}}({\mathbf{z}}_{T})$ as refined initializations for sampling.

Given a new batch of images $\mathcal{X}$ decoded from the clean latents $\mathcal{Z}_{0}$ generated by SD, with corresponding initial noises $\mathcal{Z}_{T}$ sampled from $\pi_{\text{HERO}}({\mathbf{z}}_{T})$ in Eq. (5), the human evaluator provides their evaluation as described in Section 4.1. The process is repeated until the feedback budget is exhausted or the evaluator is satisfied with the generation from $\pi_{\text{HERO}}({\mathbf{z}}_{T})$. After obtaining the fine-tuned SD model $\phi$ and $\pi_{\text{HERO}}({\mathbf{z}}_{T})$ through HERO, we use SD random noises from refined $\pi_{\text{HERO}}({\mathbf{z}}_{T})$ and generate images using any DM sampler (Song et al., ).

### 5 Experimental Results

We demonstrate HERO’s performance on a variety of tasks, including hand deformation correction, content safety improvement, reasoning, and personalization. Many of them cannot be easily solved by the pretrained model, prompt enhancement, or prior methods. A full list of tasks and their success conditions are shown in Table 1. We adopt SD v1.5 (Rombach et al., ) as the base T2I model, using DDIM (Ho et al., ; Song et al., ) with 50 diffusion steps (20 for hand deformation correction for fair comparison to the baselines) as the sampler.

Figure: Figure 3: Hand anomaly correction success rates. Performance of methods except D3PO are average of 8 seeds, where each seed is evaluated on 128 images per epoch. DB, SD-P, and SD-E are DreamBooth, SD-pretrained, and SD-enhanced, respectively.
Refer to caption: extracted/5907427/figures/hand_normalcy_plot_broken_v5.png

We compare HERO to the following baselines:

- •
SD-pretrained prompts the pretrained SD model with the original task prompt shown in Table 1.
- •
SD-enhanced prompts the pretrained SD model with an enhanced version of the prompt generated by GPT-4 (Brown, ; Achiam et al., ).
- •
DreamBooth (DB; Ruiz et al., ) finetunes diffusion models via supervised learning, taking images as input. We use the four best images chosen by the human evaluators as model inputs.
- •
D3PO (Yang et al., ) utilize online human feedback for DPO (Rafailov et al., )-based diffusion model finetuning. Due to the high feedback cost for training, this baseline is considered only for the hand anomaly correction task directly adopted from their work. Success rates are reported as presented in the original paper.

#### 5.1 Hand Deformation Correction

Following the problem setup of D3PO (Yang et al., ), we use the prompt “1 hand” for image generation and use human discretion to evaluate the normalcy of the generated hand images. Parameters such as sampling steps are set to be consistent with D3PO. In each epoch of HERO, feedback on 128 images is collected, and the human evaluator provides a total of 1152 feedback over 9 epochs. Performance of HERO in comparison to the baselines is shown in Figure 3. As shown in Figure 3, the pretrained SD model struggles on this task, with a normalcy rate of 11.9% (SD-pretrained) and 7.5% (SD-enhanced), and DB achieves 28%. D3PO reaches 33.3% normalcy rate at 5K feedback, while HERO achieves a comparable success rate of 34.2% with only 1152 feedback (over 4$\times$ more feedback efficient). The sampled images are shown in Figure 8 in the appendix.

#### 5.2 Demonstration on the Variety of Tasks

**Table 1: Task summary.**
| Task Name | Prompt | Task Categories |
| --- | --- | --- |
| hand | “1 hand” | correction, feasibility |
| blue-rose | “photo of one blue rose in a vase” | reasoning, counting |
| black-cat | “a black cat sitting inside a cardboard box” | reasoning, feasibility, functionality |
| narcissus | “narcissus by a quiet spring and its reflection in the water” | feasibility, homonym distinction |
| mountain | “beautiful mountains viewed from a train window” | reasoning, functionality, personalization |

We further demonstrate the effectivity of HERO on a variety of tasks involving reasoning, correction, feasibility and functionality quality enhancement, and personalization. Tasks are listed in Table 1, and descriptions of task success conditions and task categories are found in Appendix C. For each task, human evaluators are presented with 64 images per epoch and provide a total of 512 feedback over 8 epochs. We report the average and standard deviation of the success rates across three seeds, where success is evaluated on 64 images generated in the final epoch. For methods that require human feedback (DB and HERO), three different human evaluators were each assigned a different seed to provide feedback on. Each evaluator was also responsible for evaluating the success rates of all methods for their assigned seed. Results are shown in Table 2. For all tasks, HERO achieves a success rate at or above 75%, outperforming all baselines. This trend is consistent for all three human evaluators, suggesting HERO’s robustness to individual differences among human evaluators. Sample images generated by SD-pretrained, DB, and HERO are shown in Figure 4 and more results can be found in Figures 9, 10, 11 and 12 in the appendix. While the baselines often struggle in attribute reasoning (e.g., color, count), spatial reasoning (e.g., inside), and feasibility (e.g., reflection consistent with the subject), HERO models consistently capture these aspects correctly.

**Table 2: Task performance. Mean and standard deviation of success rates of different methods on the four tasks. HERO achieves a success rate at or above 75% and outperforms all baselines, demonstrating effectiveness on a variety of tasks.**
| Method | blue-rose | black-cat | narcissus | mountain |
| --- | --- | --- | --- | --- |
| SD-Pretrained | 0.354 (0.020) | 0.422 (0.092) | 0.406 (0.077) | 0.412 (0.063) |
| SD-Enhanced | 0.479 (0.030) | 0.365 (0.134) | 0.276 (0.041) | 0.938 (0.022) |
| DB | 0.479 (0.085) | 0.453 (0.142) | 0.854 (0.092) | 0.922 (0.059) |
| HERO (ours) | 0.807 (0.115) | 0.750 (0.130) | 0.912 (0.007) | 0.995 (0.007) |

Figure: Figure 4: Qualitative results. The randomly generated samples for the four tasks are shown, with denoting successful samples and for failures. In the blue-rose task, the pretrained SD model often omits the vase, while DB generates roses with incorrect color or count. In narcissus, SD frequently fails to capture the subject or produces inconsistent reflections. For black-cat, baseline models exhibit more issues (e.g., the cat’s body penetrating the box). In mountain, baseline images often miss the window frame or depict impossible views. Our fine-tuned models mitigate these issues and show significantly higher success rates across all tasks.
Refer to caption: extracted/5907427/figures/superclass-visuals-v2-min.png

#### 5.3 Ablations

This section presents ablation studies illustrating the roles of each component of HERO. In regards to Feedback-Aligned Representation Learning, we investigate the effects of (1) computation of rewards using learnable feedback-aligned representations and (2) “similarity-to-best” design for reward computation. For Feedback-Guided Image Generation, the effect of best image ratio is explored.

##### 5.3.1 Effect of Feedback-Aligned Representation Learning and Reward Design

**Table 3: Representation learning and reward design ablation.**
| Method | Success rate |
| --- | --- |
| SD-Pretrained | 0.40 |
| HERO-binary | 0.78 |
| HERO-noEmbed | 0.76 |
| HERO-positives | 0.82 |
| HERO | 0.91 |

The effects of using learned feedback-aligned representations and our reward design are investigated through three ablation experiments. Firstly, we demonstrate the benefit of converting discrete human feedback into continuous reward signal by investigating HERO-binary, a variant of HERO using binary rewards for training.
Secondly, we explore the effect of learned representations by
replacing the learned representations in
HERO with SD image latents $\mathcal{Z}_{0}^{+}$
(HERO-noEmbed).
Finally, we explain our choice for the “similarity-to-best” reward design by discussing an alternative reward design using similarity to the average of all $\mathcal{Z}_{0}^{+}$ and $z_{0}^{\text{best}}$ (HERO-positives).
For each setting, we test on the narcissus task with 512 feedback for training and 200 images generated by the finetuned model for success rate evaluation. HERO outperforms all other settings, and results are summarized in Table 3.

Directly using human labels as binary rewards. An intuitive way to extract a reward signal from binary human feedback is to directly convert the feedback into a binary reward. To investigate the effect of similarity-based conversion of human feedback to continuous rewards, we test HERO-binary, a variant where the reward in HERO is replaced with a binary reward. Images labeled as “good” or “best” receive a reward of 1.0, and all other images receive a reward of 0.0. HERO-binary only reaches 78% success rate while HERO reaches 91%. This may be because the continuous rewards contain additional information beneficial for DDPO training: While the binary reward only labels images as “good” or “bad”, the continuous reward additionally captures a gradation of human ratings within the “good” and “bad” categories, supplying additional information such as which “good” images are nearly “best”, and which are barely “good”.

Computing rewards from pretrained image representations.
Experiments with binary rewards showed the benefit of using continuous rewards in the learned representation space. To further understand HERO’s use of feedback-aligned learned representations, we replace the learned representations $E_{\theta}(\mathcal{Z}_{0})$ with SD’s clean latents $\mathcal{Z}_{0}$, obtained by denoising SD’s initial noises $\mathcal{Z}_{T}$, and call this setup HERO-noEmbed. Without embedding map training, $\mathcal{Z}_{0}^{+}$ no longer cluster around $z_{0}^{\text{best}}$, making a “similarity-to-best” reward design impractical.
Thus, we only consider the “similarity-to-positives” reward design for this ablation. While HERO-positives reach 82% success, HERO-noEmbed reaches 76%, suggesting the benefit of learned representations. Training the embedding map additionally offers the “similarity-to-best” reward design option that gives superior performance.

Computing reward as similarity to average of all “good” representations.
The reward in HERO is computed as the similarity to $z_{0}^{\text{best}}$. However, another natural choice is to compute similarity to the average of all $\mathcal{Z}_{0}^{+}$. Comparing this “similarity-to-positives” design to the “similarity-to-best” design employed in HERO, we find that the “similarity-to-best” design achieves 91% success, while the “similarity-to-positives” design reaches 82%. We adopt the “similarity-to-best” design, which empirically gives superior performance.

Figure: Figure 5: Effect of best image ratio $\beta$ evaluated on the black-cat task. Three iterations with different seeds are performed for each setting, and the mean and standard deviation of the success rate are reported separately for clearer visualization. “random” refers to the case where random noise latents are used for sampling (good and best noises latents are not used).
Refer to caption: extracted/5907427/figures/alpha_ablation_separate_plot_v4.png

##### 5.3.2 Effect of Best Image Ratio in Feedback-Guided Image Generation

To investigate the effect of the best image ratio, we compare the performance of the black-cat task for $\beta=0.0,0.5,1.0$. Further, we compare to the case where the images are sampled from random SD noise latents to demonstrate the benefit of using $\mathcal{Z}_{T}^{+}$ and $z_{T}^{\text{best}}$ as initial noises for image generation. Results are shown in Figure 5. Sampling all images from the $z_{T}^{\text{best}}$ ($\beta=1.0$) reaches an average of $70.8\%$ success at the end of the training. However, as the high standard deviation in the initial stage of training suggests, over-exploiting a single “best” noise latent can cause instability in training, potentially causing the model to settle on a suboptimal output. Sampling uniformly from $\mathcal{Z}_{T}^{+}$ and $z_{T}^{\text{best}}$ ($\beta=0.0$) results in a similar success rate as $\beta=1.0$, but is less likely to converge to a suboptimal point. We empirically find that, for our tasks, $\beta=0.5$ results in the highest success rate while avoiding the risks of fully relying on the single “best” noise latent, thus using $\beta=0.5$ for our experiments. When images are sampled from random SD noise latents, the task success rate does not grow significantly slower in the given amount of feedback, demonstrating the benefit of using $\mathcal{Z}_{T}^{+}$ and $z_{T}^{\text{best}}$ for efficient fine-tuning.

#### 5.4 Transferability

While HERO is trained to optimize for a single input prompt, we observe that some personal preferences and general concepts learned from one prompt can generalize to other related prompts in some cases.

Transfer of personal preference. In the mountain task, we observe the transfer of learned individual preferences. Two human evaluators trained two separate models for the mountain task, where one evaluator preferred green scenery while the other preferred snowy scenery. Each evaluator’s trained model as well as the corresponding $\mathcal{Z}_{T}^{+}$ and $z_{T}^{\text{best}}$ are used to generate images for a related task “hiker watching beautiful mountains from the top of a hill”. As shown in Figure 6, the preference for green or snowy scenery transfers to this new task.

Transfer of content safety. To further investigate whether a general concept, such as content safety, learned through one task can transfer to another, we prompt the SD model using the prompt “sexy” and train it to reduce NSFW content in the generated images. The fine-tuned model (as well as the saved $\mathcal{Z}_{T}^{+}$ and $z_{T}^{\text{best}}$) are used to generate images from a set of 14 potentially-unsafe prompts used in D3PO’s content safety task. Utilizing the finetuned model and the saved SD noise latents significantly improves the content safety rate from 57.5% of the pretrained SD model to 87.0%, demonstrating HERO-finetuned model’s potential to transfer a general concept learned from one prompt to a set of related, unseen prompts. Visual results are shown in Figure 7, and the full list of prompts with more results are shown in Figure 13 in the appendix.

Figure: Figure 6: Demonstration of personal preference transferability. Models trained with two distinct personal preferences (green and snowy) generate images that inherit these preferences when prompted with a similar task (“hiker watching beautiful mountains from the top of a hill”).
Refer to caption: extracted/5907427/figures/transfer_mountain_v3-min.png

Figure: Figure 7: Qualitative results for the NSFW content hidden task showcasing transferability of HERO. The images were randomly generated using the potentially unsafe prompt set provided by Yang et al. (). The model is the HERO-finetuned version, trained with the “sexy” prompt to reduce nudity. The safety rate improves from 57.5% (pretrained SD) to 87.0% (HERO), showing HERO’s ability to transfer the concept of safety to unseen, potentially unsafe prompts.
Refer to caption: extracted/5907427/figures/transfer_safety_hidden_v2-min.png

### 6 Conclusion

This work introduces HERO, an RLHF framework for fine-tuning SD using online human feedback. By learning a feedback-aligned representation, we capture implicit human preferences, converting simple human feedback into a continuous reward signal that enhances DDPO fine-tuning. Using human-preferred image noise latents as initial noise further accelerates alignment with preferences. Combining these components, HERO achieves high efficiency in fine-tuning SD, requiring $4\times$ less feedback than the baseline. Additionally, it shows potential for transferring personal preferences and concepts to related tasks.

### Appendix

## 

### Appendix A Theoretical Explanations

In this section, we provide theoretical justifications for the validity of our proposed distribution $\pi_{\mathrm{HERO}}$ in Eq. (5) from two perspectives, refining the initial distribution for human-feedback-aligned generation.

#### A.1 Concentration of Human-Selected Noises in SD’s Prior Distribution

It is known that the initial distribution of SD sampling is typically the standard normal distribution $\mathcal{N}(\mathbf{0},\mathbf{I}_{D})$, which yields a random vector that concentrates around the sphere of radius $\sqrt{D}$ with high probability. In the following proposition, we show that a random vector drawn from our proposed distribution $\pi_{\mathrm{HERO}}$ also concentrates around the sphere of radius $\sqrt{D}$ with high probability, provided that the variance $\varepsilon_{0}>0$ of the Gaussian mixture is sufficiently small. This ensures that the sampling from the refined initial noise provided by $\pi_{\mathrm{HERO}}$ remains consistent with the sampling from the original prior distribution of the SD model.

###### Proposition A.1 (Concentration of $\pi_{\mathrm{HERO}}$ ) .

Let $\pi$ be a Gaussian mixture with each component as $\mathcal{N}(\bm{\mu}_{i},\varepsilon_{0}^{2}\mathbf{I}_{D})$, where each mean $\bm{\mu}_{i}\sim\mathcal{N}(\mathbf{0},\mathbf{I}_{D})$, and $\varepsilon_{0}>0$ is a small constant. Let ${\mathbf{y}}\sim\pi$ be a random vector drawn from $\pi$. Then, for any $\delta>0$, we have the following concentration if $\varepsilon_{0}$ is sufficiently small:

$$ $\displaystyle\mathbb{P}\left(\sqrt{D}(1-\varepsilon_{0})\leq\|{\mathbf{y}}\| \leq\sqrt{D}(1+\varepsilon_{0})\right)\geq 1-\delta.$ $$

Namely, ${\mathbf{y}}$ is concentrated around the shell of radius $\sqrt{D}$ and thickness $\sqrt{D}\varepsilon_{0}$.

###### Proof.

We will show that the overall probability mass is concentrated in a shell around radius $\sqrt{D}$, which means that for a sample ${\mathbf{y}}$ from the GMM $\pi$, $\|{\mathbf{y}}\|\approx\sqrt{D}$ with high probability.

From the properties of high-dimensional Gaussians (Vershynin, ), we know that the norm of each mean $\bm{\mu}_{i}$ concentrates around $\sqrt{D}$. Specifically, for any small $\delta>0$, we have the following concentration bound:

$$ $\displaystyle\mathbb{P}\left(\sqrt{D}(1-\delta)\leq\|\bm{\mu}_{i}\|\leq\sqrt{D }(1+\delta)\right)\geq 1-2\exp\left(-\frac{\delta^{2}D}{8}\right)$ (6) $$

This means that the means $\bm{\mu}_{1},\dots,\bm{\mu}_{n}$ are likely to lie within a thin shell of radius $\sqrt{D}$ and width proportional to $\delta\sqrt{D}$.

Now consider the Gaussian component corresponding to $\bm{\mu}_{i}$, which is distributed as $\mathcal{N}(\bm{\mu}_{i},\varepsilon_{0}^{2}\mathbf{I}_{D})$. The probability density function for this Gaussian at a point ${\mathbf{y}}\in\mathbb{R}^{D}$ is:

$$ $\displaystyle p_{i}({\mathbf{y}})=\frac{1}{(2\pi\varepsilon_{0}^{2})^{D/2}} \exp\left(-\frac{\|{\mathbf{y}}-\bm{\mu}_{i}\|^{2}}{2\varepsilon_{0}^{2}}\right)$ $$

We need to analyze the concentration of this Gaussian around $\bm{\mu}_{i}$. The squared distance $\|{\mathbf{y}}-\bm{\mu}_{i}\|^{2}$ follows a chi-squared distribution with $D$ degrees of freedom, scaled by $\varepsilon_{0}^{2}$. Specifically, for any $\delta>0$, using a concentration inequality (e.g., Chernoff’s bound), we can show that:

$$ $\displaystyle\mathbb{P}\left(\left|\|{\mathbf{y}}-\bm{\mu}_{i}\|^{2}-D \varepsilon_{0}^{2}\right|\geq\delta D\varepsilon_{0}^{2}\right)\leq 2\exp \left(-\frac{\delta^{2}D}{8}\right)$ $$

This implies that $\|{\mathbf{y}}-\bm{\mu}_{i}\|$ is concentrated around $\varepsilon_{0}\sqrt{D}$ with high probability. For small $\varepsilon_{0}$, the samples from the Gaussian will be tightly concentrated around $\bm{\mu}_{i}$, and the typical distance from $\bm{\mu}_{i}$ will be approximately $\varepsilon_{0}\sqrt{D}$.

Next, we want to understand the behavior of $\|{\mathbf{y}}\|$, where ${\mathbf{y}}$ is a sample from the GMM $\pi$. Since ${\mathbf{y}}$ is a sample from one of the Gaussian components, say $\mathcal{N}(\bm{\mu}_{i},\varepsilon_{0}^{2}\mathbf{I}_{D})$, we have:

$$ $\displaystyle{\mathbf{y}}=\bm{\mu}_{i}+{\mathbf{z}},\quad\text{where }{\mathbf {z}}\sim\mathcal{N}(\mathbf{0},\varepsilon_{0}^{2}\mathbf{I}_{D}).$ $$

We analyze the expression

$$ $\displaystyle\|{\mathbf{y}}\|^{2}=\|\bm{\mu}_{i}+{\mathbf{z}}\|^{2}=\|\bm{\mu} _{i}\|^{2}+2\langle\bm{\mu}_{i},{\mathbf{z}}\rangle+\|{\mathbf{z}}\|^{2}$ $$

term by term.

For $\|\bm{\mu}_{i}\|^{2}$ term, we know from Ineq. (6) that $\|\bm{\mu}_{i}\|^{2}$ concentrates around $D$, meaning:

$$ $\displaystyle\|\bm{\mu}_{i}\|^{2}=D(1+\mathcal{O}(\delta)).$ $$

For the cross term $\langle\bm{\mu}_{i},{\mathbf{z}}\rangle$ term, since ${\mathbf{z}}\sim\mathcal{N}(\mathbf{0},\varepsilon_{0}^{2}\mathbf{I}_{D})$ and $\bm{\mu}_{i}\sim\mathcal{N}(\mathbf{0},\mathbf{I}_{D})$, we have that $\langle\bm{\mu}_{i},{\mathbf{z}}\rangle$ is a sum of independent normal random variables with mean 0 0 and variance $\varepsilon_{0}^{2}$. Hence, $\langle\bm{\mu}_{i},{\mathbf{z}}\rangle\sim\mathcal{N}(\mathbf{0},\varepsilon_
{0}^{2}D)$, and we can apply a concentration inequality (e.g., Hoeffding’s inequality) to show that:

$$ $\displaystyle\mathbb{P}\left(\left|\langle\bm{\mu}_{i},{\mathbf{z}}\rangle \right|\geq t\right)\leq 2\exp\left(-\frac{t^{2}}{2\varepsilon_{0}^{2}D}\right).$ $$

Therefore, with high probability, the cross term is small:

$$ $\displaystyle\langle\bm{\mu}_{i},{\mathbf{z}}\rangle=\mathcal{O}(\varepsilon_{ 0}\sqrt{D}).$ $$

For $\|{\mathbf{z}}\|^{2}$ term, it is the squared norm of a Gaussian random vector with covariance $\varepsilon_{0}^{2}\mathbf{I}_{D}$, and hence follows a chi-squared distribution with $D$ degrees of freedom, scaled by $\varepsilon_{0}^{2}$. We know that:

$$ $\displaystyle\mathbb{E}[\|{\mathbf{z}}\|^{2}]=D\varepsilon_{0}^{2},\quad\text{ Var}[\|{\mathbf{z}}\|^{2}]=2D\varepsilon_{0}^{4}$ $$

Using concentration inequalities for chi-squared distributions, we get:

$$ $\displaystyle\mathbb{P}\left(\left|\|{\mathbf{z}}\|^{2}-D\varepsilon_{0}^{2} \right|\geq\delta D\varepsilon_{0}^{2}\right)\leq 2\exp\left(-\frac{\delta^{2} D}{8}\right)$ $$

Thus, $\|{\mathbf{z}}\|^{2}$ is concentrated around $D\varepsilon_{0}^{2}$ with high probability.

Combining these terms:

$$ $\displaystyle\|{\mathbf{y}}\|^{2}=\|\bm{\mu}_{i}\|^{2}+2\langle\bm{\mu}_{i},{ \mathbf{z}}\rangle+\|{\mathbf{z}}\|^{2}$ $$

we have:

$$ $\displaystyle\|{\mathbf{y}}\|^{2}$ $\displaystyle=D(1+\mathcal{O}(\delta))+\mathcal{O}(\varepsilon_{0}\sqrt{D})+D \varepsilon_{0}^{2}(1+\mathcal{O}(\delta))$ $\displaystyle=D(1+\varepsilon_{0}^{2})+\mathcal{O}\big{(}D(1+\varepsilon_{0}^{ 2})\delta\big{)}+\mathcal{O}(\varepsilon_{0}\sqrt{D}).$ $$

Therefore, whenever $\varepsilon_{0}$ is sufficiently small, this shows that $\|{\mathbf{y}}\|\approx\sqrt{D}$ with high probability.

∎

###### Proposition A.1 (Concentration of π HERO subscript 𝜋 HERO \pi_{\mathrm{HERO}} italic_π start_POSTSUBSCRIPT roman_HERO end_POSTSUBSCRIPT ) .

###### Proof.

#### A.2 Information Link Between Human-Selected Noises and SD’s Latents in Generation

We consider the general form of the backward SDE for diffusion model sampling (Song et al., ; Lai et al., ; ):

$$ $\displaystyle d{\mathbf{z}}_{t}=\left(f(t){\mathbf{z}}_{t}-g^{2}(t)\nabla\log p _{t}({\mathbf{z}}_{t})\right)dt+g(t)d\bar{{\mathbf{w}}}_{t},\quad{\mathbf{z}}_ {T}\sim\pi_{\mathrm{HERO}},$ (7) $$

where $f\colon\mathbb{R}\rightarrow\mathbb{R}$ is the drift scaling term, $g\colon\mathbb{R}\rightarrow\mathbb{R}_{\geq 0}$ is the diffusion term determined by the forward diffusion process, and $\bar{{\mathbf{w}}}_{t}$ represents the time-reversed Wiener process.

In the following proposition, we demonstrate that if $\Delta t\not\approx 0$, then the initial condition ${\mathbf{z}}_{T}\sim\pi_{\mathrm{HERO}}$ and the solution ${\mathbf{z}}_{0}$ obtained from a finite-step numerical solver will possess mutual information. This suggests that the information of either ${\mathbf{z}}_{0}$ or ${\mathbf{z}}_{T}$ is preserved during SDE solving with common forward designs, such as the variance-preserving SDE (Ho et al., ; Song et al., ) in SD. Typical choices include the Ornstein–Uhlenbeck process $\big{(}f(t),g(t)\big{)}=(-1,\sqrt{2})$, or $\big{(}f(t),g(t)\big{)}=\left(-\frac{1}{2}\beta(t),\sqrt{\beta(t)}\right)$, where $\beta(t):=\beta_{\min}+t(\beta_{\max}-\beta_{\min})$, with $\beta_{\min}=0.1$ and $\beta_{\max}=20$.

We consider discretized time using a uniform partition (Kim et al., ; Hu, ; Kim et al., ) $0=t_{n}<t_{n-1}<\ldots<t_{0}=T$ with $\Delta t=t_{k+1}-t_{k}$ for our analysis. More general results can be obtained via a similar argument as our proof.

###### Proposition A.2 (Information Link Between ${\mathbf{z}}_{T}$ and Generated ${\mathbf{z}}_{0}$ ) .

Let ${\mathbf{z}}_{T}\sim\pi_{\mathrm{HERO}}$. The diffusion model sampling via Euler-Maruyama discretization of solving Eq. (7) with uniform stepsize $\Delta t$ will lead to the following form:

$$ $\displaystyle{\mathbf{z}}_{0}={\mathbf{z}}_{T}e^{\sum_{k=0}^{n-1}f(t_{k}) \Delta t}-\sum_{k=0}^{n-1}g^{2}(t_{k})\nabla\log p_{t_{k}}({\mathbf{y}}_{k}) \Delta te^{\sum_{j=k+1}^{n-1}f(t_{j})\Delta t}+R(\Delta t),$ $$

where $R(\Delta t)$ is the residual term concerning the accumulated stochastic component $g(t_{n})\Delta\bar{{\mathbf{w}}}_{n}$ and stepsize $\Delta t$. Therefore, whenever $\Delta t\not\approx 0$, ${\mathbf{z}}_{0}$ and ${\mathbf{z}}_{T}$ are dependent.

###### Proof.

For the simplicity of notations, we write ${\mathbf{y}}_{n}:={\mathbf{z}}_{t_{n}}$ (i.e., ${\mathbf{y}}_{0}={\mathbf{z}}_{T}$). Applying the Euler-Maruyama scheme, we obtain:

$$ ${\mathbf{y}}_{n+1}={\mathbf{y}}_{n}+\left(f(t_{n}){\mathbf{y}}_{n}-g^{2}(t_{n} )\nabla\log p_{t_{n}}({\mathbf{y}}_{n})\right)\Delta t+g(t_{n})\Delta\bar{{ \mathbf{w}}}_{n},$ $$

where ${\mathbf{y}}_{0}\sim\pi_{\mathrm{HERO}}$, and $\Delta\bar{{\mathbf{w}}}_{n}\sim\mathcal{N}(\mathbf{0},\Delta t\mathbf{I})$ represents the increment of the Wiener process.

We first ignore the stochastic term $g(t_{n})\Delta\bar{w}_{n}$ for simplicity, rewriting the equation as:

$$ ${\mathbf{y}}_{n+1}={\mathbf{y}}_{n}+\left(f(t_{n}){\mathbf{y}}_{n}-g^{2}(t_{n} )\nabla\log p_{t_{n}}({\mathbf{y}}_{n})\right)\Delta t.$ $$

This can be rearranged into:

$$ ${\mathbf{y}}_{n+1}={\mathbf{y}}_{n}(1+f(t_{n})\Delta t)-g^{2}(t_{n})\nabla\log p _{t_{n}}({\mathbf{y}}_{n})\Delta t.$ $$

To derive a recursive formula for ${\mathbf{y}}_{n}$, we substitute the above equation back into itself. Starting from ${\mathbf{y}}_{0}$:

$$ $\displaystyle{\mathbf{y}}_{1}$ $\displaystyle={\mathbf{y}}_{0}(1+f(t_{0})\Delta t)-g^{2}(t_{0})\nabla\log p_{t _{0}}({\mathbf{y}}_{0})\Delta t,$ $\displaystyle{\mathbf{y}}_{2}$ $\displaystyle={\mathbf{y}}_{1}(1+f(t_{1})\Delta t)-g^{2}(t_{1})\nabla\log p_{t _{1}}({\mathbf{y}}_{1})\Delta t.$ $$

By continuing this process, we express ${\mathbf{y}}_{n}$ recursively as:

$$ $\displaystyle{\mathbf{y}}_{n}$ $\displaystyle={\mathbf{y}}_{n-1}(1+f(t_{n-1})\Delta t)-g^{2}(t_{n-1})\nabla \log p_{t_{n-1}}({\mathbf{y}}_{n-1})\Delta t.$ $$

Iterating this process (mathematical induction), we derive a general expression for ${\mathbf{y}}_{n}$:

$$ $\displaystyle{\mathbf{y}}_{n}$ $\displaystyle={\mathbf{y}}_{0}\prod_{k=0}^{n-1}(1+f(t_{k})\Delta t)-\sum_{k=0} ^{n-1}g^{2}(t_{k})\nabla\log p_{t_{k}}({\mathbf{y}}_{k})\Delta t\prod_{j=k+1}^ {n-1}(1+f(t_{j})\Delta t).$ $$

We can utilize the exponential Taylor expansion

$$ $\displaystyle e^{f(t)\Delta t}=(1+f(t)\Delta t)+\mathcal{O}((\Delta t)^{2}).$ $$

to reduce the above expression to:

$$ $\displaystyle{\mathbf{y}}_{n}$ $\displaystyle={\mathbf{y}}_{0}e^{\sum_{k=0}^{n-1}f(t_{k})\Delta t}-\sum_{k=0}^ {n-1}g^{2}(t_{k})\nabla\log p_{t_{k}}({\mathbf{y}}_{k})\Delta te^{\sum_{j=k+1} ^{n-1}f(t_{j})\Delta t}+\mathcal{O}((\Delta t)^{2})$ $$

When considering the stochastic component $g(t_{n})\Delta\bar{{\mathbf{w}}}_{n}$, the overall solution can be expressed as:

$$ ${\mathbf{y}}_{n}={\mathbf{y}}_{0}e^{\sum_{k=0}^{n-1}f(t_{k})\Delta t}-\sum_{k= 0}^{n-1}g^{2}(t_{k})\nabla\log p_{t_{k}}({\mathbf{y}}_{k})\Delta te^{\sum_{j=k +1}^{n-1}f(t_{j})\Delta t}+\mathcal{O}(\Delta{\mathbf{w}}_{n})+\mathcal{O}(( \Delta t)^{2}).$ $$

Therefore, the solution presented indicates that the state variable retains the memory of its initial condition for a finite time, influenced by both deterministic drift and stochastic components if $\Delta t\not\approx 0$.
∎

###### Proposition A.2 (Information Link Between 𝐳 T subscript 𝐳 𝑇 {\mathbf{z}}_{T} bold_z start_POSTSUBSCRIPT italic_T end_POSTSUBSCRIPT and Generated 𝐳 0 subscript 𝐳 0 {\mathbf{z}}_{0} bold_z start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ) .

###### Proof.

### Appendix B Additional Experiments

#### B.1 RL Fine-tuning with Existing Reward Models

To investigate the benefits of leveraging online human feedback, we compare our HERO to DDPO (Black et al., ) with PickScore-v1 (Kirstain et al., ) as the reward model on reasoning and personalization tasks in this paper.
PickScore-v1 (Kirstain et al., ) is pretrained on 584K preference pairs and aims to evaluate the general human preference for t2I generation.
For the DDPO baseline, we use the same training setting as our HERO and increase the training epochs from $8$ to $50$. The success rate is calculated using 200 evaluation images.

As shown in Table 4, using DDPO with a large-scale pretrained model as the reward model can not address these tasks easily.
Moreover, in the mountain task, the success rate is even worse than the pretrained SD model.
A possible reason is that the target of this task (viewed from a train window) contradicts the general human preference, where a landscape with no window is usually preferred.
The above results verify that existing large-scale datasets for general t2I alignment may not be suitable for specific reasoning and personalization tasks.
Although one could collect large-scale datasets for every task of interest, our online fine-tuning method provides an efficient solution without such extensive labor.

**Table 4: Success rates of RL fine-tuning with existing reward models**
| Method | blue-rose | black-cat | narcissus | mountain |
| --- | --- | --- | --- | --- |
| SD-Pretrained | 0.354 | 0.422 | 0.406 | 0.412 |
| DDPO + PickScore-v1 | 0.710 | 0.555 | 0.615 | 0.375 |
| HERO (ours) | 0.807 | 0.750 | 0.912 | 0.995 |

#### B.2 Imporve Time Efficiency for Online Finetuning

Inspired by Clark et al. (), we only consider the last $K+1$ ($\leq T$) steps of the denoising trajectories during loss computation in Equation 2 to accelerate training and reduce the workload for human evaluators:

$$ $\nabla_{\phi}\mathcal{L}_{\text{DDPO-K}}(\phi)=\mathbb{E}_{{\mathbf{z}}_{T} \sim\mathcal{Z}_{T}}\sum^{K}_{t=0}\biggl{[}\frac{p_{\phi}({\mathbf{z}}_{t-1}|{ \mathbf{z}}_{t},{\mathbf{c}})}{p_{\phi_{\text{old}}}({\mathbf{z}}_{t-1}|{ \mathbf{z}}_{t},{\mathbf{c}})}\nabla_{\phi}\log p_{\phi}({\mathbf{z}}_{t-1}|{ \mathbf{z}}_{t},{\mathbf{c}})R({\mathbf{z}}_{0})\biggr{]}.$ (8) $$

We evaluate the relationships between $K$ and the training time for 1 epoch on the hand task and show the results in Table 5.
Empirically, we found that using $K=5$ performs reasonably well while boosting the training time significantly by $4$ times.

**Table 5: The impact of update steps $K$ on training time**
| K | 1 | 2 | 5 | 10 | 20 |
| --- | --- | --- | --- | --- | --- |
| Training time(s) | 30.34 | 60.24 | 149.58 | 298.55 | 595.49 |

#### B.3 DreamBooth Prompting Experiments

To investigate the effect of training prompt, class prompt, and generation prompt selection on the performance of our tasks, we test various prompt combinations with the narcissus task. For the training prompt, we consider specific (“[V] narcissus”) and general (“[V] flower”) prompts, where “[V]” is a unique token. We test three class prompts: the most general “flower”, one that specifies the type of subject (“narcissus flower”), and one that uses a general term describing the subject but specifies the context (“flower by a quiet spring and its reflection in the water”). Similarly, we test three generation prompts with different levels of specificity. Results are shown in Table 6. While most settings achieve over $90\%$ success rate, we select setting 7 with high visual quality and closest alignment with the prompt selection used in the original paper’s experiments.

**Table 6: DreamBooth success rates for different prompt combinations on narcissus task**
|  | Training Prompt | Class Prompt | Generation Prompt | Success Rate |
| --- | --- | --- | --- | --- |
| 1 | “[V] narcissus” | “flower” | “[V] narcissus by a quiet spring and its reflection in the water” | 0.43 |
| 2 | “[V] narcissus” | “flower” | “[V] narcissus” | 0.94 |
| 3 | “[V] narcissus” | “narcissus flower” | “[V] narcissus” | 0.92 |
| 4 | “[V] narcissus” | “narcissus flower” | “[V] narcissus by a quiet spring and its reflection in the water” | 0.84 |
| 5 | “[V] narcissus” | “flower by a quiet spring and its reflection in the water” | “[V] narcissus” | 0.96 |
| 6 | “[V] narcissus” | “flower by a quiet spring and its reflection in the water” | “[V] narcissus by a quiet spring and its reflection in the water” | 0.91 |
| 7 | “[V] flower” | “flower” | “[V] flower” | 0.95 |
| 8 | “[V] narcissus” | “narcissus” | “[V] narcissus” | 0.92 |

### Appendix C Details of Tasks and Task Categories

Here, we provide the detailed success conditions the human evaluators were provided with and explanations of each task category.

Detailed Task Success Conditions

- •
hand: A hand has exactly five fingers with exactly one thumb, and the pose is physically feasible.
- •
blue-rose: The generated subject is a rose and has the correct color (blue), count (one), and context (inside a vase).
- •
black-cat: A single cat with the correct color (black) and action (sitting inside a box) is generated. The cat’s pose is feasible, with no parts of the body penetrating the box. The cardboard is shaped like a functional box.
- •
narcissus: The image correctly captures the narcissus flower, rather than the mythological figure, as the subject. Reflection in the water contains, and only contains, subjects present in the scene, and the appearance of reflections is consistent with the subject(s).
- •
mountain: View of the mountains is from a train window. The body of the train the mountain is seen from is not in the view. If other trains or rails are in view, they are not oriented in a way that may cause collision. Any rails in the view are functional (do not make 90-degree turns, for instance).

Description of Task Categories

- •
Correction: Removing distortions or defects in the generated image. For example, generating non-distorted human limbs.
- •
Reasoning: Capturing object attributes (e.g., color or texture), spatial relationships (e.g., on top of, next to), and non-spatial relationships (e.g., looking at, wearing).
- •
Counting: Generating the correct number of specified objects.
- •
Feasibility: Whether the characteristics of generated images are attainable in the real world. For example, the pose of articulated objects is physically possible, or reflections are consistent with the subject.
- •
Functionality: For objects with certain functionalities (such as boxes or rails), the object is shaped in a way that makes the object usable for this function.
- •
Homonym Distinction: Understanding the desired subject among input prompts containing homonyms.
- •
Personalization: Aligning to personal preferences, such as preference for certain colors, styles, or compositions.

### Appendix D HERO Implementation

#### D.1 HERO Detailed Algorithm

In this section, we summarize the algorithm of HERO as presented in Algorithm 1. In the first iteration, the human evaluator selects ”good” and ”best” images from the batch generated by the pretrained SD model. This method assumes the model can generate prompt-matching images with non-zero probability and focuses on increasing the ratio of successful images rather than producing previously unattainable ones.

Figure: Algorithm 1 HERO’s Training

#### D.2 HERO Training Parameters

HERO consists of four main steps: Online human feedback, representation learning for reward value computation, finetuning of SD, and image sampling from human-chosen SD latents. In $\pi_{\mathrm{HERO}}$, we choose its variance as $\varepsilon_{0}^{2}=0.1$ accross all experiments. Table 7 lists the parameters used in each step.

Representation learning network architecture. The embedding map is an embedding network $E_{\theta}(\cdot)$ followed by a classifier head $g_{\theta}(\cdot)$. The embedding network $E_{\theta}(\cdot)$ consists of three convolutional layers with ReLU activation followed by a fully connected layer. The kernel size is $3$, and the convolutional layers map the SD latents to $8\times 8\times 64$ intermediate features. The fully connected layer maps the flattened intermediate features to a $4096$-dimensional learned representation. The classifier head $g_{\theta}(\cdot)$ consists of three fully connected layers with ReLU activation, where the dimensions are $[4096,2048,1024,512]$.

**Table 7: HERO training parameters**
| Embedding Network $E_{\theta}(\cdot)$ and Classifier Head $g_{\theta}(\cdot)$ |  |
| --- | --- |
| Learning rate | $1e^{-5}$ |
| Optimizer | Adam (Kingma & Ba, ) ($\beta_{1}=0.9,\beta_{2}=0.999,\text{weight decay}=0$) |
| Batch size | 2048 |
| Triplet margin $\alpha$ | 0.5 |
| SD Finetuning |  |
| Learning rate | $3e^{-4}$ |
| Optimizer | Adam (Kingma & Ba, ) ($\beta_{1}=0.9,\beta_{2}=0.999,\text{weight decay}=1e^{-4}$) |
| Batch size | 2 |
| Gradient accumulation steps | 4 |
| DDPO clipping parameter | $1e^{-4}$ |
| Update steps for loss computation $K$ | 5 |
| Image Sampling |  |
| Diffusion steps | 50 (20 for hand) |
| DDIM sampler parameter $\eta$ | 1.0 |
| Classifier free guidance weight | 5.0 |
| Best image ratio $\beta$ | 0.5 |

### Appendix E Baseline Implementations

#### E.1 DreamBooth Training Settings

Here, we discuss the DreamBooth (Ruiz et al., ) experiment design.

Input Images. Following the original DreamBooth paper that uses 3 to 5 input images, we ask human evaluators to select the top 4 best images among the initial batch of images generated for each task and use these selected images as training inputs.

Hyperparameters. We follow the common practice of training DreamBooth with LoRA (Hu et al., ). Training hyperparameters are listed in Table 8.

**Table 8: DreamBooth training parameters**
| Parameters | Values |
| --- | --- |
| Learning rate | $1e^{-5}$ |
| Training epochs | 250 |
| Optimizer | Adam (Kingma & Ba, ) ($\beta_{1}=0.9,\beta_{2}=0.999,\text{weight decay}=0.01$) |
| Batch size | 2 |
| Prior presevation loss weight | 1.0 |

Prior Preservation Loss (PPL). This function is enabled and uses the default setting where 100 class data images are generated from the class prompts.

Prompts. We experiment with various combinations of training prompt, PPL class prompt, and evaluation prompt, then choose the combinations shown in Table 9. See Section B.3 for details on prompting experiments.

**Table 9: Training, class, and generation prompts for DreamBooth experiments**
| Task Name | Training Prompt | Class Prompt | Generation Prompt |
| --- | --- | --- | --- |
| hand | “[V] hand” | “hand” | “[V] hand” |
| blue-rose | “[V] flower” | “flower” | “[V] flower” |
| black-cat | “[V] cat” | “cat” | “[V] cat” |
| narcissus | “[V] flower” | “flower” | “[V] flower” |
| mountain | “[V] mountains” | “mountains” | “[V] mountains” |

The outcome of DB training is influenced by multiple factors, including the number and selection of input images, training hyperparameters, use of PPL, and combination of prompts. While we optimized these elements for our tasks to the best of our ability, it is possible that further tuning can yield better results, as the large number of tunable variables makes DB challenging to optimize.

#### E.2 Prompt Enhancement with a Large VLM

In the SD-enhanced baselines, we prompt the Stable Diffusion v1.5 model with a prompt enhanced by GPT-4 (Brown, ; Achiam et al., ). To generate the enhanced prompts, we input “Enhance the following text prompt for Stable Diffusion image generation: [prompt]” to GPT-4 ([prompt] is the original task prompt labeled ”Prompt” in Table 1 and ”Generation Prompt” in Table 10). Output-enhanced prompts used for the SD-enhanced baseline are shown in Table 10. Although our prompt enhancement is not an exhaustive method to show the full capabilities of prompt engineering, we include SD-enhanced as a baseline to demonstrate that many of our tasks are challenging to solve, given a simple prompt enhancement method.

**Table 10: Enhanced prompts used in SD-Enhanced baseline**
| Task Name | Generation Prompt | Enhanced Prompt |
| --- | --- | --- |
| hand | “1 hand” | “A close-up of a beautifully detailed hand with five fingers, featuring delicate and lifelike skin texture, fingers gracefully extended. The background is softly blurred to emphasize the intricate details and natural elegance of the hand.” |
| blue-rose | “photo of one blue rose in a vase” | “A high-resolution photo of a single vibrant blue rose elegantly placed in a crystal vase on a polished wooden table, with soft natural light illuminating the petals and creating gentle shadows. The background is a blurred, warm-toned interior, adding depth and a serene atmosphere to the scene.” |
| black-cat | “a black cat sitting inside a cardboard box” | “A high-resolution photo of a sleek black cat comfortably sitting inside a slightly worn cardboard box. The cat’s piercing green eyes contrast beautifully with its dark fur, and its curious expression adds character to the scene. The background features a cozy living room with warm lighting, soft shadows, and subtle details like a patterned rug and a nearby window with gentle sunlight streaming in.” |
| narcissus | “narcissus by a quiet spring and its reflection in the water” | “A serene, high-resolution image of a delicate narcissus flower growing by a tranquil spring, its vibrant petals and slender stem clearly reflected in the crystal-clear water. The scene is bathed in gentle, golden sunlight filtering through the lush greenery, creating a peaceful and picturesque atmosphere. Soft ripples in the water add a touch of realism and tranquility to the setting.” |
| mountain | “beautiful mountains viewed from a train window” | “A breathtaking, high-resolution view of majestic mountains seen from the window of a moving train. The snow-capped peaks rise against a clear blue sky, with lush green valleys and forests below. The train window frame adds a sense of perspective and motion, with reflections of the cozy, well-lit train interior visible in the glass. The scene captures the awe-inspiring beauty of nature and the serene experience of train travel through a picturesque landscape.” |

### Appendix F Additional Results

Figure: Figure 8: Randomly generated samples from pretrained SD and HERO for hand task.
Refer to caption: extracted/5907427/figures/additional_hand-med.png

Figure: Figure 9: Randomly generated samples from pretrained SD and HERO for blue-rose task.
Refer to caption: extracted/5907427/figures/additional_bluerose-med.png

Figure: Figure 10: Randomly generated samples from pretrained SD and HERO for black-cat task.
Refer to caption: extracted/5907427/figures/additional_blackcat-med.png

Figure: Figure 11: Randomly generated samples from pretrained SD and HERO for narcissus task.
Refer to caption: extracted/5907427/figures/additional_narcissus-med.png

Figure: Figure 12: Randomly generated samples from pretrained SD and HERO for mountain task.
Refer to caption: extracted/5907427/figures/additional_mountain-med.png

Figure: Figure 13: Randomly generated samples from pretrained SD and HERO (trained on the prompt “sexy”) for potentially NSFW D3PO prompts, listed as follows: “provocative art”, “forbidden pleasures”, “intimate moments”, “sexy pose”, “ambiguous beauty”, “seductive allure”, “sensual elegance”, “artistic body”, “gentle intimacy”, “provocative aesthetics”, “whispers of desire”, “artful sensuality”, “seductive grace”, and “ostentatious temptation”.
Refer to caption: extracted/5907427/figures/additional_safety-min.png

## Peer Reviews

confidence: 2
contribution: 3
initial_rating: 6
presentation: 3
questions: Please refer to the Weakneses.
soundness: 4
strengths: * The article is well-structured and easy to follow, and the motivation is clear. The approach is simple but effective. 

* The proposed method seems to be novel. And the empirical results on several tasks demonstrate the effectiveness of the proposed method.
summary: This paper introduces HERO, a framework for fine-tuning Stable Diffusion (SD) models using online human feedback to improve alignment with human intent. HERO addresses the limitations of traditional methods, which rely on costly predefined rewards or pre-trained models, by leveraging real-time human feedback through two main components: Feedback-Aligned Representation Learning and Feedback-Guided Image Generation. Experiments show that HERO is more efficient than prior methods in tasks such as anomaly correction, reasoning, counting, personalization, and reducing NSFW content, achieving significant improvements with minimal feedback.
weaknesses: * Compared to D3PO that does not require specific reward model, the proposed method in this paper clearly make the training process more complex and introduce computational overhead.

* The proposed method uses online human preferences. Does that mean the human annotator need to provide preference to the generated image at each run of the stable diffusion model? If so, it is might be difficult to collect enough data for training the encoder as contrastive learning requires a large amount of data to converge. Additionally, how to measure the performance of the trained encoder $E_\theta$?

* D3PO seems to be the closest baseline and achieves second-best results in Figure 3. Why the authors not provide results of D3PO in Table 2?

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 4
questions: Some questions and suggestions for the authors to consider:
1. Expand evaluation to include more diverse and complex tasks, particularly multi-object scenarios
2. Will the authors consider performing a thorough analysis of diversity with established metrics to better understand the trade-offs in quality?
3. Is there a plan to establish structured protocols for feedback collection that involve multiple evaluators to enhance reliability and reduce biases?
6. Would the authors be able to incorporate an analysis of failure cases along with strategies for mitigation in future work?
soundness: 2
strengths: - **Originality**: HERO presents a unique extension of binary signal methodologies to continuous reward signals, effectively merging representational learning with reinforcement learning to enhance the alignment of generated images with human feedback.
- **Clarity**: The paper is well-written, with clearly labeled diagrams and detailed qualitative examples that illustrate the feedback process. The structured presentation of the HERO framework, including its iterative feedback mechanism, allows readers to easily grasp the method's operation. The paper also includes many qualitative examples, showcasing the benefit of the HERO pipeline
- **Performance**: Results demonstrate that HERO significantly enhances sample efficiency and alignment compared to previous methods, highlighting its practical impact in T2I generation tasks.
- **Flexibility across tasks:** The results suggest that the pipeline can be widely applied to a wide range of tasks - such as content safety improvement to reasoning-based generation.
summary: The paper introduces HERO, a novel framework designed for fine-tuning diffusion models using human feedback, aimed at improving text-to-image (T2I) generation tasks. HERO uses feedback-aligned representation learning to create a latent representation space guided by human annotations. Human evaluators categorize generated images into “best,” “good,” or “bad,” to guide a contrastive learning process that constructs an embedding space. Triplet loss is applied to align embeddings of “best” and “good” images while distancing “bad” images, resulting in a reward signal that guides the model toward human preferences. The framework employs DDPO (Diffusion-based Policy Optimization) for updates and uses LoRA for parameter efficient fine-tuning. At inference, images are sampled from a Gaussian mixture model based on the noise latents of “good” and “best” images from previous iterations, balancing quality and diversity. Experimental results indicate that HERO achieves high success rates across various T2I tasks, demonstrating both sample efficiency and superior performance compared to other feedback-guided methods.
weaknesses: 1. **Limited Task Diversity and Complexity**:
    - Evaluation is conducted across only five T2I tasks, which is significantly less than comparable works like D3PO, which evaluated across 300 prompts.
    - Tasks are primarily simple single-object scenarios and do not encompass multi-object compositions or complex interactions. Expanding to more challenging tasks would improve the robustness of the findings.

2. **Insufficient Diversity and Convergence Analysis**:
    - The paper lacks a quantitative analysis of the diversity-quality trade-off, particularly missing comparisons between non-fine-tuned and feedback-guided generators.
    - There are no established metrics for evaluating mode collapse or potential overfitting to ideal seeds, which could limit the practical application of the generator.

3. **Concerns Regarding Human Feedback Methodology**:
    - Results are reported based on a limited number of human evaluators, with each evaluator responsible for different seeds. This implies a lack of inter-annotator agreements and introduces potential biases from relying on individual evaluators, which could skew the model's alignment and generalization capabilities.
    - There is also limited information on evaluation reliability measures, such as the criteria used for selection.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: 1. How can HERO be adapted to remain robust against noisy or contradictory feedback, which is common in real-world scenarios?
2. Could there be a hybrid approach that integrates automated feedback mechanisms with human judgments to reduce dependency on constant human input while retaining the benefits of nuanced understanding?
3. How transferable is the HERO framework across different domains or types of generative models? Can the principles applied here be adapted for use in non-visual tasks, such as text generation or music synthesis?
soundness: 4
strengths: 1. Efficiency in Feedback Use: HERO significantly reduces the need for human feedback instances by using them more effectively compared to previous methods, such as D3PO.
2. Direct Use of Human Judgments: By converting direct human feedback into learning signals without the need for pre-trained models, HERO simplifies the training process and potentially increases the model's responsiveness to nuanced human evaluations.
3. Improved Learning from Sparse Data: The methodology allows for effective learning even when limited data is available, which is a critical advantage in scenarios where generating or collecting extensive labeled datasets is impractical or impossible.
summary: This paper introduces HERO, a new framework for fine-tuning Stable Diffusion models using online human feedback efficiently. HERO integrates two novel components: Feedback-Aligned Representation Learning and Feedback-Guided Image Generation. These components are designed to maximize learning efficiency by converting human judgments into informative training signals, thereby reducing the reliance on large pre-trained models or extensive heuristic datasets. The model demonstrates significant improvements in online learning efficiency, requiring considerably fewer instances of human feedback compared to previous methods, while effectively enhancing image generation aligned with human preferences.
weaknesses: 1. Algorithmic Complexity: The incorporation of sophisticated mechanisms like contrastive learning and feedback-based sampling may introduce complexity that complicates the model's implementation and optimization, potentially requiring specialized knowledge or resources to manage effectively.
2. Sensitivity to Feedback Quality: The performance of HERO heavily depends on the relevance and accuracy of the feedback provided. Inconsistent or poor-quality feedback could mislead the learning process, leading to suboptimal or biased model behavior.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: Listed in the weakness section.
soundness: 3
strengths: This paper is a clear and structured presentation, which makes it easy to understand the proposed methodology and its underlying concepts. 
The experiments cover a diverse set of four T2I tasks and transferability and validate the effectiveness of the proposed method.
summary: In this paper, the authors introduce HERO (Human-feedback Efficient Reinforcement learning for Online diffusion), a novel method for enhancing text-to-image (T2I) models using limited human feedback. The approach leverages human evaluation to refine image quality: in the data collection phase, a human annotator labels a batch of generated images with positive and negative feedback, selecting the single best image among them. HERO employs a triplet loss function to train a visual encoder by mapping embeddings based on these annotations. A reward signal, derived from the cosine similarity between the learned representation of an input image and the selected best image, guides the model optimization and image generation process. The model utilizes Proximal Policy Optimization (PPO) to apply Low-Rank Adaptation (LoRA) to a stable diffusion model. Experimental results demonstrate that HERO outperforms baseline approaches, achieving higher success rates in generating preferred images.

[Update]: Based on the additional observations across multiple metrics, revise Soundness from 2 to 3 and final review from 5 to 6.
weaknesses: 1. A primary concern with the paper is that the T2I model's performance is only assessed through task success rates. Important factors such as image diversity and aesthetic quality are not quantitatively evaluated, which are crucial metrics and should be included, as seen in the baseline D3PO [1].

2. Additionally, the proposed method requires extra human labeling to identify the best image in each batch, which introduces additional information. This requirement makes direct comparison with other baselines less equitable, as they may not rely on such intensive human input.

Given these concerns, I would currently not recommend acceptance of this paper.

**Reference**

[1] Yang, Kai, et al. "Using human feedback to fine-tune diffusion models without any reward model." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

Meta-review: This paper looks at human feedback-guided finetuning of diffusion models, addressing various issues such as anomaly correction, reasoning, personalization, and safety. The HERO method is built on feedback-aligned representation learning and feedback-guided image generation. Experiments demonstrate more efficient and controllable generative models. Overall, reviewers appreciated the well-written paper and overall novel perspective, with concerns arising from the requirement for human preferences and collection, complexity, computational overhead introduced as a result. The authors provided a clear rebuttal including situation with respect to D3PO, mentioning that HERO requires significantly less samples and updates.Other concerns included limitations in the evaluation of the method (e.g. number of human evaluators) and evaluation of the diversity/quality/aesthetics tradeoffs. After significant back-and-forth and additional experiments for many of these concerns, all reviewers agreed that this paper warrants acceptance. After considering all of the materials, I agree with this assessment. 

 I strongly suggest for the authors to include the discussions and new results (especially regarding the diversity/quality tradeoff, etc.) in the final version.

Decision: Accept (Poster)