HERO: Human-Feedback Efficient Reinforcement Learning for Online Diffusion Model Finetuning
Authors: Ayano Hiranaka
Shang-Fu Chen
Chieh-Hsin Lai
Dongjun Kim
Naoki Murata
Takashi Shibuya
Wei-Hsiang Liao
Shao-Hua Sun
Yuki Mitsufuji
Venue: ICLR 2025 Poster (2025)
Keywords: Online RLHF
Diffusion Model Finetuning
Primary area: generative models
TL;DR: Fine-tuning Stable Diffusion with minimal online human feedback, achieving 4x more efficiency in controllable generation compared to previous methods.

Abstract: Controllable generation through Stable Diffusion (SD) fine-tuning aims to improve fidelity, safety, and alignment with human guidance. Existing reinforcement learning from human feedback methods usually rely on predefined heuristic reward functions or pretrained reward models built on large-scale datasets, limiting their applicability to scenarios where collecting such data is costly or difficult. To effectively and efficiently utilize human feedback, we develop a framework, HERO, which leverages online human feedback collected on the fly during model learning. Specifically, HERO features two key mechanisms: (1) Feedback-Aligned Representation Learning, an online training method that captures human feedback and provides informative learning signals for fine-tuning, and (2) Feedback-Guided Image Generation, which involves generating images from SD's refined initialization samples, enabling faster convergence towards the evaluator's intent. We demonstrate that HERO is 4x more efficient in online feedback for body part anomaly correction compared to the best existing method. Additionally, experiments show that HERO can effectively handle tasks like reasoning, counting, personalization, and reducing NSFW content with only 0.5K online feedback. The code and project page are available at [https://hero-dm.github.io/](https://hero-dm.github.io/).

Published as a conference paper at ICLR 2025
HERO: HUMAN-FEEDBACKK-FRICE REINFORCEMENT LEARNING FOR ONLINE DIFFUSION MODEL FINETUNING
Ayano Miranaka*1.3
Shang-Fu Chen 112 Chieh-Hsin Lai"!
Dongjun Kim Naoki Murata' Takashi Shibuya' Wei-Hsiang Liao'
Shao-Hua Sunt Yuki Mitsufujit
Sony AI, 2 Graduate Institute of Communication Engineering, National Taiwan University, Universitt of Southern California, "Stanford University
ABSTRACT
Controllable generation through Stable Diffusion (SD) fine-tuning aims to im- prove fidelity, safety, and alignment with human guidance. Existing reinforce- ment learning from human feedback methods usually rely on predefined heuristic reward functions or pretrained reward models built large-scale datasets, limit- ing their applicability to scenarios where collecting such data is costly or difficult. To effectively and efficiently utilize human feedback, we develop framework. HERO, which leverages online human feedback collected on the fly during model learning- Specifically, HERO features two key mechanisms: (1) Feedback-Aligne Representation Learning. an online training method that captures human feed- back and provides informative learning signals for fine-tuning, and (2) Feedback- Guided Image Generation, which involve generating images from SD's refined initialization samples, enabling faster convergence towards the evaluator's intent. We demonstrate that HERO is 4x more efficient in online feedback for body part anomaly correction compared to the best existing method. Additionally, experi- ments show that HERO can effectively handle tasks like reasoning, counting, per- sonalization, and reducing NSFW content with only 0.5K online feedback. The code and project page are available at hhttps://het dim github..bb..... .io/
INTRODUCTION
Controllable text-to-image (T2I) generation focuses on aligning model outputs with user intent, such as producing realistic images, e.g., undistorted human bodies, or accurately reflecting the count, se- manties, and attributes specified by users. To tackle this problem. common paradigm involves fine-tuning latent diffusion models (DM) like Stable Diffusion (SD; (Sm:Rah et al. [0022) using su- pervised fine-tuning (SFT: [ee et al. 2023), which mostly learn from pre-collected, offline datasets. To further enhance the alignment, online reinforcement learning (RL) fine-tuning methods (Fan al. 2024 Black et al. 2024) utilize online feedback that specifically evaluates the samples gener- ated by the model during training. With such dynamic guidance provided on the fly, these methods demonstrate superior performance on various T2I tasks, such as aesthetic quality improvement. Yet, these approaches rely on either predefined heuristic reward functions or pretrained reward models learned from large-scale datasets, which could challenging to obtain, especially for tasks involv- ing personalized content generation (e.g., capturing cultural nuances) or concepts like specific colors or compositions.
To address the above issue, Yang et al. (2024b) introduces D3PO, alternative method that directly leverages online human feedback for fine-tuning diffusion models. Instead of learning from heuris- tic reward functions pretrained reward models, D3PO leverages the samples generated by the
Equaa contributions. This work was done while Ayano Hiranaka and Shang-Fu Chen interned Sony AI. Corresponding authors: Ayano Hiranaka <ahi Sahhrrnna@usc.. edu>, Shang-Fu Chen <sam. s sfchen amail. com>, and Chieh-Hsia Lai <chieh- schieh-hsin. maiisson.ccom Equal advisory.

---

Published as a conference paper at ICLR 2025
Prompt
"a photo of one blue in
Human feedback 8 good
Learnable embedding map
Learned Representation
Stable Diflusion
Generate images
Eg
Sample from "good"' image noise latents
Similarity-basee based
Rewards Computation
Figure 1: @ Online Human Feedback on Generated Images: Each epoch, SD generates a batch of images, evaluated by a human as "good" or "bad", with the "best" among the "good" selected. The corresponding SD noises and latents saved. ① Feedback-Aligned Representation Learning: Human- annottaed images train an embedding map via contrastive learning, converting feedback into continuous representations. These rated by cosine similarity to one of the "best" images and used to fine-tune SD via DDPO (Black et al. 2024). 2 Feedback-Guie
Image Generation: New images generated from Gaussian mixture centered around the recorded noises of "good" images. This process is repeated until the feedback budget exhausted.
model as well as human annotations collected during training. With online human feedback, D3PO addresses various tasks, such as distorted human body correction and NSFW content prevention, without requiring pretrained reward model for each individual task. However, it still necessitates approximately 5K instances of online human feedback during training (Yang et al., 2024b Lehara eet al., 2 placing a significant burden on the human evaluator and restricting the use of cus- tomized fine-tuning to match individual preferences.
To further improve the feedback efficiency of T2I alignment using online human feedback, this work proposes a feeaan- Efficient Reinforcement learning for Online diffusion model fine-tuning framework, dubbed HERO, efficiently and effectively utilize online human feedback to fine-tune a SD model, as illustrated in Figure Specifically, we propose two novel components: (1) Feeilback-Aligned Representation Learning, an traine- embedding map that creates representation space that implicitly captures human preferences and provides continuous reward signals for RL fine-tuning, and (2) Feedback-Guuee Guided Image Generation, which involve generating images from SD's refined initialization samples aligned with human intent, for faster convergence to the evaluator's preferences.
Feedback- -alignd representation learn- ing (Figure S ①) aims to create rep- resentation space that implicitly reflects human preferences, offering continu- reward signals for RL fine-tining. At each epoch, SD generates a batch of images, and human evaluator clas- sifies the images as "good" or "bad", selecting one "best" image from the "good" set. The latents of the human- annotated images are then employed train an embedding map through con- trastive learning (Chen et al. (2000), aiming to develop a feedback-aligned representation space. By calculating the cosine similarity to the "best" rep- resentation vector in the learned repre- sentation space, obtain continuous evaluation for each latent. Subsequently, utilize the computed similarity as continuous reward signals to fine-tune SD via LoRA (Hu et al. (022).
SD-Pretrained
SD-Enhanced
DreamBooth
HERO (Ours)
Figure Result preview. Randomly sampled out- puts generated by HERO and baselines given the prompt "photo of one blue rose in a vase" are presented. Suc- cessful samples marked with & and unsuccessful sam- ples are marked with X which fail to accurately capture the specified count (more than roses), color (non-blue roses), and context (missing vase). HERO successfully captures these aspects, outperforming the baselines.
After fine-tuning the SD for the first iteration, feedback-guided image generation (Figure 2) samples new batch images from a Gaussian mixture centered on the stored "good" and "best" initial noises from the previous iteration. This process facilitates the generation of images that align
2

---

Published as a conference paper at ICLR 2025
with human intentions better than random initial noises, thereby enhancing the efficiency of fine- tuning. HERO effectively achieves controllable T2I generation with minimal online human feedback through iterative feedback-guided image generation. feedback-aligned representation learning, and SD model finetuning.
We conduct extensive experiments various T21 tasks to compare HERO with existing methods. The experimental results show that HERO can effectively fine-tune SD to reliably follow given text prompts with 4x fewer amount of human feedback compared to D3PO (Yang et al. 2024b). On the other hand, the results show that these tasks are difficult to solve through prompt enhance- ment (Winata et al. 2024) or fine-tuning approaches, e.g.. DreamBooth (Ruiz et al. 2023). that rely on a few reference images (Gal et al. (202)). Figureß]
preview of the results. Extensive ab- lation studies verify the effectiveness our proposed feedbbakk-aligned representation learning and the technique of generating images from refined noises. Additionally, we show that the model fine- tuned by HERO demonstrates transferability to previously unseen inference prompts, showcasing that the desired concepts were acquired by the model.
RELATED WORKS
Recent research has explored controllable generation with SD for tasks like T2I alignment ct al. 2024 Prabhudesai al. 2023), conceptual generation (Yang al. [222a Zhong al. 2023
BBlack
correcting generation flaws (Zhang et al.] (023), personalization (Gal et al. 2023 Ruiz et al. 2023
and removing NSFW content (Gandikota al. 2023 Kumari et al. 2023 Lu al. 2024).
Supervised fine-tuning, DreamBooth (DB; Ruiz et al. 2023) and Textual Inversion (Gal et al.
2023) take images as input and fine-tunes SD via supervised learning to learn the specific sub- ject present in the input images. However, such methods require reference images, limiting their applicability to general T2I tasks, such as conceptual generation, e.g., emotional image content generation (Yang et al. 2024a), or accurately reflecting user-specified counts, semantics, and at- ; Xu
tributes (Lin et al. 2024). On the other hand, Prabhudesaii et al. 2023); Gandikota et al. (023 et al. (2024); Clark al. 2024) pretrained reward models to calculate differentiable gradients for SD fine-tuning. However, such pretrained models are not always accessible for tasks of interest, and moreover, these methods cannot directly utilize human feedback, which is non-differentiable.
RL fine-tuning. Various methods have explored incorporating non-differentiable signals, such as human feedback, rewards to fine-tune SD using RI. For example, DDPO (Black al. 2024)
uses predefined reward functions for tasks like compressibility. DPOK (Fan et al. 20224) leverages feedback from an AI model trained on large-scale human dataset, and SEIKO (Uchara obtain rewards from custom reward functions trained from extensive feedback datasets. Yet, these methods require predefined reward function or reward model, which can be difficult to obtain for tasks that involve generating personalized content (e.g., reflecting cultural nuances) or abstract concepts, such as specific colors or compositions (Amadeus al.) 2024 Kannen et al. 2024).
2024)
Direct preference optimization (DPO). Diffusion-DPO (Wallace et al. 2023) applies DPO (Rafailov et al. (023) to directly utilize preference data to fine-tune SD,, eliminating the need for predefined rewards. Despite encouraging their results, such method requires large-scale pre-collected human preference dataset e.g., Diffusion
uses the Pick-a-Pic dataset with 851K preference pairs, making it costly to collect and limiting its applicability to various tasks, including personalization. Instead of leveraging offline datasets, D3PO (Yang et al. 2024b) uses online human feedback collected on-the-fly during model training for DPO-style finetuming of SD. It demonstrates success in tasks such as body part deformation correction and content safety improvement while avoiding the demand for large-scale offline datasets. However, the amount of human feedback quired for D3PO is still high, requiring 5-10k feedback instances per task, which motivates us to framework.
develop a more
3 PRELIMINARIES
Stable Diffusion (SD) operates in two stages. First, an autoencoder compresses images from pixel space into latent representations Zp. which can later be decoded back to pixel space. Second, a diffusion model (DM) is trained model the distribution of these latent representations conditioned
3

---

Published as a conference paper at ICLR 2025
on text c. The forward diffusion process is defined as p(z) (zz au oi where ag and o, are pre-defined time dependent constants for E [0, Both the forward transition kernel p( p(z, (z-1,, c) and the backward conditioned transition kernel p(z,- | e, zo) are Gaussian with closed-form expressions. The DM is trained to predict the clean sample 2n using a neural network (2 1, c), denoising the noisy sample 2, at time
(z(zi-1lz,, c) == p(zz-1 C, 2o == :- 2 (2,, t,))
by optimizing the following objective:
min E.,. [2 + (, -21 =1²]
6 6~ 6~N(O.T).
At inference, random noise ZT is sampled from a prior and iteratively denoised using samplers like DDPM (Ho et al. 2020) and DDIM (Song et al. (020a) to obtain latent code Zo. which is then decoded into an image. This denoising and decoding process forms text-to-image generative model, with random noise 2y sampled from prior and the user-provided prompt.
Denoising Diffusion Policy Optimization (DDPO) formulates the denoising process of diffu- sion models multi-step Markov decision process. With this formulation, can make di- rect Monte Carlo estimates of the reinforcement learning objective. Given a denoising trajectory {ZT, ZT ...y zo}. the denoising diffusion RL update is defined as the following:
VoLDDRL = Σ Ve p c)r(zo, c)
(I)
where ø is the diffusion model, and r(Xg, c) is the received reward computed according the output image Xo and the input prompt c. Based the above update, DDPO further utilizes the impor- tance sampling estimator (Kakade & Langford] 2002) and the trust region clipping from Proximal Policy Optimization (PPO; Schullmn et al. [2017] 7) to perform multiple steps of optimization while maintaining the diffusion model Φ not deviating too far from the previous iteration Oold. The DDPO update is defined as the following:
V CDDPO (Φ)
Po (Zi Ze, c)
V log c)r(zo) c2
Penss (zt-1 [ee, c)
4 PROBLEM SETUP AND THE PROPOSED METHOD
Given a user-specified text prompt, our goal is to fine-tune SD to generate images that align with the prompt by learning from human feedback guidance. In this paper, focus challenging T2I tasks that require spatial reasoning, counting, feasibility understanding, etc., as detailed in Table [ To efficiently and effectively utilize online human feedback, we propose numan-feedback--ffice reinforcement learning for online diffusion model fine-tuning framework, dubbed HERO, as illus- trated in Figure ! Feedback-Aligned Representation Learning (Figure [ D) makes efficient use of limited human feedback by converting discrete feedback to informative, continuous reward sig- nals. In addition, Feedback- Guidedd Image Generation (Figure [ 2) leverages human-prefent preferred noise latents from previous iterations and encourages SD outputs to align more quickly with human intention, further improving sample efficiency.
4.1 ONLINE HUMAN FEEDBACK
In the first iteration of HERO, we generate synthetic images X' from a batch of random noises Σ7 sampled from SD's prior distribution THFRO(²T) := (2(2; 0, I) using DDIM (Song et al. 2020a denoted {2r, (rr,Zr-1,"
Ho et al. 2020 For each zr E Z, the sampling trajectories and each Zo is decoded to an image for human evaluation. A human evaluator reviews X. selects the "good" images X² and labels the remaining images as X To obtain gradation among all "good" images and all "bad" images by representation learning, we ask the evaluator to identify the "best" image in x denoted as hest The details of feedback- are discussed in the following section and we store the following for future use: the sets of images X. X X²,, best, their corresponding SD's clean latents Zo. 2 2, 2 from which they are decoded; and their initial noises (at time T) Zr, 2,,, ZT. 2.7 used in SD's sampling.
representation learning
4

---

Published as a conference paper at ICLR 2025
4.2 FEEDBACK-ALIGNED REPRESENTATION LEARNING
HERO fine-tunes SD with minimal online human feedback by learning representations via a con- trastive objective that captures discrepancies between the best SD's clean latent zf",, positive 2 and negative 2 SD's clean latents (Section 4.2.1p. By caleulating similarity to the best image's s rep- resentation, we use these similarity scores continuous rewards for RL. fine-tuning (Section 4.2.2) This approach bypasses reward model training by directly converting human feedback into learning signals, avoiding the need for over 100k training samples typically required to train a reward model for unseen data (Wallace et al. 2023 Rafailov et al. 2023
4.2.1 LEARNING REPRESENTATIONS
To learn a representation space of Zn aligned with human feedback, we build on the contrastive learning framework of Chen et al. (020).. We design an embedding network Ea(-) to map Zo into the representation space, Tollowed by a projection head go(-) for loss calculation. Triplet margin loss is applied to the projection head's 's output:
[(0; best 20: 2 = E mgg ~2j 25 max s go [Eo (Eo(z))
g0 (Eo (zf
(3)
- S -s( go EEs (z)(z)) go (EO +a,
Eo Fzb serves the anchor in the contrastive loss, with S(::)) representing the similarity score (using cosine similarity) and a as the triplet margin set to 0.5. By using the best image in the triplet loss, we obtain gradation within positive and negative categories based on the distance the best sample. With the learned representation Eg(Zo) for Zo Zo. we can compute continuous rewards for RL fine-tuning tinettunig.
4.2.2 SIMILARITY-BASHR REWARDS COMPUTATION
After training the embedding E-)-) on the current batch of human feedback, reward values are computed the cosine similarity in the learned representation space between each Eo (zo) for 2n Zo and Eg (2:::
Eg (2) Eg (z
R (Z))
for each zo e Zo.
max
Eg 2 11z
Egg 2
best
where 8 1 10-8 8 avoid zero division. By using the learned representations convert simple (discrete) human feedback into continuous reward signals, we avoid the need for large pretrained reward model or costly training of such model.
Besides the "similarity-to-bes"" design, we also consider momnilarity-t--osiiiin design, which uses the similarity between an image and the average of all "good" images the learned repre- sentation space. We choose the "similarity-to-bess design for its superior performance. Further discussion is available in Section 5
4.2.3 DIFFUSION MODEL FINETUNING
DDPO fine-tunes SD by reweighting the likelihood with reward values. For noise latent Zr E Zr and its sampling trajectory {zy, -1. 2o). we incorporate the reward R(zo) from Eq. 4 into the DDPO update rule in Eq. (2) to fine-tune the SD model ø. To reduce costly gradient computa- tions, we adopt LoRA (Hu al. 2022) for fine-tuning.
4.3 SREDBACK-GUIDERS IMAGE GENERATION
After the previous iteration of fine-tuning, we propose feedback- -ggided image generation to facil- itate the fine-tuning process by generating images that reflect human intentions. We sample the noise latents for a new batch images from the Gaussian mixture with means centered around the human-selected "good" 2 and "hest" 2
SD noise latents from the previous iteration, with
5

---

Published as a conference paper at ICLR 2025
small variance Eu- Specifically, sample the noise latent Zy from the distribution THERO(Zy) defined as:
(MTr: I, (ZT) (N(z:: zTT 101
first iteration
(1-9)
Expree; ((zr; 2y E 1,
otherwise.
(5)
Here, introduce hyperparameter best image ratio / to control the proportion of the next batch sampled from the "best" image noise latent. We find that leveraging hest with a larger 3 can accel- erate training convergence to evaluator preferences but may reduce the diversity or the converged accuracy. The above tradeoff can be controlled by the best image ratio 3. We generally set 8 = 0.5 to balance these effects. Further discussion on the best image ratio parameter is in Section 5.3.2
We remark that since the variance So is small, after a few iterations, samples from THERO (T) still concentrate near the prior . (zr; 0,1) at high probability (see Proposition A. Also, zip and z may retain semantic information about human alignment from z good through the finite-step discretization of the SD sampler (see Proposition A.2 Thus, these validate our proposed THERO as refined initializations for sampling.
and Zo best as they are connected
Given a new batch of images X' decoded from the clean latents Zn generated by SD, with corre- sponding initial noises 27 sampled from THERo (ZT) Eq. (5). the human evaluator provides their evaluation as described in Section 4.1 The process is repeated until the feedback budget is exhausted or the evaluator satisfied with the generation from THERo After obtaining the fine-tuned SD SD random noises from refined THERo(Z7) and
model & and THERo (ZT) through HERO, generate images using any DM sampler (Song et al. 2020a).
5 EXPERIMENTAL RESULTS
We demonstrate HERO's performance on variety of tasks, including hand deformation correction, content safety improvement, reasoning, and personalization. Many of them cannot be easily solved by the pretrained model, prompt enhancement, or prior methods. A full list of tasks and their success conditions are shown in Table T We adopt SD v1.5 (Rombach et al. 2022) as the base T2I model, using DDIM (Ho et al., 20200 Song al. 2020a) with 50 diffusion steps (20 for hand deformation correction for fair comparison to the baselines) as the sampler.
We compare HERO to the following baselines:
HEnO
38PO
5D-P
SD-pretrained prompts the pretrained SD model with the original task prompt shown [
Hand Nomaley Rate
0.4
0.3
SD-enhanced prompts the pretrained SD model with an enhanced version of the prompt generated by GPT- 4 (Brown, 2020 Achiam et al. 2023).
0.2
0.1
DreamBooth (DB; Ruiz et al. 2023) finetunes diffusion
models via supervised learning, taking images as input. We use the four best images chosen by the human evalua- tors as model inputs.
1000
4000
5000
Number of Feedback Instances
Figure 3: Hand anomaly correc- tion success rates. Performance of methods except D3PO are average of 8 seeds, where each seed is evalu- ated on 128 images per epoch. DB, SD-P, and SD-E are DreamBooth, SD-pretrained, and SD-enhanced, re- spectively.
D3PO Yang et al.) 2024b) utilize online human feedback for DPO (Rafailov et al. finetuning. Due to the high feedback cost for training. this baseline is considered only for the hand anomaly correc- tion task directly adopted from their work. Success rates are reported as presented in the original paper.
2023)-based diffusion model
5.1 HAND DEFORMATION CORRECTION
Following the problem setup of D3PO (Yang et al. 2024b), we use the prompt "1 hand" for image generation and human discretion to evaluate the normaley of the generated hand images. Param- eters such as sampling steps are set to be consistent with D3PO. each epoch of HERO, feedback on 128 images is collected, and the human evaluator provides total of 1152 feedback over 9 epochs.

---

Published as a conference paper at ICLR 2025
Performance of HERO in comparison to the baselines is shown in Figure 3 As shown in Figure
the pretrained SD model struggles on this task, with a normalcy rate of 11.9% (DD-pretraine) and 7.5% (SD-enhanced), and DB achieves 28%. D3PO reaches 33.3% normalcy rate at 5K feedback, while HERO achieves a comparable success rate of 34.2% with only 1152 feedhack (over 4x more feedback efficient). The sampled images are shown in Appendix H in the appendix.
5.2 DRMONSTRATION ON THE VARIETY OF TASKS
Table I: Task summary.
Task Name
Prompt
Task Categories
hand blue-rose
"I hand"
correction, feasibility reasoning. counting
"photo of blue rose vase" "o black cat sitting inside cardboard box" "narcissus by quiet spring and its reflection the water" "beautiful mountains viewed from train window" "
black-cat
reasoning, feasibility, functionality feasibility, homonym distinction
narci nassissus
mountain
reasoning, functionality, personalization
We further demonstrate the effectivity of HERO on variety of tasks involving reasoning, correction, feasibility and functionality quality enhancement, and personalization. Tasks are listed in Table and descriptions of task success conditions and task categories are found in Appendix [ For each task, human evaluators are presented with 64 images per epoch and provide total of 512 feedback over epochs. We report the average and standard deviation of the success rates across three seeds, where success is evaluated human feedback (DB and HERO), three different human evaluators were each assigned a different seed to provide feedback on. Each evaluator also responsible for evaluating the success rates of all methods for their assigned seed. Results are shown in Table For all tasks, HERO achieves a success rate at or above 75%, outperforming all baselines. This trend is consistent for all three hu- man evaluators, suggesting HERO's robustness to individual differences among human evaluators. Sample images generated by SD -r-pretrained, DB, and HERO are shown in Figure and more results can be found in Appendix H While the baselines often struggle in attribute reasoning (e.g., color, count), spatial reasoning (e.g., inside), and feasibility (e.g., reflection consistent with the subject), HERO models consistently capture these aspects correctly.
64 images generated in the final epoch. For methods that require
In Appendix B
we comprehensively evaluate HERO using various metrics beyond the success rate, including aesthetic quality, image diversity, and text-to-image alignment.
Table 2: Task performance. Mean and standard deviation of success rates of different methods on the four tasks. HERO achieves a success rate at or above 75% and outperforms all baselines, demonstrating effectiveness on variety of tasks.
Method
blue-rose black-cat
narcissus
mountain
Ereetraad
0.354 (0.020)
0.422 (0.092)
0,406 (0.077)
0,412 (0.063)
SDD-Ehaaceed
0.479 (0.030)
0.365 (0.134)
0.276 (0.041)
0.938 (0.022)
DB
0.479 (0.085)
0.453 (0.142)
0.854 (0.092)
0.922 (0.059)
HERO (ours)
0.807 (0.115)
0.750 (0.130)
0.912 (0.007)
0.995 (0.007)
5.3 ABLATIONS
This section presents ablation studies illustrating the roles of each component HERO. In regards to Feedback-AAigee AAligned Representation Learning, we investigate the effects of (1) computation of rewards using learnable feedback-alignee representations and (2) "similarity-to to-itty--tbbe""" design for reward com- putation. For Feedback- Guided Image Generation, the effect of best image ratio is explored.
5.3. EFFECT OF SEEDBACK-ALIGNEED REPRESENTATION LEARNING AND REWARD DESIGN
The effects of using leamed feedback-aligned representations and our reward design are investi- gated through three ablation experiments. Firstly, we demonstrate the benefit of converting discrete human feedback into continuous reward signal by investigating HERO-binary using binary rewards for training. Secondly, we explore the effect of learned representations by re- placing the learned representations in HERO with SD image latents 2 (HmbeO-noEmbed), Finally,
variant of HERO
7

---

Published as a conference paper at ICLR 2025
blue-rose rose
narcissus
black-cat
mountain
SD-Pretrained
X
X
X
x
X
x
X
DreamBooth
X
HERO (Ours)
Figure 4: Qualitative results. The randomly generated samples for the four tasks are shown, with denoting successful samples and X for failures. the blue-rose task, the pretrained SD model often omits the vase, while DB generates roses with incorrect color count. In narcissus, SD frequently fails to capture the subject or produces inconsistent reflections. For black cat,, base- line models exhibit more issues (e.g., the cat's body penetrating the box). In mount in, baseline images often miss the window frame or depict impossible views. Our fine-tuned models mitigate these issues and show significantly higher success rates across all tasks.
we justify our choice of the "similarity-to-best"" reward design by comparing with an alternative using similarity to the average of all Zt and zhest (HERO-positives)) We test each setting on the narcissus task with 512 training feedback and 200 images generated by the fine-tuned model for success rate evaluation. HERO outperforms all other settings, with results summarized in Table
Directly using human labels as binary rewards. An intu- itive way extract reward signal from binary human feed- back to directly convert the feedback into a binary reward. To investigate the effect of similarity-based conversion of hu- man feedback to continuous rewards, we test HERO-binary, a variant where the reward in HERO is replaced with binary reward. Images labeled as "good" or "best" receive reward of 1.0, and all other images receive a reward of 0.0. HERO- binary only reaches 78% success rate while HERO reaches
Table 3: Representation learning and reward design ablation.
Method
Success rate
SD-Pretrained HERO-binan
0.40
0.78
nembed
0.76
VIERO-positvess
0.82
HERO
0.91
91%. This may be because the continuous rewards contain additional information beneficial for DDPO training: While the binary reward only labels images as "good" or "bad", the continuous reward additionally captures a gradation human ratings within the "good" and "bad" categories, supplying additional information such as which "good" images nearly "best", and which are barely "good".
Computing rewards from pretrained image representations. Experiments with binary rewards showed the benefit of using continuous rewards in the learned representation space. To further understand HERO's of feedhack-aligned learned representations, replace the learned repre-
8

---

Published as a conference paper at ICLR 2025
0.80
Success Rate
0.75
Standard Devistion
0.20
f 0.0
modom
8=100
0.5
0.70
0.65
0.15
0.60
0.10
0.55
0.05
0.50
0.45
100
200
300
400
500
0
100
200
300
400
500
Number Feedback Instances
Number of Feedback Instances
Figure 5: Effect of best image ratio 3 evaluated on the black-cat task. Three iterations with different seeds are performed for each setting, and the mean and standard deviation of the success rate are reported separately for clearer visualization. "random" refers to the case where random noise latents are used for sampling (good and best noises latents are not used).
sentations Ep 2a) with SD's clean latents Zo. obtained by denoising SD's initial noises žr, and call this setup HERO-noEmbed, Without embedding map training, 20 no longer cluster around 20 making "similarity-to-bess"" reward design impractical. Thus, we only consider the "similarity-to- positives" reward design for this ablation. While HERO-positives achieves 82% success, HERO- noEmbed reaches 76%, highlighting the advantage of learned representations. Training the embed- ding map also enables the "similarity-to best" reward design, which yields superior performance.
Computing reward as similarity to average of all "good" representations. The reward in HERO computed as the similarity to zhest However, another natural choice is to compute similarity to the average of all 2 Comparing this "similarity-to-pel to-oostiveess design to the "similarity-to-best design employed in HERO, we find that the Similaritt-to-beste design achieves 91% success, while the "similarity- posiiiives" design reaches 82%. We adopt the "similarity to-btstt" design, which empirically gives superior performance.
5.3.2 EFFECT OF BEST IMAGE RATIO IN FEEDBACK- GUIDDD IMAGE GENERATION
To investigate the effect of the best image ratio, we compare the performance of the black-cat task for 0.0, 0.5, 1.0. Further, compare the case where the images sampled from random SD noise latents to demonstrate the benefit of using 2 and best as initial noises for image 2T
generation. Results are shown in Figure 5 Sampling all images from the hest (3 1.0) reaches an average of 70.8% success the end of the training. However, as the high standard deviation in the initial stage of training suggests, over-exploiting single "best" noise latent can cause instability in training. potentially causing the model to settle on suboptimal output. Sampling uniformly from Z and 2/ (3 = 0.0) results in a similar success rate 8 = 1.0, but is less likely to converge to suboptimal point. We empirically find that, for our tasks,
27
0.5 results in the highest success rate while avoiding the risks of fully relying on the single "best" noise latent, thus using 3 0.5 for our experiments. When images are sampled from random SD noise latents, the task success rate does not grow significantly slower in the given amount of feedback, demonstrating the benefit of using Z and best for efficient fine-tuning.
5.4 TRANSFERABILITY
While HERO is trained to optimize for single input prompt, we observe that some personal pref- erences and general concepts learned from one prompt can generalize other related prompts in some cases.
Transfer of personal preference. In the mountain task, we observe the transfer of learned in- dividual preferences. Two human evaluators trained two separate models for the mountair task, where one evaluator preferred green scenery while the other preferred snowy scenery. Each eval- uator's trained model well the corresponding Z and 2T are used to generate images for related task "hiker watching beautiful mountains from the top of a hill". As shown in Figure 6 the preference for green snowy scenery transfers to this new task.

---

Published as a conference paper at ICLR 2025
Transfer of content safety. To further investigate whether general concept, such content safety, learned through one task can transfer to another, we prompt the SD model using the prompt "sexy" and train it to reduce NSFW content in the generated images. The fine-tuned model (as well as the saved 2tt and 2) are used to generate images from a set of 14 potentially-unsafe prompts used in D3PO's content safety task. Utilizing the finetuned model and the saved SD noise latents significantly improves the content safety rate from 57.5% of the pretrained SD model to 87.0%, demonstrating HERO-finetuned model's potential to transfer a general concept learned from one prompt to set of related, unseen prompts. Visual results are shown in Figure 7 and the full list of prompts with more results are shown in Appendix Hiin the appendix.
Hefore HbRO Finetuning
Train green preference
Transfor to new pww.prnn prompp
Train with snowy preference
"hecnitiful moruntzain viewed from train window"
"beautifuu mountains viewed from a troin window"
"hiker watching becuitful mowntoins from top of hill"
Figure 6: Demonstration of personal preference transferability. Models trained with two dis- tinct personal preferences (green and snowy) generate images that inherit these preferences when prompted with a similar task (" ("hiker watching beautiful mountains from the top of a hill").
Octsisston Innssiituion
Figure 7: Qualitative results for the NSFW content hidden task showcasing transferability of HERO. The images randomly generated using the potentially unsafe prompt provided by Yang et al. (2024b). The model is the VEERO-finetune' version, trained with the "sexy" prompt reduce nudity. The safety rate improves from 57.5% (pretrained SD) to 87.0% (HERO), showing HERO's ability to transfer the concept of safety to unseen, potentially unsafe prompts.
CONCLUSION
This work presents HERO, framework that uses online human feedback to fine-tune SD with RLHF. By learning representation aligned with feedback, capture human preferences and turn simple feedback into a continuous reward that enhances DDPO fine-tuning. Starting with human- preferred image noise speeds up the alignment with preferences. Together, these elements make HERO much more efficient, needing 4 times less feedback than the baseline. also shows potential for transferring personal preferences and concepts to similar tasks.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We sincerely thank the reviewers for their constructive feedback, which greatly helped improve this work. We are also deeply grateful to our colleague, Takida Yuhta, for revising the manuscript and providing valuable comments, Their contributions have been instrumental in enhancing the quality of this paper. Shao-Hua Sun was supported by the Yushan Fellow Program by the Ministry of Education, Taiwan.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.087770 2023.
Marcellus Amadeus, William Alberto Cruz Castañeda, André Felipe Zanella, and Felipe Ro- drigues Perche Mahlow. From pampas to pixels: Fine-tuning diffusion models for ga) (ucho heritage. arXiv preprint S0.dv::2401.0552220 2024.
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. The Twelfth International Conference on Learning Rep- resentations, 2024.
Tom B Brown. Language models are few-shot learners, arXiv preprint arXiv:2005.14165 2020.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912 9994412-9924, 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning. 2020.
Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. Directly fine-tuning diffusion models differentiable rewards. In The Twelfth International Conference on Learning Representations, 2024.
Ying Fan, Olivia Watkins, Yuqing Du Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh. Kangwook Lee, and Kimin Lee. Dpok: reinforcement learning for fine-tuning text-to-image diffusion models. In Proceedings of the 37th International Conference on Neural Information Processing Systems, 2024.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or. An image is worth one word: Personalizing text-to-image generation using textual inversion. In The Eleventh International Conference on Learning Representations, 2023.
Rohit Gandikota, Joanna Materzynska, Jaden Fiotto-Kann Kaufman, and David Bau. Erasing concepts from diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-9733, 2020.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840 68581, 2020.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference on Learning Representations, 2022.
Yaozhong Hu. Semi-implicit euler euler-maruyama maruyama scheme for stiff stochastic equations. In Stochastic Analysis and Related Topics V: The Silivri Workshop, 1994, pp. 183-202. Springer, 1996.
11

---

Published as a conference paper at ICLR 2025
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learning, pp. 267-274, 2002.
Nithish Kannen. Arif Ahmad. Marco Andreetto, Vinodkumar Prabhakaran, Utsav Prabhu, Adji Bousso Dieng, Pushpak Bhattacharyya, and Shachi Dave. Beyond aesthetics: Cultural com- petence text-to-image models. arXiv preprint arXiv:2407 3rX.v22407.06663, 2024.
Dongjun Kim, Chieh-Hsin Lai, Wei-Hsiang Liao, Naoki Murata, Yuhta Takida, Toshimitsu Uesaka, Yutong He, Yuki Mitsufuji, and Stefano Ermon. Consistency trajectory models: Learning proba- bility flow ode trajectory of diffusion. In International Conference on Learning Representations, 2024a.
Dongjun Kim, Chieh-Hsing Lai, Wei- Hei-n Liao, Yuhta Takida, Naoki Murata, Toshimitsu Uesaka, Yuki Mitsufuji, and Stefano Ermon. Pagoda: Progressive growing of one-step generator from low-resolution diffusion teacher. arXiv preprint arXiv:2405. /4822, 2024b.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations, 2015.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-pic: An open dataset of user preferences for text-to- to-image generation. Advances in Neural Information Processing Systems, 36:36652-3666) 2023.
Nupur Kumari, Bingliang Zhang, Sheng- Yu Wang, Eli Shechtman, Richard Zhang, and Jun- Yan Zhu. Ablating concepts in text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision pp. 22691-22702, 2023.
Chieh-Hsin Lai, Yuhta Takida, Naoki Murata, Toshimitsu Uesaka, Yuki Mitsufuji, and Stefano Er- mon. Fp-diffusion: Improving score-based diffusion models by enforcing the underlying score fokker-planck equation. In International Conference on Machine Learning, pp. 18365-18398. PMLR, 2023a.
Chieh-Hsin Lai, Yuhta Takida, Toshimitsu Uesaka, Naoki Murata, Yuki Mitsufuji. and Stefano Er- mon. On the equivalence of consistency-type models: Consistency models, consistent diffusion models, and fokker-planck regularization. In ICML 2023 Workshop on Structured Probabilistic Inference and Generative Modeling, 2023b.
Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint 2302. 2023.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping aanuuae--mage pre- training for unified vision- languaee understanding and generation. ICML, 2022.
Wang Lin, Jingyuan Chen, Jiaxin Shi, Yichen Zhu, Chen Liang, Junzhong Miao, Tao Jin, Zhou Zhao, Fei Wu, Shuicheng Yan, al. Non-confusing generation of customized concepts in diffusion models. arXiv preprint arXiv:2405.069/ 2024.
Shilin Lu, Zilan Wang, Leyang Li, Yanzhu Liu, and Adams Wai-Kin Kong. Mace: Mass concept erasure in diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1430-6448 2024.
Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning text-to- image diffusion models with reward backpropagation. arXiv preprint arXiv:2310.03739, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and πlya Sutskever. Learning transferable visual models from natural language supervision. In ICMI. 2021.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Thirty- seventh Conference Neural Information Processing Systems, 2023.
12

---

Published as a conference paper at ICLR 2025
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- on computer vision and pattern recognition, pp. 10684-10695, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 2017.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations, 2020a.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations, 2020b.
Masatoshi Uehara, Yulai Zhao, Kevin Black, Ehsan Hajiramezanali, Gabriele Scalia, Nathaniel Lee Diamant, Alex Tseng, Sergey Levine, and Tommaso Biancalani. Feedback efficient online fine-tuning of diffusion models. In Proceedings of the 41st International Conference Machine Learming. 2024.
Roman Vershynin. High-dimensional probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018
Dimitri von Rütte, Elisabetta Fedele, Jonathan Thomm, and Lukas Wolf. Fabric: diffusion models with iterative feedback, 2023.
Personalizing
Bram Wallace, Meihua Dang. Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference Computer Viston and Pattern Recognition, 20233
Genta Indra Winata, Hanyang Zhao, Anirban Das, Wenpin Tang, David D Yao, Shi-Xiong Zhang, and Sambit Sahu. Preference tuning with human feedback on language, speech, and vision tasks: A survey. arXiv preprint 24X9v 2/64, 2024.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
Jingyuan Yang. Jiawei Feng, and Hui Huang. Emogen: Emotional image content generation with text-to-image diffusion models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2024a.
Kai Yang. Jian Tao, Jiafei Lyu, Chunjiang Ge. Jiaxin Chen, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human feedback to fine-tune diffusion models without any reward model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8941 -8551, 2024b.
Lvmin Zhang, Anyi Rao, and Mancesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3836-3847, 2023.
Shanshan Zhong, Zhongzhan Huang, Wushao Wen, Jinghui Qin, and Liang Lin. Sur-adapter: En- hancing text-to-image pre-trained diffusion models with large language models. In The 31 ACM International Conference on Multimedia, 2023.
13

---

Published as a conference paper at ICLR 2025
APPENDIX
Table of Contents
A Theoretical Explanations
14
A.I
Concentration of Human-Selected Noises in SD's Prior Distribution
14
A.2
Information Link Between -umannSSllcctdd Noises and Latents Generation
B Additional Evaluation Metries
17
B.1
Mcasurement in Inference
17
B.2
Mcasurements in Training Progress
18
Additional Experiments
19
C.I RL Fine-tuning with Existing Reward Models
19
C.2 Imporve Time Efficiency for Online Finetuming
20
C.3 DreamBooth Prompting Experiments
D Details of Tasks and Task Categories
20
E
HERO Implementation
21
E.1 HERO Detailed Algorithm
21
E.2
HERO Training Parameters
22
F
Baseline Implementations
22
F.1
DreamBooth Training Settings
22
F22
Prompt Enhancement with Large VLM
24
G Additional Elaboration of HERO's Mechanisms
24
Additional Results
25
A THEORETICAL EXPLANATIONS
In this section, provide theoretical justifications for the validity of our proposed distribution THERO in Eq. (5) from two perspectives, refining the initial distribution for human- -ew.dback--aigneedd aligned generation.
A.1 CONCENTRATION HUMAN-SELECTED NOISES SD's PRIOR DISTRIBUTION
It is known that the initial distribution of SD sampling is typically the standard normal distribution (O, Ip), which yields random vector that concentrates around the sphere of radius vD with high probability.
the following proposition, we show that random vector drawn from our pro- posed distribution тHERо also concentrates around the sphere of radius VD with high probability, provided that the variance Ep > O of the Gaussian mixture is sufficiently small. This ensures that sampling from the refined initial noise provided by THERO remains consistent with the sampling from the original prior distribution of the SD model.
Proposition A.1 (Concentration of hEERO). Let π be a Gaussian mixture with each component as N(p, DD), where each mean μı; NO, Ip), and en O is a small constant. Let y æ be random vector drawn from n. Then, for any d > O, we have the following concentration if sn is sufficiently small:
P (VD(1- 5o) < 1lyl & VDI1 + E0)
> 1 8.
Namely, is concentrated around the shell of radius v D and thickness DEO-
14

---

Published as a conference paper at ICLR 2025
Proof. We will show that the overall probability mass is concentrated in shell around radius VD, which means that for a sample from the GMM |yyl vD with high probability.
From the properties of high-dimensional Gaussians Vershynin 2018), we know that the norm of each mean μi; concentrates around vD. Specifically, for any small 0, we have the following concentration bound:
DDD
IP D(1
8) ())
( | vD(L+ 21 exp
(6)
8
This means that the means P1.... : Pn are likely to lie within a thin shell of radius v D) and width proportional DyD. D.
Now consider the Gaussian component corresponding to μú: which is distributed as N (M; ID). The probability density function for this Gaussian point y R is:
y mill
p.(y)
(2TE)) exp
2
We need analyze the concentration of this Gaussian around μ4; The squared distance |yy - - | follows a chi-squared distribution with D degrees of freedom, scaled by s Specifically, for any 5 using a concentration inequality (e.g., Chernoff's bound), we can show that:
P (llly 12 Dej dDee) 2eep (- 82D
(-88
This implies that y pill is concentrated around sov D) with high probability. For small co. the samples from the Gaussian will be tightly concentrated around μ4, and the typical distance from μa; will be approximately vD
Next, we want to understand the behavior of ||y||, where y is a sample from the GMM æ. Since y sample from one of the Gaussian components, say 5 ID), we have:
p4 z, where z ~ NO, LD).
We analyze the expression
|y = l|p + 2||2 |1ail + 2(14; 2) + |z||
term by term.
For |. 2 term, we know from Ineq. 6 that p,
concentrates around D, meaning:
|p:l1 = DDL 0(8))
For the cross term (μ; z> term, since V(O,,āīn) and μ4 (N we have that (Mi, z) sum of independent normal random variables with mean and variance e/. Hence, {H4, z) A( =)D) and we can apply concentration incquality (e.g., Hoeffding's inequality) to show that:
(K<45 2) > () & 2exp
(-
25
Therefore, with high probability, the cross term is small:
(n, 2) OsovVD).
For |z| term, the squared norm of Gaussian random vector with covariance e and hence follows chi-squared distribution with D degrees of freedom, scaled by s We know that:
|
Dea. Varl ||| 2D
Using concentration inequalities for chi chi-squared distributions, we get:
DD
PP (1 Deal 8De
2 exp
Thus, [|z]|| i is concentrated around D&f with high probability.
15

---

Published as a conference paper at ICLR 2025
Combining these terms:
| 2 ||,11 1/ 2 2) + |z||²
we have:
1yl = +1 ( +g)0) O(EoVD).
O(5ov D) DE O()))
Therefore, whenever En is sufficiently small, this shows that ||yl| 2 vD with high probability.
A.2 INFORMATION LINK BETWEEN HUMAN-SELECTED NOISES AND SD's LATENTS IN GENERATION
We consider the general form of the backward SDE for diffusion model sampling (Song et al. 2020b Lai et al. 2023a
dz: (J(()z+ o2 (1)V (z:)) di g(()dwr, ZT THERO,
where f: R - R is the drift scaling term, g: IR - Rzo is the diffusion term determined by the forward diffusion process, and w, represents the time-reversed Wiener process.
In the following proposition, we demonstrate that if At * O, then the initial condition ZT ~ THERO and the solution Zg obtained from a finite-step numerical solver will possess mutual information. This suggests that the information of either Zu or Zy is preserved during SDE solving with common
forward designs, such the variance preseevvig SDE (Ho et al. 2020 Song et al. 20206) in SD.
Typical choices include the Ommstin-UUlenbeee process (f()),((t)
(-1,22), or (f(t).((t))
(-19(1), V ((1)), where 3(t) Amin + 1(3max Bmin), with Aain
0.1 and 8max 20.
We consider discretized time using uniform partition (Kim et al. 2024a Hu 1996 Kim et al., 2024b) 0 t., tn-1 ... < to T with At fx+1 ty for our analysis. More general results can be obtained via a similar argument as our proof.
Proposition A.2 (Information Link Between 2r and Generated 2o). Let THERO- The diffusion model sampling via Euler-Maruyana discretization of solving Eq. 7 with uniform stepsize At will lead to the following form:
Zo Zre f(t))
(tx)) log Vogg. (e. 1G)A: R(At),
where R( (((tt) the residual term concerning the accumulated stochastic component g( Aww, and stepsize At. Therefore, whenever At % 0, Zg and zr dependent.
Proof. For the simplicity of notations, we write yn := Zt,. (i.c., Yo ZT). Applying the Euler- Maruyama scheme, we obtain:
Ynt1 = Yn + (f(ta)yn - (t,, log,t, (yw)) At + (t.,)
where Yo THERO. and Aw, NNO0, AtI) represents the increment of the Wiener process. We first ignore the stochastic term g((n) Aw, for simplicity, rewriting the equation as: Yat1 Yn (f(tn)y (t,ttn)Vlogggg Vlog (y.)) At.
This can be rearranged into:
Yn+1 (1 + S(,)) g (1w) (w))) log (t.)...
To derive recursive formula for yn. substitute the above equation back into itself. Starting from
Yo:
y1 yo + f(to) - (2o) V log At,
y2 y1(1 f(t)Att - (4) log Att)At.
16

---

Published as a conference paper at ICLR 2025
By continuing this process, express yn recursively Yn = yn + ((t) g (1 log Pt (nt-
lterating this process (mathematical induction), we derive a general expression for yn::
7-1
1-
yn yo (1 f(())a))
Σ Vogg,, (yb)
I 5
(1 f(t))At).
k=
We can utilize the exponential Taylor expansion
((()) (1 +()))))
to reduce the above expression
12
yn = Yoe f(t))
Σ
(tx) Pt,
Pt, (y Ate S()) + ((()))
When considering the stochastic component gt,,,) Aww the overall solution be expressed as:
Yn Yoe-i-o f(te)
(1x) Ae f(y) O(Aw,)) + +(((t)))
Therefore, the solution presented indicates that the state variable retains the memory of its initial condition for finite time, influenced by both deterministic drift and stochastic components A/ * 0.
B ADDITIONAL EVALUATION METRICS
In this section, we present evaluation metries beyond task success rates and supplement the results of these measurements during inference time in Appendix [B.]] as well as during training in Ap- pendix B.2
B. MEASUREMENT IN INFERENCE
Results of samples from the final epoch for aesthetic quality, image diversity, and text-to-image alignment are presented in Figure The descriptions each measurement are detailed as follows.
Aesthetic Quality. We report ImageReward (Xu et al., 2024) scores, which demonstrate stronger perceptual alignment with human judgment compared to traditional metrics. Higher scores reflect better aesthetic quality. Although human evaluators prioritized task success based on the criteria in Appendix D over aesthetic quality and were not instructed to consider aesthetics, HERO demon- strates comparable aesthetic performance the baselines, surpassing them in out of tasks.
Image Diversity, Following Section 4.3.3 of von Rütte et al. (2023), we compute In-Batch Di- versity", defined as the complement of the average similarity of CLIP image embeddings (Radford et al. 2021) between pairs of images in a generated batch. Specifically, for a batch of generated images 11, 12,,.. ... IN. and the cosine similarity CLIPSim(44,.)) 4y) of their embeddings in the CLIP feature space, the in-batch diversity is calculated as:
Deatch = N(N - LEKjEN
2
Σ CLIPSim(1,, I2),
where 1 CLIPSim(I,, I,) represents the dissimilarity between two images. A higher Dratch sig- nifies greater diversity. Although HERO shows slight reduction in diversity compared the pre- finetuned Stable Diffusion model, it generally outperforms the DreamBooth-fintunee model, except in the black-cat example and mountain example. HERO remains comparable to Stable Diffusion with enhanced prompts in terms of diversity.
Text-to-Imne Im-ee Alignment CLIP Score (Radford et al. 2021) evaluates the similarity between text and image embeddings, while BLIP Score (Li al.) 2022) assesses the probability of text-to-image matching. Together, these metrics provide a quantitative measure how well the generated images align with the given prompts. Higher scores both metrics indicate better alignment between the generated images and the prompts. HERO's finetuned model generally produces images that are more aligned with the given prompts.
17

---

Published as a conference paper at ICLR 2025
S0-P
S0-U
D8
HERO
SD-P
50-41
D36
HERR
0.12
1.0
Io-Barch Diversity
InugeRReward
0.5
0.0
0.06
-0.5
0.04
-1.5
blackcat
bluerose
murcissus
mountain
hand
blackcat
bluerose
narcissus
mountain
Iannd
37.5
1.0
32.5
0.8
Spore
BLII Score
30.0
0.6
27.5
0.4
25.0
0.2
22.5
20.0
0.0
blackeat
blecose
narcissus
mountain
hand
blackcat
bluerose
narcissus
mountain
hand
Figure 8: Additional evaluation results. For all metrics, higher value indicates better performance. Top Left. Aesthetic quality measured with ImageReward (Xu et al. (024). Top Right. In-Batch
Diversity computation following [Rafford al. (2021). Bottom. CLIP (Radford al. (02T) and
BLIP (Li et al. 2022) Text-to-image alignment scores.
B.2 MEASUREMENTS TRAINING PROGRESS
We also provide supplementary results showing different metrics versus training epochs observe the influence the number of feedback samples. As shown in Figure 9 we present results from samples generated during the first epochs, where observe the following trends:
Aesthetic Quality (measured with ImageReward): Aesthetic quality is generally main- tained throughout the fine-tuning process, demonstrating that HERO does not compromise aesthetic appeal even with increased human feedback.
Image Diversity (measured with In-Batch Diversity Score): As HERO fine-tuning pro- gresses, the generated outputs may become more aligned with human intentions, potentially reducing diversity. This aligns with the common phenomenon where stronger guidance of- ten leads to lower diversity. Note that HERO still generally outperforms the DreamBooth- finetuned model in terms of the diversity score.
T2I Alignment (measured with CLIP and BLIP Scores): The alignment between prompts and generated images consistently improves with HERO fine-tuning. This provides implicit evidence that HERO fine-tuning effectively converges toward human intention, as reflected the prompts.
18

---

Published as a conference paper at ICLR 2025
Blackcat
Bluerose
Narcissus
Mountain
Hand
1.5
Diversity
ImageRewand
0.00
0.00
-0.5
-1.
0.04
Epochs
Epochs
(a) Aesthetic Quality (ImageReward)
(b) Diversity (In-batch Diversity Score)
36
1.0
22
CLIP
0.7
0
Epochs
Epochs
(c) T2T Alignment (CLIP Score)
T2I Alignment (BLIP Score)
Figure 9: Addtional Evaluation Measurements across Training Progress. We present additional evaluation results by assessing samples generated at each training epoch across all tasks, measuring aesthetic quality (a), diversity (b), and T2I alignment quality (c and d).
C ADDITIONAL EXPERIMENTS
C.I RL FINE-TUNING WITH EXISTING REWARD MODELS
To investigate the benefits of leveraging online human feedback, we compare our HERO to DDPO (Black et al., (004) with PickScore-vl (Kirstain al. (023) as the reward model on rea- soning and personalization tasks in this paper. PickScore-vl (Kirstain et al., (023) is pretrained on 584K preference pairs and aims to evaluate the general human preference for t21 generation. For the DDPO baseline, from 8 to 50. The success rate is calculated using 200 evaluation images.
the same training setting
HERO and increase the training epochs
As shown in Table 4 using DDPO with large-scale pretrained model the reward model can not address these tasks casily. Moreover, the mount task, the success rate even worse than the pretrained SD model. A possible reason is that the target of this task (viewed from train window) contradicts the general human preference, where a landscape with no window is usually preferred. The above results verify that existing large-scale datasets for general t2I alignment may not be suitable for specific reasoning and personalization tasks. Although one could collect large- scale datasets for every task of interest, our online fine- me-tuning method provides an efficient solution without such extensive labor.
Table 4: Success rates of RL fine-tuning with existing reward models
Method
blue-row rose
black-cat
narcissus
mountaiin
Pretreiiiee
0.354
0.422
0.406
0.412
DDPO + PickScore-vl
0.710
0.555
0.615
0.375
HERO (ours)
0.807
0.750
0.912
0.995
19

---

Published as a conference paper at ICLR 2025
C.2 IMPORVE TIME EFFICIENCY FOR ONLINE FINETUNING
Inspired by Clark et al. (204), we only consider the last K + (< 7) steps of the denoising trajectories during loss computation in Equation for human evaluators:
accelerate training and reduce the workload
Pe Palzi- 2e, 1-0 Porca (z-11 Zz, c)
K
df CDDPO-(()) -E Egg 2r
Σ
V log Palzi-1lze R(zo)
We evaluate the relationships between K and the training time for 1 epoch the hand task and show the results in 3 Empirically, we found that using K performs reasonably well while boosting the training time significantly by 4 times.
Table 5: The impact of update steps K on training time
K
2
5
10
20
Training time(s)
30.34
60.24
149.58
298.55
595.49
C.3 DREAMBOOTH PROMPTING EXPERIMENTS
To investigate the effect of training prompt, class prompt, and generation prompt selection on the performance of our tasks, we test various prompt combinations with the narcissus task. training prompt, we consider specific ("/V) narcissus" ") and general ("/V) flower") prompts, where most general "flower", one that specifies the type of subject ("narcissus flower"), and one that uses general term describing the subject but specifies the context ("flower by a quiet spring and its reflection in the water"). Similarly, we test three generation prompts with different levels of specificity. Results shown in Table 6 While most settings achieve over 90% success rate, we select setting with high visual quality and closest alignment with the prompt selection used in the original paper's experiments.
"[V]" a unique token. test three class prompts:
Table DreamBooth success rates for different prompt combinations on narcissus task
Training Prompt
Class Prompt
Generation Prompt
Success Rate
"[V] narcissus quiet spring and its reflection in the water"
"/V] narcismss
"flower"
0.43
2 "[V] narcissus"
"flower"
"[V] narcissus"
0.94
3 "[V] narcismus"
"narcissux flower"
"[V] narcissus"
0.92
"[V] narcissus by quiet spring
"[V] narcissus"
"narcissus flower"
0.84
4
and its reflection in the water"
"flower by quiet spring and its reflection in the water" "flower by quiet spring and its reflection in the water"
"[V] narcissus"
"[V] narcissus"
0.96
5
"[V] narcissus by quiet spring and its reflection in the water"
"[V] narcissus"
0.91
6
"[oerf
"Rower"
"[V] flower"
0.95
7
8 "[V] nareissus"
"narcissus"
"[V] nareissus"
0.92
D DETAILS OF TASKS AND TASK CATEGORIES
Here, we provide the detailed success conditions the human evaluators were provided with and explanations of each task category.
Detailed Task Success Conditions
hand: A hand has exactly five fingers with exactly thumb, and the pose is physically feasible.
20

---

Published as a conference paper at ICLR 2025
b blue-rose: The generated subject is and has the correct color (blue), count (one), and context (inside a vase).
black-cat: single cat with the correct color (black) and action (sitting inside box) is gen- erated. The cat's pose is feasible, with no parts of the body penetrating the box. The cardboard is shaped like functional box.
narcissus:
The image correctly captures the narcissus flower, rather than the mythological figure, as the subject. Reflection in the water contains, and only contains, subjects present in the scene, and the appearance of reflections is consistent with the subject(s).
mounta View of the mountains is from train window. The body of the train the mountain is seen from is not the view. If other trains or rails are view, they are not oriented way that may cause collision. Any rails in the view functional (do not make 90-degree turns, for instance).
Description Task Categories
Correction:
Removing distortions or defects in the generated image. For example, generating non-distorted human limbs.
Capturing object attributes (e.g., color or texture), spatial relationships (e.g., on top of, next to), and non-spatial relationships (e.g., looking at, wearing).
Reasoning:
Counting: Generating the correct number of specified objects.
Feasibility: Whether the characteristics of generated images are attainable in the real world. For example, the pose of articulated objects is physically possible, or reflections are consistent with the subject.
For objects with certain functionalities (such as boxes or rails), the object shaped in way that makes the object usable for this function.
Functionality:
Homonym Distinction:
Understanding the desired subject among input prompts containing
homonyms.
Personalization: Aligning to personal preferences, such as preference for certain colors, styles, or compositions.
E
HERO IMPLEMENTATION
E.1 HERO DETAILED ALGORITHM
In this section, we summarize the algorithm of HERO as presented in Algorithm [ iteration, the human evaluator selects "good" and "best" images from the batch generated by the pretrained SD model. This method assumes the model can generate prompt-matching images with non-zero probability and focuses increasing the ratio of successful images rather than producing previously unattainable ones.
the first
Algorithm HERO's ss Training
Require: pretrained SD weights d, best image ratio 8, feedback budget Nip
Initialize: learnable weights 0, # of feedback 7&b 0, latent distribution THERo W(zy; 0,7000,1 0)
1: while rith < Nn.
2: Sample Yhaach noise latents from THERO
t> Feedback-Guider Image Generation
3: Perform denoising process for each zr to obtain trajectory {ZT, 2T-1: 20)-
4: Decode Zo with SD decoder for images X.
5:
Query human feedback on X, and save corresponding Zt. Z7, zhre
6:
Update d of Eg and go by minimizing Eq.
Feedhack-Aligned Representation Learning
7:
Compute reward (()z) according to Eq. 4
8:
Update ó via DDPO by minimizing Eq.
9:
Update latents distribution THeRo using Eq.
7gh += Tbatch-
11: end while
21

---

Published as a conference paper at ICLR 2025
E.2 HERO TRAINING PARAMETERS
HERO consists of four main steps: Online human feedback, representation learning for reward value computation, finetuning SD, and image sampling from human-chosen SD latents. In THERO- choose its variance as Ej 0.1 accross all experiments. Table 7 lists the parameters used in each step.
Representation learning network architecture. The embedding map is an embedding network E.(-) followed by a classifier head ge(-). The embedding network Ea(-) consists of three convo- lutional layers with ReLU activation followed by fully connected layer. The kernel size is 3, and the convolutional layers map the SD latents to x x 64 intermediate features. The fully con- 4096-dimensional learned representation. The classifier head go(-) consists of three fully connected layers with ReLU activation, where the dimensions are [4096, 2048, 1024, 512].
nected layer maps the flattened intermediate features
Table 7: HERO training parameters
Embedding Network Es(-) and Classifier Head ga(+)
Learning rate
le-5
Adam (Knnmma & Ba 2015
Optimizer
(B1 0.9, 88
0.999, weight decay 0)
Batch size
2048 0.5
Triplet margin o
SD Finetuning
Learning rate
3e Adam Kingma & Ba, 2015 (8
Optimizer
0.9, P3
0.999, weight decay le +)
Batch size
2
Grudient accumulation steps
4
DDPO clipping parameter
le- 5
Update steps for loss computation
Image Sampling
Diffusion steps
50 (20 for hand)
DDIM sampler parameter 7) Classifier free guidance weight
1.0
5.0
Best image ratio 3
0.5
F BASELINE IMPLEMENTATIONS
F.1 DREAMBOOTHI TRAINING SETTINGS
Here, we discuss the DreamBooth (Ruiz et al. 2023 experiment design.
Input Images. Following the original DreamBooth paper that uses 3 to 5 input images, we ask human evaluators to select the top best images among the initial batch images generated for each task and use these selected images training inputs.
Hyperparameters. We follow the common practice of training DreamBooth with LoRA (Hu et al. 2022 Training hyperparameters are listed in Table g
Table DreamBooth training parameters
Parameters
Values
I Learning rate Training epochs
1e
250
Adam (Kingma Ba, 2015 (8: =
Optimizer
0.9, 82
0.999, weight decay 0.01) 2
Batch size
Prior presevation loss weight
1.0
22

---

Published as a conference paper at ICLR 2025
Prior Preservation Loss (PPL). This function is enabled and uses the default setting where 100 class data images are generated from the class prompts.
Prompts. We experiment with various combinations of training prompt, PPL class prompt, and evaluation prompt, then choose the combinations shown in Table See Appendix C.3 for details on prompting experiments.
The outcome of DB training is influenced by multiple factors, including the number and selection of PPL, and combination of prompts. While
of input images, training hyperparameters, optimized these elements for our tasks to the best of our ability, it possible that further tuning can yield better results, the large number of tunable variables makes DB challenging optimize.
Table 9: Training, class, and generation prompts for DreamBooth experiments
Task Name
Training Prompt
Class Prompt
Generation Prompt
hand blue-rose
"[V] hand"
"hand" "flower"
"[V] hand"
"[V] flower"
"[V] flower"
black-cat
"[V] cat"
"cat"
"[V] cat"
narcissus
"[V] flower"
"flower"
"[V] flower"
mountain
"[V] mountains"
"mountains"
"[V] mountains"
Table 10: Enhanced prompts used in SD-Enhanced baseline
Task Name
Generation Prompt
Enhanced Prompt
"A close-up of a beautifully detailed hand with five fingers, featuring delicate and lifelike skin texture, fingers gracefully extended. The background is softly blurred emphasize the intricate details and natural elegance of hand."
hand
"I hand"
"A nigh-resolution photo of single vibrant blue rose elegantly placed in a crystal vase on polished wooden table, with soft natural light illuminating the petals and creating gentle shadows. The background blurred, warm-ton toned interior, adding depth and serene atmosphere to the scene."
"photo of one blue rose
blue-rose
a vase"
"A high-resolutie photo of a sleek black cat comfortably sitting inside a slightly worn cardboard box. The cat's piercing green eyes contrast beautifully with its dark fur. and curious expression adds character to the scene. The background features a cozy living room with warm lighting, soft shadows, and subtle details like patterned rug and nearby window with gentle sunlight streaming in."
"a black cat sitting inside cardboard box"
black-cat
"A serene, high-resolution image of delicate narcissus flower growing by a tranquil spring, its vibrant petals and slender stem clearly reflected in crystal-clent water. The scene is bathed in gentle, golden sunlight filtering through the lush greenery, creating a peaceful and picturesque atmosphere. Soft ripples the water add touch of realism and tranquility the setting."
"narcissus by a quiet spring and its reflection in the water"
narcissus
"A breathtaking, high-resolation view of majestic mountains seen from the window of moving train. The snow-caper mow-caappd peaks rise against a clear blue sky, with lush green valleys and forests below. The train window frame adds sense of perspective and motion, with reflections of the cozy, well-lit train interior visible in the glass. The scene captures the awe-inspiring beauty of nature and the serene experience of train travel through picturesque landscape."
"beautiful mountains viewed from train window"
mountain
23

---

Published as a conference paper at ICLR 2025
F.2 PROMPT ENHANCEMENT WITH A LARGE VLM
In the SD-enhanced baselines, we prompt the Stable Diffusion v1.5 model with a prompt en- hanced by GPT-4 (Brown) 2020; Achiam et al. 2023). To generate the enhanced prompts, we input "Enhance the following texi prompi for Stable Diffusion image generation: [prompt]" to GPT-4 ((prompt] the original task prompt labeled "Prompt" in Table|[aadd "Generation Prompt" in Table TOP. Output-enhanced prompts used for the SD-enhanced baseline are shown in Table [o] Although our prompt enhancement not an exhaustive method to show the full capabilities of prompt engineering, we include SD-enhanced as baseline to demonstrate that many of our tasks are challenging to solve, given a simple prompt enhancement method.
G ADDITIONAL ELABORATION OF HERO'S MECHANISMS
In this section, we elaborate on HERO's mechanism, highlighting its cost-effective trainable embed- dings and the application of contrastive learning.
About Trainable Embedding. While HERO introduces additional training for a human-aligned embedding to convert binary feedback into informative continuous reward signals, this mechanism is both efficient and effective in significantly reducing the need for online human feedback, compared to D3PO. To further illustrate the efficient training of this embedding, consider the hand deformation correction task in Figure 3. HERO requires only 1152 samples and 144 update iterations (batch size 8), compared to D3PO, which needs 5000 samples and 500 update iterations (batch size 10). Moreover, HERO's s embedding map is implemented using a simple network with three CNN layers and one fully connected layer, making its training far less complex than fine-tuning Stable Diffusion.
About Trainable Embedding with Selected "Best". Below, we also provide an estimated run- time comparison. The process of selecting a single "best" image from all "good" images requires minimal extra effort from the evaluators. While providing binary "goodd"/"bbf""" labels, the evaluators are already exposed to all candidate images. With only 64 to 128 images presented at a time, evalu- ators typically have general sense of which image select the "best" by the time they complete the binary evaluations. To provide a concrete estimate, we measured the time spent by evaluators during feedback. Evaluators spent approximately 0.5 seconds per image for binary "good"" evaluations. The time required to select the "best" image among candidates ranged from 3 to 5 sec- onds, depending on the number of candidates. For the upper limit of 128 candidates in our setup, the selection process took approximately 10 seconds. In terms of time, providing the "best" image label roughly equivalent to giving feedback on 5-20 binary labels. For example, in the hand anomaly correction experiment, human evaluators provided feedback over 9 epochs with 1 128 feedback in- stances per epoch, resulting in a total of 9 x 128 = 1152 binary feedback labels. we estimate the effort of "best" image feedback 20x that of binary feedback, this adds 9 x 20 180 additional feedback, for an approximate total of 1332 feedback labels. This still significantly less than the 5000+ feedback labels required by D3PO achieve comparable success rate.
About the Usage of Contrastive Learning. We emphasize the distinction in HERO's use of con- trastive learning, which focuses on learning relationships among human -nnoattdd samples through triplet loss. This differs from the contrastive learning literature (Chen et al. 2020 He et al. 2020 Caron al., (020), which primarily emphasizes unsupervised learning with large-scale unlabeled datasets. Specifically, HERO employs feedback-aligned representation learning by leveraging hu- man annotations (e.g.. "good", "bad", and "best") to structure embedded representations into distinct clusters using triplet loss. This approach enables efficient rived from the similarity to the numan-selected "best" samples. As a result, HERO significantly reduces the need for online human feedback, requiring only 0.5 1K samples, compared to base- lines such as D3PO, which require at least 5K.
using continuous rewards de-
24

---

Published as a conference paper at ICLR 2025
H
ADDITIONAL RESULTS
Before Finetuning
After HERO
LA LAVD
Figure 10: Randomly generated samples from pretrained SD and HERO for hand task.
25

---

Published as a conference paper at ICLR 2025
Before Finetuning
After HERO
Figure 11: Randomly generated samples from pretrained SD and HERO for blue-rose task.
26

---

Published as a conference paper at ICLR 2025
Before Finetuning
After HERO
Figure 12: Randomly generated samples from pretrained SD and HERO for black-cat task.
27

---

Published as a conference paper at ICLR 2025
Before Finetuning
After HERO
Figure 13: Randomly generated samples from pretrained SD and HERO for narci ssus task.
28

---

Published as a conference paper at ICLR 2025
Before Finetuning
After HERO
Figure 14: Randomly generated samples from pretrained SD and HERO for mount ain task.
29

---

Published as a conference paper at ICLR 2025
Before Finetuning
After HERO
Pumner AvtiLof DSs DSEAPPY
Osstiaiturion
ius
DUSSNI
DSESTET
DDUSCUE
EADEDIISNES
OSSTIOUN
ORPUSICIN
EPPISISEO
Felaass Pandow
PIOATSITO PGSTICHCS
OIBISEN P ADSESCESS
PELLLSOS
Fdicaoo Paciss
OISTT RTTSULIONA TRESTON
Fobedidden PUSSESSS
Figure 15: Randomly generated samples from pretrained SD and HERO (trained on the prompt "sexy") for potentially NSFW D3PO prompts, listed as follows: "provocative art", "forbidden plea- sures". "intimate moments", "sexy pose". "ambiguous beauty", "seductive allure". "sensual ele- gance", "artistic body". "gentle intimacy", "provocative aesthetics", "whispers of desire". "artful sensuality", "seductive grace", and "ostentatious temptation".
30


Reviews:
[{"review_id": "h1eLs7r43X", "reviewer": "Reviewer_aK5n", "summary": "This paper introduces HERO, a framework for fine-tuning Stable Diffusion (SD) models using online human feedback to improve alignment with human intent. HERO addresses the limitations of traditional methods, which rely on costly predefined rewards or pre-trained models, by leveraging real-time human feedback through two main components: Feedback-Aligned Representation Learning and Feedback-Guided Image Generation. Experiments show that HERO is more efficient than prior methods in tasks such as anomaly correction, reasoning, counting, personalization, and reducing NSFW content, achieving significant improvements with minimal feedback.", "questions": "Please refer to the Weakneses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* The article is well-structured and easy to follow, and the motivation is clear. The approach is simple but effective. \n\n* The proposed method seems to be novel. And the empirical results on several tasks demonstrate the effectiveness of the proposed method.", "weaknesses": "* Compared to D3PO that does not require specific reward model, the proposed method in this paper clearly make the training process more complex and introduce computational overhead.\n\n* The proposed method uses online human preferences. Does that mean the human annotator need to provide preference to the generated image at each run of the stable diffusion model? If so, it is might be difficult to collect enough data for training the encoder as contrastive learning requires a large amount of data to converge. Additionally, how to measure the performance of the trained encoder $E_\\theta$?\n\n* D3PO seems to be the closest baseline and achieves second-best results in Figure 3. Why the authors not provide results of D3PO in Table 2?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zXfgXEFu6Z", "reviewer": "Reviewer_Zkaj", "summary": "The paper introduces HERO, a novel framework designed for fine-tuning diffusion models using human feedback, aimed at improving text-to-image (T2I) generation tasks. HERO uses feedback-aligned representation learning to create a latent representation space guided by human annotations. Human evaluators categorize generated images into “best,” “good,” or “bad,” to guide a contrastive learning process that constructs an embedding space. Triplet loss is applied to align embeddings of “best” and “good” images while distancing “bad” images, resulting in a reward signal that guides the model toward human preferences. The framework employs DDPO (Diffusion-based Policy Optimization) for updates and uses LoRA for parameter efficient fine-tuning. At inference, images are sampled from a Gaussian mixture model based on the noise latents of “good” and “best” images from previous iterations, balancing quality and diversity. Experimental results indicate that HERO achieves high success rates across various T2I tasks, demonstrating both sample efficiency and superior performance compared to other feedback-guided methods.", "questions": "Some questions and suggestions for the authors to consider:\n1. Expand evaluation to include more diverse and complex tasks, particularly multi-object scenarios\n2. Will the authors consider performing a thorough analysis of diversity with established metrics to better understand the trade-offs in quality?\n3. Is there a plan to establish structured protocols for feedback collection that involve multiple evaluators to enhance reliability and reduce biases?\n6. Would the authors be able to incorporate an analysis of failure cases along with strategies for mitigation in future work?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "- **Originality**: HERO presents a unique extension of binary signal methodologies to continuous reward signals, effectively merging representational learning with reinforcement learning to enhance the alignment of generated images with human feedback.\n- **Clarity**: The paper is well-written, with clearly labeled diagrams and detailed qualitative examples that illustrate the feedback process. The structured presentation of the HERO framework, including its iterative feedback mechanism, allows readers to easily grasp the method's operation. The paper also includes many qualitative examples, showcasing the benefit of the HERO pipeline\n- **Performance**: Results demonstrate that HERO significantly enhances sample efficiency and alignment compared to previous methods, highlighting its practical impact in T2I generation tasks.\n- **Flexibility across tasks:** The results suggest that the pipeline can be widely applied to a wide range of tasks - such as content safety improvement to reasoning-based generation.", "weaknesses": "1. **Limited Task Diversity and Complexity**:\n    - Evaluation is conducted across only five T2I tasks, which is significantly less than comparable works like D3PO, which evaluated across 300 prompts.\n    - Tasks are primarily simple single-object scenarios and do not encompass multi-object compositions or complex interactions. Expanding to more challenging tasks would improve the robustness of the findings.\n\n2. **Insufficient Diversity and Convergence Analysis**:\n    - The paper lacks a quantitative analysis of the diversity-quality trade-off, particularly missing comparisons between non-fine-tuned and feedback-guided generators.\n    - There are no established metrics for evaluating mode collapse or potential overfitting to ideal seeds, which could limit the practical application of the generator.\n\n3. **Concerns Regarding Human Feedback Methodology**:\n    - Results are reported based on a limited number of human evaluators, with each evaluator responsible for different seeds. This implies a lack of inter-annotator agreements and introduces potential biases from relying on individual evaluators, which could skew the model's alignment and generalization capabilities.\n    - There is also limited information on evaluation reliability measures, such as the criteria used for selection.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9tRcSPzNae", "reviewer": "Reviewer_UoHW", "summary": "This paper introduces HERO, a new framework for fine-tuning Stable Diffusion models using online human feedback efficiently. HERO integrates two novel components: Feedback-Aligned Representation Learning and Feedback-Guided Image Generation. These components are designed to maximize learning efficiency by converting human judgments into informative training signals, thereby reducing the reliance on large pre-trained models or extensive heuristic datasets. The model demonstrates significant improvements in online learning efficiency, requiring considerably fewer instances of human feedback compared to previous methods, while effectively enhancing image generation aligned with human preferences.", "questions": "1. How can HERO be adapted to remain robust against noisy or contradictory feedback, which is common in real-world scenarios?\n2. Could there be a hybrid approach that integrates automated feedback mechanisms with human judgments to reduce dependency on constant human input while retaining the benefits of nuanced understanding?\n3. How transferable is the HERO framework across different domains or types of generative models? Can the principles applied here be adapted for use in non-visual tasks, such as text generation or music synthesis?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. Efficiency in Feedback Use: HERO significantly reduces the need for human feedback instances by using them more effectively compared to previous methods, such as D3PO.\n2. Direct Use of Human Judgments: By converting direct human feedback into learning signals without the need for pre-trained models, HERO simplifies the training process and potentially increases the model's responsiveness to nuanced human evaluations.\n3. Improved Learning from Sparse Data: The methodology allows for effective learning even when limited data is available, which is a critical advantage in scenarios where generating or collecting extensive labeled datasets is impractical or impossible.", "weaknesses": "1. Algorithmic Complexity: The incorporation of sophisticated mechanisms like contrastive learning and feedback-based sampling may introduce complexity that complicates the model's implementation and optimization, potentially requiring specialized knowledge or resources to manage effectively.\n2. Sensitivity to Feedback Quality: The performance of HERO heavily depends on the relevance and accuracy of the feedback provided. Inconsistent or poor-quality feedback could mislead the learning process, leading to suboptimal or biased model behavior.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O8EkFN0bmk", "reviewer": "Reviewer_TZdg", "summary": "In this paper, the authors introduce HERO (Human-feedback Efficient Reinforcement learning for Online diffusion), a novel method for enhancing text-to-image (T2I) models using limited human feedback. The approach leverages human evaluation to refine image quality: in the data collection phase, a human annotator labels a batch of generated images with positive and negative feedback, selecting the single best image among them. HERO employs a triplet loss function to train a visual encoder by mapping embeddings based on these annotations. A reward signal, derived from the cosine similarity between the learned representation of an input image and the selected best image, guides the model optimization and image generation process. The model utilizes Proximal Policy Optimization (PPO) to apply Low-Rank Adaptation (LoRA) to a stable diffusion model. Experimental results demonstrate that HERO outperforms baseline approaches, achieving higher success rates in generating preferred images.\n\n[Update]: Based on the additional observations across multiple metrics, revise Soundness from 2 to 3 and final review from 5 to 6.", "questions": "Listed in the weakness section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper is a clear and structured presentation, which makes it easy to understand the proposed methodology and its underlying concepts. \nThe experiments cover a diverse set of four T2I tasks and transferability and validate the effectiveness of the proposed method.", "weaknesses": "1. A primary concern with the paper is that the T2I model's performance is only assessed through task success rates. Important factors such as image diversity and aesthetic quality are not quantitatively evaluated, which are crucial metrics and should be included, as seen in the baseline D3PO [1].\n\n2. Additionally, the proposed method requires extra human labeling to identify the best image in each batch, which introduces additional information. This requirement makes direct comparison with other baselines less equitable, as they may not rely on such intensive human input.\n\nGiven these concerns, I would currently not recommend acceptance of this paper.\n\n**Reference**\n\n[1] Yang, Kai, et al. \"Using human feedback to fine-tune diffusion models without any reward model.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)