Training Diffusion Models with Reinforcement Learning
Authors: Kevin Black
Michael Janner
Yilun Du
Ilya Kostrikov
Sergey Levine
Venue: ICLR 2024 poster (2024)
Keywords: reinforcement learning
RLHF
diffusion models
Primary area: generative models

Abstract: Diffusion models are a class of flexible generative models trained with an approximation to the log-likelihood objective. However, most use cases of diffusion models are not concerned with likelihoods, but instead with downstream objectives such as human-perceived image quality or drug effectiveness. In this paper, we investigate reinforcement learning methods for directly optimizing diffusion models for such objectives. We describe how posing denoising as a multi-step decision-making problem enables a class of policy gradient algorithms, which we refer to as denoising diffusion policy optimization ( DDPO), that are more effective than alternative reward-weighted likelihood approaches. Empirically, DDPO can adapt text-to-image diffusion models to objectives that are difficult to express via prompting, such as image compressibility, and those derived from human feedback, such as aesthetic quality. Finally, we show that DDPO can improve prompt-image alignment using feedback from a vision-language model without the need for additional data collection or human annotation. The project’s website can be found at http://rl-diffusion.github.io.

TRAINING DIFFUSION MODELS WITH REINFORCEMENT LEARNING
Kevin Black*! Michael Janner
Ilya Kostrikov' 2 Massachusetts Institute of Technology
Yilun Du?
Sergey Levine
1 University of California, Berkeley
(kvablack, janner, kostrikov, sergey. ( edu yilunduemit.edd
ABSTRACT
Diffusion models are class of flexible generative models trained with approximation to the log-likelihooo objective. However, most use cases diffusion models are not concerned with likelihoods, but instead with downstream objectives such as human-perceived image quality or drug effectiveness. In this paper, we investigate reinforcement learning methods for directly optimizing diffusion models for such objectives. We describe how posing denoising as multi-step decision- making problem enables class of policy gradient algorithms, which we refer to as denoising diffusion policy optimization (DDPO), that are more effective than alternative reward-weipe swwrd--eighttd likelihood approaches. Empirically, DDPO can adapt text-to-image diffusion models to objectives that are difficult to express via prompting, such as image compressibility, and those derived from human feedhack, such as aesthetic quality. Finally, we show that DDPO can improve prompt-image alignment using feedback from a vision-language model without the need for additional data collection or human annotation. The project's website can be found at http::/rett dif fusion. www.onn..iiiiii.iioooo..
INTRODUCTION
Diffusion probabilistic models (Sohl-Dickstein et al., 2015) have recently emerged as the de facto standard for generative modeling in continuous domains. Their flexibility in representing complex, high-ding ligh-dimensional distributions has led to the adoption of diffusion models in applications including image and video synthesis (Ramesh al., 2021: Saharia et al., 2022: Ho et al., 2022), drug and material design (Xu et al., 2021; Xie et al., 2021; Schneuing et al., 2022), and continuous control (Janner et al., 2022; Wang et al., 2022; Mansen-Estruch al., 2023). The key idea behind diffusion models is to iteratively transform a simple prior distribution into target distribution by applying sequential denoising process. This procedure conventionally motivated as a maximum likelihood estimation problem, with the objective derived as variational lower bound on the log-likelihood of the training data.
of diffusion models not directly concerned with likelihoods, but instead with downstream objective such as suman-perceiver image quality or drug effectiveness. In this paper, consider the problem of training diffusion models satisfy such objectives directly, opposed to matching a data distribution. This problem is challenging because exact likelihood computation with diffusion models intractable, making difficult to apply many conventional reinforcement learning (RL) algorithms. We instead propose to frame denoising as multi-step decision-making task, using the exact likelihoods each denoising step place of the approximate likelihoods induced by full denoising process. We present a policy gradient algorithm, which we refer to as denoising diffusion policy optimization (DDPO). that can optimize a diffusion model for downstream tasks using only black-box reward function.
However, most
We apply our algorithm the finetuning of large text-to-image diffusion models. Our initial evaluation focuses tasks that are difficult to specify via prompting, such as image compressibility, and those derived from human feedback, such as aesthetic quality. However, because many reward functions of interest are difficult to specify programmatically, finetuning procedures often rely on large-scale human labeling efforts to obtain a reward signal (Ouyang et al., 2022). In the case of text-to-image diffusion, we propose method for replacing such labeling with feedback from vision-langue model (VLM). Similar to RLAIF finetuning for language models (Bai et al., 2022b), the resulting procedure allows for diffusion models be adapted to reward functions that would otherwise require

---

Compressibilit; llama
Aesthetic Quality: rabbit
Prompt-Image Alignment: a raccoon washing dishes
Figure (Reinforcement learning for diffusion models) We propose reinforcement learning algorithm, DDPO, for optimizing diffusion models on downstream objectives such as compressibility. aesthetic quality, and prompt-image alignment determined by vision-language models. Each row shows a progression of samples for the same prompt and random seed over the course of training.
additional human annotations. We use this procedure to improve prompt-image alignment for unusual subject-setting compositions.
Our contributions are as follows. We first present the derivation and conceptual motivation of DDPO. We then document the design of various reward functions for text-to-image generation, ranging from simple computations to workflows involving large VLMs, and demonstrate the effectiveness of DDPO compared to alternative seward-weighter likelihood methods in these settings. Finally, demonstrate the generalization ability of our finetuning procedure to unseen prompts.
RELATED WORK
Diffusion probabilistic models. Denoising diffusion models (Sohl-Dickstein et al., 2015; Ho al., 2020) have emerged
effective class of generative models for modalities including images (Ramesh et al., 2021; Saharia et al., 2022), videos (Ho et al., 2022; Singer et al., 2022), 3D) shapes (Zhou al., 2021; Zeng et al., 2022), and robotic trajectories (Janner et al., 2022: Ajay et al., 2022; Chi et al., 2023). While the denoising objective is conventionally derived as an approximation likelihood, the training of diffusion models typically departs from maximum likelihood in several ways (Ho et al., 2020). Modifying the objective to more strictly optimize likelihood (Nichol & Dhariwal, 2021; Kingma al., 2021) often leads to worsened image quality, likelihood is not faithful proxy for visual quality. In this paper, we show how diffusion models can be optimized directly for downstream objectives.
Controllable generation with diffusion models. Recent progress in text-to-image diffusion mod- els (Ramesh et al., 2021; Saharia et al., 2022) has enabled fine-grained high-resolution image synthesis. To further improve the controllability and quality of diffusion models, recent approaches have in- vestigated finetuning on limited user esovided data (Ruiz et al., 2022), optimizing text embeddings for concepts (Gal al., 2022), composing models (Du et al., 2023; Liu al., 2022), adapters for additional input constraints (Zhang & Agrawala, 2023), and inference sitreence-iime techniques such as classifier (Dhariwal & Nichol, 2021) and classifier-free (Ho & Salimans, 2021) guidance.
2

---

Reinforcement learning from human feedback. A number of works have studied using human feedback to optimize models in settings such as simulated robotic control (Christiano et al., 2017), game-playing (Knox & Stone, 2008), machine translation (Nguyen et al., 2017), citation retrieval question-answering (Nakano et al., 2021), summarization
(Menick et al., 2022), browsing-basee (Stiennon al., 2020; Ziegler al., 2019), cnstructioo-followiii (Ouyang al., 2022), and alignment with specifications (Bai et al., 2022a). Recently, Lee et al. (2023) studied the alignment of text-to- image diffusion models to human preferences using method based on geward-weighted likelihood maximization. In our comparisons, their method corresponds to one iteration of the reward- -eiihhe regresion (RWR) method. Our results demonstrate that DDPO significantly outperforms even multiple iterations of weighted likelihood maximization (RWR-style) optimization.
Diffusion models as sequential decision-making processes. Although predating diffusion models, Bachman & Precup (2015) similarly posed data generation sequential Decision-makinn problem and used the resulting framework to apply reinforcement learning methods to image generation. More recently, Fan & Lee (2023) introduced policy gradient method for training diffusion models. However, this paper aimed to improve data distribution matching rather than optimizing downstream objectives, and therefore the only reward function considered was a GAN-like discriminator. In concurrent work to ours, DPOK (Fan et al., 2023) built upon Fan & Lee (2023) and Lee et al. (2023) better align text-to-image diffusion models human preferences using policy gradient algorithm. Like Lee et al. (2023), DPOK only considers a single preference- 2023); additionally, their work studies KL-regularization and primarily focuses on training a different diffusion model for each prompt. In contrast, we train on many prompts at once (up to 398) and demonstrate generalization to many more prompts outside of the training set. Furthermore, we study how DDPO can be applied to multiple reward functions beyond those based on human feedback, including how rewards derived automatically from VLMs can improve prompt-image alignment. We provide direct comparison to DPOK in Appendix C.
reward function (Xu et al.,
PRELIMINARIES
In this section, we provide a brief background on diffusion models and the RL problem formulation.
3.1 DIFFUSION MODELS
In this work, consider conditional diffusion probabilistic models (Sohl- Sich--Dickstein et al., 2015; Ho et al., 2020), which represent a distribution p(xolc) over dataset of samples Xo and corresponding contexts c. The distribution is modeled as the reverse of a Markovian forward process q(x, x,-1), 1).
which iteratively adds noise to the data. Reversing the forward process can be accomplished by training a neural network ee (xx, c) with the following objective:
LDDPM (f) E c)xxxx)) t~2(0,,T), x, xx [Ila((x. () (xo, c, []]]
(1)
where j is the posterior mean of the forward process, a weighted average of xo and x This objective is justified as maximizing a variational lower bound on the log-likelihood of the data (Ho et al., 2020).
Sampling from diffusion model begins with drawing random XT A/ I) and following the reverse process pe (x Xr, c) to produce a trajectory (xT, Xy-1,... Xo) ending with sample Xo. The sampling process depends not only on the predictor μg but also the choice of sampler. Most popular samplers (Ho et al., 2020; Song et al., 2021) use an isotropic Gaussian reverse process with a fixed dimestep-dependent variance:
(x Xt, c) = ((x p4 (xs, (),,,(t)),
(2)
3.2 MARKOV DECISION PROCESSES AND REINFORCEMEN LEARNING
A Markov decision process (MDP) is formalization of sequential decision-making problems. An MDP is defined by a tuple (S, .., Po. P, R). in which S is the state space, A is the action space, Po is the distribution of initial states, P is the transition kernel, and R is the reward function. At each timestep t, the agent observes a state s, E S. takes an action a, E A, receives a reward R(s,, az). P( 82, a.). An agent acts according to policy xt s).
and transitions
state 8
3

---

As the agent acts in the MDP, it produces trajectories, which are sequences of states and actions т = (So. ag. $1, ,
Sy , ar)))r)............,,..r,...)a ).). The reinforcement learning (RL) objective is for the agent to maximize JRL. (T), the expected cumulative reward over trajectories sampled from its policy:
(x) ((pplr)s)
Σ R(s ax)
4 REINFORCEMENT LEARNING TRAINING OF DIFFUSION MODELS
We now describe how RL algorithms can be used to train diffusion models. We present two classes of methods and show that each corresponds different mapping of the denoising process the MDP framework.
4.1 PROBLEM STATEMENT
assume pre-existing diffusion model, which may be pretrained or randomly initialized. Assuming fixed sampler, the diffusion model induces sample distribution (x c). The denoising diffusion RL objective is to maximize a reward signal r defined on the samples and contexts:
JDDRL (0) eep))() xo-pe(xole) [r(Xo; c)]
for some context distribution p(c) of our choosing.
4.2 REWARD-WEIGHTER REGRESSION
To optimize JDDRL with minimal changes to standard diffusion model training, we can use the denoising loss CDDPM (Equation 1), but with training data Xo (x c) and an added weighting that depends on the reward (xxo, c). Lee et al. (2023) describe single-round version of this procedure for diffusion models, but in general this approach can be performed for multiple rounds of alternating sampling and training, leading to an online RL method. We refer to this general class of algorithms as reward-weighted regression (RWR) (Peters & Schaal, 2007).
standard weighting scheme uses exponentiated rewards to ensure nonnegativity,
1 Z exp (Br(xo- c)),
WRWR (Xo: c)
where I is inverse temperature and Z is normalization constant. We also consider simplified weighting scheme that uses binary weights,
Wsparse (Xo, c) [r(xo. c) 2 c].
where C' is reward threshold determining which samples used for training. In supervised learning terms, this is equivalent to repeated filtered finetuning on training data coming from the model.
Within the RL formalism, the RWR procedure corresponds the following one-step MDP:
S é
- X
((a
s) po (x,
Po(s) A p(c) R(s, a) & r(xo.c) c)
with transition kernel P that immediately leads absorbing termination state. Therefore, maximizing (DDRL(()) is equivalent to maximizing the RL objective (RL(T) in this MDP.
From RL literature, weighting a log-likelihood objective by WRWR is known approximately max- imize JRL(TT) subject to KL divergence constraint on (Nair et al., 2020). However, LDDPM (Equation 1) does involve an exact log- likelihood is instead derived as a variational bound on log pe (x c). Therefore, the RWR procedure applied to diffusion model training is not theoretically justified and only optimizes JDdRL. very approximately.
4.3 DENOISING DIFFUSION POLICY OPTIMIZATION
RWR relies approximate log- likelihood because it ignores the sequential nature of the denoising process, only using the final samples Xg. In this section, we show how the denoising process can be reframed as multi-step MDP, allowing directly optimize JDdRl using policy gradient
4

---

estimators. This follows the derivation in Fan & Lee (2023), who prove equivalence between their method and a policy gradient algorithm where the reward a GAN-like discriminator. We present a general framework with an arbitrary reward function, motivated by desire to optimize arbitrary downstream objectives (Section 5). We refer to this class of algorithms as denoising diffusion policy optimization (DDPO) and present two variants based on specific gradient estimators.
Denoising as a multi-step MDP. We map the iterative denoising procedure to the following MDP:
St (c, (x)
r(a 8t) _ po (x
Xx, c)
P(s++1 8t, ar) - (6e, 1,,0
R(s,, as) - [r(x, c) tf
A X
(so) (p(c), (((O,T
otherwise
in which by is the Dirac delta distribution with nonzero density only at y. Trajectories consist of T timesteps, after which P leads termination state. The cumulative reward of each trajectory is equal to r(Xo, c), so maximizing JDDRL (0) ))) is equivalent to maximizing (R) in this MDP.
The benefit of this formulation is that if use standard sampler with po xx X c) parameterized as in Equation 2, the policy π becomes an isotropic Gaussian as opposed to the arbitrarily complicated distribution pe (Xa c) as is in the RWR formulation. This simplification allows for the evaluation of exact logelikelssoods and their gradients with respect to the diffusion model parameters.
Policy gradient estimation. With access to likelihoods and likelihood gradients, we can make direct Monte Carlo estimates of VoJDDRL. Like RWR, DDPO alternates collecting denoising trajectories {XT, XT-1, } via sampling and updating parameters via gradient descent.
The first variant of DDPO, which call DDPOsF. uses the score function policy gradient estimator, also known as the likelihood ratio method or REINFORCE (Williams, 1992; Mohamed et al., 2020):
V JDDRL
E
V log Xx. c) r(xo, c)
(DDPOse)
t=
where the expectation is taken over denoising trajectories generated by the current parameters 6.
However, this estimator only allows for step of optimization per round of data collection, the gradient must be computed using data generated by the current parameters. To perform multiple steps of optimization, may
importance sampling estimator (Kakade & Langford, 2002):
T
V JDDDRL = E
Σ
Xx, c) 1=1 Pea (x:-1
Po (xx-
Ve Volope(xx-11 x,, c) r(xo. c)
(DDPO1s)
X2,
where the expectation is taken estimator becomes inaccurate if pe deviates too far from Pe... which can be addressed using trust regions (Schulman et al., 2015) to constrain the size of the update. practice, we implement the trust region via clipping, as in proximal policy optimization (Schulman et al., 2017).
denoising trajectories generated by the parameters Oala- This
5 REWARD FUNCTIONS FOR TEXT-TO-IMAGE DIFFUSION
In this work, we evaluate our methods on text-to-image diffusion. Text-to-image diffusion serves valuable test environment for reinforcement learning due the availability of large pretrained models and the versatility of using diverse and visually interesting reward functions. In this section, outline our selection of reward functions. We study a spectrum of reward functions of varying complexity, ranging from those that are straightforward to specify and evaluate to those that capture the depth real-world downstream tasks.
5.1 COMPRESSIBILITY AND INCOMPRESSIBILITY
The capabilities of text-to-image diffusion models are limited by the co-occurrences of text and images in their training distribution. For instance, images are rarely captioned with their file size, making it impossible to specify desired file size via prompting. This limitation makes reward functions based on file size convenient case study: they are simple to compute, but not controllable through the conventional methods of likelihood maximization and prompt engineering.
5

---

"what is happening in this imagee 2
LLaVA
BERTSScore
similariy--hmon reward
"a monkey
Diffusion Model
minkary washing dishro..."
Figure 2 (VLM reward function) Illustration of the VLM-based reward function for prompt-image alignment. LLaVA (Liu et al., 2023) provides a short description of a generated image; the reward is the similarity between this description and the original prompt as measured by BERTScore (Zhang et al., 2020).
We fix the resolution of diffusion model samples at 512x512, such that the file size is determined solely by the compressibility of the image. We define two tasks based on file size: compressibility. in which the file size of the image after JPEG compression is minimized, and incompressibility. in which the same measure is maximized.
5.2 AESTHETIC QUALITY
To capture reward function that would be useful to human user, we define task based on perceived aesthetic quality. use the LAION aesthetics predictor (Schuhmann, 2022), which trained 176,000 human image ratings. The predictor is implemented top of CLIP embeddings (Radford et al., 2021). Annotations range between and 10, with the highest-rated images mostly containing artwork. Since the aesthetic quality predictor is trained human judgments, this task constitutes reinforcement learning from human feedback (Ouyang et al., 2022; Christiano et al., 2017; Ziegler et al., 2019).
linear model on
5.3 AUTOMATED PROMPT ALIGNMENT WITH VISION-LANGUAGE MODELS
A very However, specifying reward that captures generic prompt alignment is difficult, conventionally requiring large-scale human labeling efforts. We propose using an existing VLM to replace additional human annotation. This design is inspired by recent work on RLAIF (Bai et al., 2022b), in which language models are improved using feedback from themselves.
reward function for training text-to-image model is prompt-image alignment.
We use LLaVA (Liu al., 2023), state-of-the of-the-arrr art VLM, to describe an image. The finetuning reward is the BERTScore (Zhang et al., 2020) recall metric, measure of semantic similarity. using the prompt as the reference sentence and the VLM description as the candidate sentence. Samples that more faithfully include all of the details of the prompt receive higher rewards, to the extent that those visual details are legible to the VLM.
In Figure 2, we show one simple question: "what is happening in this image?" While this captures the general task of prompt-image alignment, in principle any question could be used to specify complex hard-to-defin reward functions for particular language model to automatically generate candidate questions and evaluate responses based on the prompt. This framework provides flexible interface where the complexity of the reward function is only limited by the capabilities the vision and language models involved.
case. One could even employ
EXPERIMENTAL EVALUATION
The purpose of our experiments is to evaluate the effectiveness of RL algorithms for finetuning diffusion models to align with a variety of user-specified objectives. After examining the viability of the general approach, focus the following questions:
1. How do variants of DDPO compare to RWR and to each other?
2. Can VLMs allow for optimizing rewards that are difficult to specify manually?
3. Do the effects of RI. finetuning generalize prompts not during finetuning?

---

Prefrained
Aesthetic Quality
Compressibility
Incompressibility
Figure 3 (DDPO samples) Qualitative depiction of the effects of RL finetuning different reward functions. DDPO transforms naturalistic images into stylized artwork to maximize aesthetic quality, removes background content and applies foreground smoothing to maximize compressibility, and adds high-frequency noise maximize incompressibility.
Aesthetic Quality
JPEG Compressibility
JPEG Incompressibility
-25
LAION Actiarte
-10
Fitetto (1b)
-10
Negazive
-125
100
10k
201c
32k
11k
10kc
1/k.
16k
10k
10k
45k
Reward Queeies
Reward Queries
Reward Queries
DDPO1s
DDPOsF
RWR
RWR RWR... sparse
Figure 4 (Finetuning effectiveness) The relative effectiveness of different RL algorithms on three reward functions. We find that the policy gradient variants, denoted DDPO, are more effective optimizers than both RWR variants.
6.1 ALGORITHM COMPARISONS
We begin by evaluating all methods on the compressibility, incompressibility, and aesthetic quality lasks, as these tasks isolate the effectiveness of the RL approach from considerations relating to the VLM reward function. We use Stable Diffusion v1.4 (Rombach al., 2022) as the base model for all experiments. Compressibility and incompressibilityy prompts are sampled uniformly from all 398 animals the ImageNet- 1000 (Deng et al., 2009) categories. Aesthetic quality prompts are sampled uniformly from a smaller set of 45 common animals.
As shown qualitatively in Figure 3, DDPO able to effectively adapt a pretrained model with only the specification of reward function and without any further data curation. The strategies found to optimize each reward are nontrivial: for example, to maximize LAION-predica DDPO transforms model that produces naturalistic images into one that produces artistic drawings. To maximize compressibility, DDPO removes backgrounds and applies smoothing to what remains. To maximize incompressibility, DDPO finds artifacts that difficult for the JPEG compression algorithm to encode, such as high-frequency noise and sharp edges. Samples from RWR are provided in Appendix G for comparison.
aesthetic quality,
7

---

dolphin riding bike
Prompt Alignment
0.34
0.81
an and playing chess
0.78
0.75
0.72
0.00
a bear washing dishes
10k
20%
300
400.
50k
Reward Queries
riding a bike
playing chess
washing dishes
Figure 5 (Prompt alignment) (L) Progression of samples for the same prompt and random seed over the course of training. The images become significantly more faithful the prompt. The samples also adopt a cartoon-like style, which we hypothesize is because the prompts are more likely depicted as illustrations than realistic photographs the pretraining distribution. (R) Quantitative improvement of prompt alignment. Each thick line is the average score for activity, while the faint lines show average scores for a few randomly selected individual prompts.
We provide a quantitative comparison of all methods in Figure We plot the attained reward function of the number of queries to the reward function, as reward evaluation becomes the limiting factor in many practical applications. DDPO shows clear advantage over RWR on all tasks, demonstrating that formulating the denoising process as multi-step MDP and estimating the policy gradient directly is more effective than optimizing a reward- -weihhed variational bound on log-likelihood. Within the DDPO class, the importance sampling estimator slightly outperforms the score function estimator, likely due to the increased number of optimization steps. Within the RWR class, the performance of weighting schemes is comparable, making the sparse weighting scheme preferable on these tasks due to its simplicity and reduced resource requirements.
6.2 AUTOMATED PROMPT ALIGNMENT
We next evaluate the ability of VLMs, in conjunction with DDPO, automatically improve the image-prompt alignment of the pretrained model without additional human labels. We focus on DDPO1s for this experiment, as found it be the most effective algorithm in Section 6. The prompts for this task all have the form "a(n) [animal] [activity]" ", where the animal comes from the same list of 45 common animals used in Section 6.1 and the activity is chosen from list of activities: "riding a bike", "playing chess", and "washing dishes".
The progression of finetuning depicted in Figure 5. Qualitatively, the samples come depict the prompts much more faithfully throughout the course of training. This trend is also reflected quanti- tatively. though less salient as small changes in BERTScore can correspond large differences in relevance (Zhang et al., 2020). It is important to note that some of the prompts in the finetuning set, such as dolphin riding a bike". ", had zero success rate from the pretrained model; if trained in isolation, this prompt would be unlikely to ever improve because there would be no reward signal. It was only via transferrable learning across prompts that these difficult prompts could improve.
Nearly all of the samples become more cartoon-like or artistic during finetuning. This was not optimized for directly. We hypothesize that this may be a function of the pretraining distribution (one would expect depictions of animals doing everyday activities to be more commonly cartoon-lit like than photorealistic) or the reward function (perhaps LLa VA has an easier time recognizing the content of simple cartoon-lite like images).
8

---

Aesthetic Quality (New Animals)
Aesthetic Quality (Nou-Aniimlls)
Prompl Aligument (New Scenurios)
Pretrained
Finetuned
Figure (Generalization) Finetuning on a limited set of animals generalizes to both new animals and non-animal everyday objects. The prompts for the rightmost two columns are "a capybara washing dishes" and "a duck taking an exam". A quantitative analysis provided in Appendix F, and more samples provided in Appendix G.
6.3 GENERALIZATION
RL finetuning on large language models has been shown produce interesting generalization properties; for example, instruction finetuning almost entirely in English has been shown to improve capabilities other languages (Ouyang et al., 2022). It is difficult to reconcile this phenomenon with our current understanding of generalization; it would priori seem more likely for finetuning to have an effect only on the finetuning prompt set or distribution. In order to investigate the same phenomenon with diffusion models, Figure 6 shows set of DDPO-finetune model samples corresponding to prompts that were not seen during finetuning. In concordance with instruction- following transfer in language modeling, with prompt distributions as narrow as 45 animals and 3 activities. We find evidence generalization animals outside of the training distribution, non-animal everyday objects, and in the case of prompt-image alignment, even to novel activities such as "taking an exam""
find that the effects of finetuning do generalize, even
7 DISCUSSION AND LIMITATIONS
We presented an RL-based framework for training denoising diffusion models to directly optimize a variety of reward functions. By posing the iterative denoising procedure multi-step decision- making problem, were able to design a class of policy gradient algorithms that are highly effective at training diffusion models. We found that DDPO was an effective optimizer for tasks that are difficult to specify via prompts, such as image compressibility, and difficult to evaluate programmatically, such as semantic alignment with prompts. To provide an automated way to derive rewards, we also proposed method for using VLMs to provide feedback the quality of generated images. While evaluation considers a variety of prompts, the full range of images in our experiments was constrained (e.g., animals performing activities). Future iterations could expand both the questions posed the VLM, possibly using language models to propose relevant questions based on the prompt, as well as the diversity of the prompt distribution. We also chose not to study the problem of overoptimization. a common issue with RL finetuning in which the model diverges too far from the original distribution be useful (see Appendix A); we highlight this as an important area for future work. We hope that this work will provide step toward more targeted training of large generative models, where optimization via RI. can produce models that are effective achieving user-specified goals rather than simply matching entire data distribution.
Broader Impacts. Generative models can be valuable productivity aids, but may also pose harm when used for disinformation, impersonation, or phishing. Our work aims to make diffusion models more useful by enabling them to optimize user-specified objectives. This adaptation has beneficial applications, such as the generation of more understandable educational material, but may also be used maliciously, in ways that we do not outline here. Work on the reliable detection of synthetic content remains important mitigate such harms from generative models.

---

8 ACKNOWLEDGEMENTS
This work was partially supported by the Office of Naval Research and computational resource donations from Google via the TPU Research Cloud (TRC), Michael Janner was supported by fellowship from the Open Philanthropy Project. Yilun Du and Kevin Black were supported by fellowships from the National Science Foundation.
CODE REFERENCES
We used the following open-source libraries for this work: NumPy (Harris et al., 2020), JAX (Bradbury et al., 2018), Flax (Heek et al., 2023), optax (Babuschkin et al., 2020), h5py (Collette, 2013), transformers (Wolf et al., 2020), and diffusers (von Platen et al., 2022).
REFERENCES
Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. con- ditional generative modeling all you need for decision-making? arXiv preprint arXiv:221 1.15677, 2022.
Igor Babuschkin, Kate Baumli, Alison Bell, Surya Bhupatiraju, Jake Bruce, Peter Buchlovsky, David Budden, Trevor Cai, Aidan Clark, Ivo Danihelka, Antoine Dedieu, Claudio Fantacci, Jonathan Godwin, Chris Jones, Ross Hemsley, Tom Hennigan, Matteo Hessel, Shaobo Hou, Steven Kapturowski, Thomas Keck, Jurii Kemaev, Michael King, Markus Kunesch, Lena Martens, Hamza Merzic, Vladimir Mikulik, Tamara Norman, George Papamakarios, John Quan, Roman Ring. Francisco Ruiz, Alvaro Sanchez, Rosalia Schneider, Eren Sezener, Stephen Spencer, Srivatsan Srinivasan, Wojciech Stokowiec, Luyu Wang, Guangyao Zhou, and Fabio Viola. The DeepMind JAX Ecosystem, 2020. URL http::ttpppteee. http::/github..bbbbb/// http://tttubu.oomm/eenmnnd
Philip Bachman and Doina Precup. Data generation as sequential decision making. Advances in Neural Information Processing Systems, 28, 2015.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk. Nelson Elhage, Zac Hatfield-oodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint ar arxiv:2204.0 2tt.:22244.8822,,, 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Ratfield-Dodee Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional Al: Harmlessness from AI feedback. arXiv preprint arXiv:2212.0807: 2022b.
Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 243-852, 2023.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Minn and Qiao Zhang. JAX: composable transformations of Python+NumPP programs, 2018. URL www.thwww..oooooooggggoo cwm cowggogg/////oo......ooo
http://github...
10

---

Cheng Chi, Siyuan Feng. Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. arXiv preprint arXiv:2303.04417,. 2023.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Neural Information Processing Systems, 2017.
Andrew Collette. Python and HDF5. O'Reilly, 2013.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In Conference on Computer Vision and Pattern Recognition, 2009,
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat GANs on image synthesis. In
Advances in Neural Information Processing Systems, 2021.
Yilun Du, Conor Durkan, Robin Strudel, Joshua B Tenenbaum, Sander Dieleman, Rob Fergus, Jascha Sohl-Dickstein, Arnaud Doucet, and Will Grathwohl. Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and meme. arXiv preprint arXiv:2302. 1552, 2023.
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Pieter Abbeel. Benchmarking deep reinforcement learning for continuous control. In International conference on machine learning. pp. 1329-1338. PMLR, 2016.
Ying Fan and Kangwook Lee. Optimizing ddpm sampling with shortcut fine-tuning. arXiv preprint arXiv:2301.1336. 2023.
Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Dpok: Reinforcement learning for fine-tuning text-to-image diffuuson models. arXiv preprint 2305. 2023.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.0166 016188 2022.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. arXiv preprint arXiv:2210. 10760, 2022.
Gabriel Goh, Nick Cammarata +, Chelsea Voss + Shan Carter, Michael Petrov, Ludwig Schubert, Alee Radford, and Chris Olah. Multimodal neurons in artificial neural networks. Distill, 2021.
https://distill.pbbbe puwpl02 //ultimodal-neurooo
Philippe Hatsen-Estruch, Ily Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. IDQL: Implicit geaarnnng as an actor-critic method with diffusion policies. arXiv preprint DxXiv:334410077. 2023.
Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández Kevin Sheppard, Tyler Reddy,
del Río, Mark Wiebe, Pearu Peterson, Pierre Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis F. Oliphant. Array programming with NumPy. Nature, 585(7825):357-3666 2020.
Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Mare van Zee. Flax: A neural network library and ecosystem for JAX, 2023. URL h/gp: thub. cwm woon/goooll//la...mmmmmooo
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abheel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, 2020.
Jonathan Ho. William Chan, Chitwan Saharia, Jay Whang. Ruigi Gao. Alexey Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet, and Tim Salimans. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02.300 2022.
11

---

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint NoXY::2006009685. 2021.
Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference Machine Learning, 2022.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learning, pp. 267-274, 2002.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. In
Neural Information Processing Systems, 2021.
W. Bradley Knox and Peter Stone. TAMER: Training an Agent Manually via Evaluative Reinforce- ment. In International Conference Development and Learning, 2008
Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv:2302. 12192,11222 2023.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. 2023.
Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum, Compositional visual generation with composable diffusion models. arXiv preprini arXiv:2206.01714 2022.
Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Sampbell-Gillingaan Geoffrey Irving, and Nat McAleese. Teach- ing language models to support answers with verified quotes. arXiv preprint arXiv:2203. 11147, 2022.
Shakir Mohamed, Mihaela Rosca, Michael Figurnov, and Andriy Mnih. Monte carlo gradient estimation in machine learning. The Journal of Machine Learning Research, 21()):5183-5244, 2020.
Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelcrating online reinforcement learning with offline datasets. arXiv preprint trXiv:2006.0935, 2020.
Reichhrro Nakano, Jacob Hilton. Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assisted -answerng with human feedback. arXiv preprint arXiv:2/12.09333 2021.
Khanh Nguyen, Hal Daumé III, and Jordan Boyd-Graber. Reinforcement learning for bandit neural machine translation with simulated human feedback. In Empirical Methods in Natural Language Processing, 2017.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, 2021.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155. 2022.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weight regression: Simple and scalable off off-po reinforcement learning. CoRR, abs/1910.00177, 2019. URL / wwtpiiiv..r///////////oooo /tw.////9999110..777
Jan Peters and Stefan Schaal. Reinforcement learning by weight-weighted regression for operational space control. In International Conference on Machine learning, 2007.
12

---

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00022, 2021.
Aditya Ramesh, Mikhail Pavlov, Scott Gray Gabriel Goh, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. arXiv preprint arXiv:2102. 2092, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. IEEE Conference on Computer Vision and Pattern Recognition, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani. Yael Pritch, Michael Rubinstein, and Kfir Aberman. Drcambooth: Fine tuning text-to-image diffusion models for subject-driven generation. arXiv preprint arXiv:2208. 12242, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv:2205. /487, 2022.
Arne Schneuing, Yuanqi Du, Arian Jamash Charles Harris, Ilia Igashov, Weitao Du, Tom Blundell, Pietro Lió, Carla Gomes, Michael Bronstein Max Welling, and Bruno Correia. Structure- based drug design with equivariant diffusion models. arXiv preprint arXiv:2210.02303, 2022.
Chrisoph Schuhmann. laion- aesthntisst
Laion aesthetics, Aug 2022.
URL https://Laiin.aiie ai/blog/
John Schulman, Sergey Levine, Picter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference on Machine Learning, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06377 2017.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209. 14792, 2022.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. International Conference on Machine Learning, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- rional Conference Learning Representations, 2021. URL http:://ppeneeiieennttttoooo id=StIgiarCHLP.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. In Neural Information Processing Systems, 2020.
Richard Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradi- ent methods for reinforcement learning with function approximation. In S. Solla, T. Leen, and K. Müller (eds.), Advances in Neural Information Processing Systems, volume 12. MIT Press, 1999. URL https: https:///e/ wwwpeeeiiinnnnnnnnnnneeeeeeeee newrips. httpr//s.cccpppp...iilee//aappae 45402228860000000000e5550----0.0
pdf.
Patrick Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, and Thomas Wolf. Diffusers: State-of-the--ar diffusion models. https:
bogsthub..com/uugginggaace /ifffffrffs.. 2022.
Zhendong Wang. Jonathan Hunt, and Mingyuan Zhou. Diffusson policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.06111 2022.
13

---

Ronald J Williams. Simple statistical cradientfolllowin algorithms for connectionist reinforcement learning. Reinforcement learning, pp. 5-32, 1992.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the tf-t---rt-tttt natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45, Online, Octoher 2020. Association for Computational Linguistics. URL httpt:::/wwwtt ac www. wrggannthloogy///eee 2020 e0020.eennpppmmmm emn.p-demos. 6.
demos.
Tian Xie, Xiang Fu, Octavian Eugen Ganea, Regina Barzilay, and Tommi Jaakkola. Crystal diffusion variational autoencoder for periodic material generation. In International Conference Learning Representations, 2021.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. arXiv preprint arXiv:2304.0058 2023.
Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. GeoDiff: A geometric diffusion model for molecular conformation generation. In International Conference on Learning Representations, 2021.
Xiaohui Zeng, Arash Vahdat, Francis Williams, Zan Gojeic, Or Litany, Sanja Fidler, and Karsten Kreis. Lion: Latent point diffusion models for 3d shape generation. arXiv preprint urXiv:2210.06975. 2022.
Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models, 2023.
Tianyi Zhang, Varsha Kishore Felix Wu, Kilian Q. Weinberger, and Yoav Artzi BERTScore: Evaluating text generation with BERT. In International Conference on Learning Representations, 2020.
Linqi Zhou, Yilun Du, and Jiajun Wu. 3d shape generation and completion through point-voxel diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5826-5835, 2021.
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv: 08593, 2019.
14

---

APPENDIX A OVEROPTIMIZATION
Incompressibility
Counting Animals
SEXC SEoX ELUNS
Fise WOUSS
SEVEN BRIEDS
Phive
S T TuttiA
Tirree meaers
NAN
Figure (Reward model overoptimization) Examples of RL overoptimizing reward functions. (L. The diffusion model eventually loses all recognizable semantic content and produces noise when optimizing for incompressibility. (R) When optimized for prompts of the form "n animals", the diffusion model exploits the VLM with a typographic attack (Goh al., 2021), writing text that interpreted the specified number instead of generating the correct number of animals.
Section 6.1 highlights the optimization problem: given a reward function, how well can an RL algorithm maximize that reward? However, finetuning on reward function, especially a learned one, has been observed to lead reward overoptimization or exploitation (Gao al., 2022) in which the model achieves high reward while moving too far away from the pretraining distribution to be useful.
Our setting is no exception, and we provide two examples of reward exploitation in Figure When optimizing the incompressibilityy objective, the model eventually stops producing semantically meaningful content, degenerating into high-frequency noise. Similarly, we observed that LLaVA is susceptible to typographic attacks (Goh et al., 2021). When optimizing for alignment with respect to prompts the form "n animals", DDPO exploited deficiencies in the VLM by instead generating text loosely resembling the specified number: for example, "sixx ftulttas" above picture of eight turtles.
There is currently no veneral-purpose method for preventing overoptimization. One common strategy to add a KL-regularization term to the reward (Ouyang et al., 2022; Stiennon et al., 2020); we refer the reader to the concurrent work of Fan al. (2023) for study of Kl-rreuularization in the context of finetuning text-to-image diffusion models. However, Gao et al. (2022) suggest that existing solutions, including KL-rreularizztion. may be empirically equivalent to early stopping. As result, in this work, we manually identified the last checkpoint before a model began to deteriorate for each method and used that the reference for qualitative results. We highlight this problem as important area for future work.
APPENDIX B COMPARISON TO CLASSIFIER GUIDANCE
Classifier guidance (Dhariwal & Nichol, 2021) was originally introduced as a way to improve sample quality for conditional generation using the gradients from an image classifier. For differentiable reward function such as the LAION aesthetics predictor (Schuhmann, 2022), one could naturally imagine extension to classifier guidance that gradients from such predictor aesthetic score. The issue is that classifier guidance uses gradients with respect to the noisy images in the intermediate stages of the denoising process, which requires retraining the guidance network on
improve
Method
Aesthetic Score
Base model
5.95 0.03
Universal guidance
6.14 0.05
DDPOjs @ 20k reward queries
6.63 + 0.03
Table Comparison of DDPO with universal guidance using the LAION acsthetic predictor. We report the mean and one standard error over 50 samples for the prompt "wolf".
15

---

noisy images. Universal guidance (Bansal et al., 2023) sidesteps this issue by applying the guidance network to the fully denoised image predicted by the diffusion model at each step.
We compare DDPO with universal guidance in Table 1. We used the official implementation of universal guidance' with the recommended hyperparameters for style transfer, substituting the guidance network with the LAION aesthetics predictor. While universal guidance is able to produce a statistically significant improvement in aesthetic score, the change is small compared to DDPO. We only report results averaged over 50 samples for single prompt, since universal guidance is very slow; on an NVIDIA A100 GPU, it takes almost minutes to generate single image, whereas standard generation (e.g., from DDPO-fineture finetuned model) takes 4 seconds.
APPENDIX C COMPARISON TO DPOK
Here directly compare our implementation of DDPO the results reported in the DPOK paper (Fan et al., 2023), which was developed concureentyy with this work. The key similarities and differences between our experimental setups are summarized below:
For this experiment only, with low-rank adaptation (LoRA; Hu et al. (2021)) in order to match DPOK.
Stable Diffusion v1-5 the base model and train the UNet
Rather than matching the hyperparameters in DPOK, we use the same hyperparameters as in other experiments (Appendix D.5) except for the learning rate which we increase to 3e-4. We found that when using LoRA, a higher learning rate is necessary to get comparable performance full finetuning.
Like DPOK, we train on four prompts: "a green colored rabbit" (color), "four wolves in the park" (count), "a dog and a cat" (composition), and dog on the moon" (location). Unlike DPOK, we train single model for all four prompts.
Like DPOK, we train the model using Reward (Xu et al., 2023) the reward function. We evaluate the model using ImageReward and the LAION aesthetics predictor (Schuhmann, 2022).
Unlike DPOK, we do not employ KL regularization.
ImageReward
Aesthetic Quality
2.0
8.2
LAION Aesthetic Score
ImogeRewar Score
1.5
8.0
5.8
0.5
6.6
6.4
0.0
5.2
10k
16k
Reward Queries
Reward Queries
Color
Compoettion
DDPOns
Count
Location
DPOK
Figure 8 Comparison of DDPOjs with DPOK. We take the DPOK numbers directly from the paper, which only reports scores at one point in training (after 20k reward queries). Like in DPOK, scores are averaged over 50 samples for each prompt.
Guided- Quided-Diffusion
16

---

Color
Count
Composition
Location
Pretrained
Flnetuned
Figure Qualtitative examples of the results of ImageReward training on the DPOK prompts: "a green colored rabbit" (color), "four wolves in the park" (count), "a dog and cat" (composition), and "a dog on the moon" (location). The finetuned images are generated from a model trained for 20k reward queries.
The results are presented in Figure Our implementation DDPO1s outperforms DPOK accross the board, without using KL regularization. Figure also doubles as a quantitative study of overop- timization (Appendix A), since the model is trained with one reward function (ImageReward) and evaluated with another (LAION aesthetic score). We find that significant overoptimization does begin happen within 25k reward queries for one of the prompts (count: "four wolves in the park"), which is reflected by drop in LAION aesthetic score. However, the overoptimization is not severe or unreasonably fast. We provide qualitative samples Figure 9 showing that the model is able to produce high-qualiz, images at 20k reward queries.
APPENDIX IMPLEMENTATION DETAILS
For all experiments, use Stable Diffusion v1.4 (Rombach al., 2022) the base model and finetune only the UNet weights while keeping the text encoder and autoencoder weights frozen.
D.1 DDPO IMPLEMENTATION
We collect 256 samples per training iteration. For DDPOsE: we accumulate gradients across all 256 samples and perform one gradient update. For DDPO,s, we split the samples into 4 minibatches and perform 4 gradient updates. Gradients are always accumulated across all denoising timesteps for a single sample. For DDPO1s. we use the same clipped surrogate objective as in proximal policy optimization (Schulman al., 2017), but find that need to standard RL tasks. We use clip range of le-4 for all experiments.
very small clip range compared
D.2 RWR IMPLEMENTATION
We compute the weights for a training iteration using the entire dataset samples collected for that training iteration. For WRWR. the weights are computed using the softmax function. For Wsparse we use percentile-basee threshold, meaning dynamically selected such that the bottom p% of given pool of samples are discarded and the rest are used for training.
D.3 REWARD NORMALIZATION
In practice, rewards are rarely used as-is, but instead are normalized have zero mean and unit variance. Furthermore, this normalization can depend on the current state; in the policy gradient context, this is analogous to a value function baseline (Sutton et al., 1999), and in the RWR context, this is analogous to advantage-weighhee normalize the rewards on a per-context basis. For DDPO, this implemented as normalization by running and standard deviation that is tracked for each prompt independently. For RWR, this is implemented by computing the softmax over rewards for each prompt independently. For RWR this is implemented by computing the percentile-based threshold C for each prompt independently.
regression (Peng al., 2019). In our experiments, we
17

---

D.4 RESOURCE DETAILS
RWR experiments were conducted on a v3-128 TPU pod, and took approximately 4 hours to reach 50k samples. DDPO experiments were conducted on v4-64 TPU pod, and took approximately hours to reach 50k samples, For the VLM-based reward function, LLaVA inference was conducted on DGX machine with 80Gb A100 GPUs.
D.5 FULL HYPERPARAMETERS
DD
DDPO
RWR
RWR
Sparse
Denoising steps (?")
50
50
50
50
Diffusion
Guidance weight (w)
5.0
5.0
5.0
5.0
Optimizer
A
Adam W
AdamW
Adam
Learning rate
lc-5
le-5
le-5
lc-5
Weight decay
le-4
le-4
le-4
le-4
Optimization
81
0.9
0.9
0.9
0.9
B2
0.999
0.999
0.999
0.999
le-8
1e-8
1e-8
le-8
Gradient clip norm
1.0
1.0
1.0
1.0
Inverse temperature (3)
0.2
Percentile
0.9
RWR
Batch size
128
128
Gradient updates per iteration
400
400
Samples per iteration
10k
10k
Batch size
64
256
Samples per iteration
256
256
DDPO
Gradient updates per iteration
Clip range
le-4
D.6 LIST OF 45 COMMON ANIMALS
This list was used for experiments with the aesthetic quality reward function and the VLM-based reward function.
cat deer
dog
horse monkey goat lion tiger
rabbit
bird raccoon fox wolf squirrel chicken duck goose bee kangaroo
zebra bear
spider
sheep
lizard
beetle ant
butterfly
fish shark whale dolphin
mouse snake pig
turtle frog
turkey fly 1lama camel
bat gorilla hedgehog
APPENDIX E ADDITIONAL DESIGN DECISIONS
E.1
CEG TRAINING
Recent text-to-image diffusion models rely critically classifier-free guidance (CFG) (Ho & Salimans, 2021) to produce perceptually high-quality results. CFG involves jointly training the diffusion model on conditional and unconditional objectives by randomly masking out the context during training. The conditional and unconditional predictions are then mixed at sampling time using guidance weight w:
Ex c) weo 1, (1 (x)-w)x()
where co is the e-prediction parameterization of the diffusion model (Ho et al., 2020) and čσ is the guided e-prediction that is used to compute the next denoised sample.
For reinforcement learning, it does not make sense to train the unconditional objective since the reward may depend on the context. However, we found that when only training on the conditional objective, performance rapidly deteriorated after the first round of finetuning. We hypothesized that
18

---

this is due to the guidance weight becoming miscalibrated each time the model is updated, leading to degraded samples, which in turn impair the next round finetuning, and so on. Our solution was to choose a fixed guidance weight and use the guided e-prediction during training well as sampling. We call this procedure CFG training. Figure 10 shows the effect CFG training on R has effect after single round of finetuning, but becomes essential for subsequent rounds.
JPEG Compressibility
Negative Filesize (kb)
-50
-100
-150
-200
10k
20k
30k
10k
Reward Queries
with CFG training
without CFG training
Figure 10 (CFG training) We run the RWR algorithm while optimizing only the conditional 6- prediction (without CFG training), and while optimizing the guided e-prediction (with CFG training). Each point denotes a diffusion model update. We find that CFG training is essential for methods that do more than one round of interleaved sampling and training.
E.2 INTERLEAVING
There are two main differences between DDPO and RWR, as compared in Section 6. the objective (DDPO uses the policy gradient) and the data distribution (DDPO is significantly more on-policy, collecting 256 samples per iteration as opposed to 10,000 for RWR). This choice is motivated by standard RL practice, in which policy gradient methods specifically require on-policy data (Sutton et al., 1999), whereas RWR is designed to work in on off-policy data (Nair et al., 2020) and is known underperform other algorithms more online settings (Duan al., 2016).
However, we can isolate the effect of the data distribution by varying how interleaved the sampling and training are in RWR. At one extreme is single-round algorithm (Lee al., 2023), in which N samples are collected from the pretrained model and used for finetuning. It is also possible to run N samples collected from the most up-to-date model. In Figure 11, we evaluate this hyperparameter and find that increased interleaving does help up to point, after which it causes performance degradation. However, RWR is still unable to match the asymptotic performance of DDPO at any level of interleaving.
k rounds of finetuning each on
APPENDIX F QUANTITATIVE RESULTS FOR GENERALIZATION
In Section 6.3, we presented qualitative evidence of both the aesthetic quality model and the image- prompt alignment model generalizing to prompts that were unseen during finetuning. In Figure 12, we provide an additional quantitative amalysis of generalization with the aesthetic quality model, where measure the average reward throughout training for several prompt distributions. In accordance with the qualitative evidence, we see that the model generalizes very well to unseen animals, and everyday objects to lesser degree.
19

---

Aesthetic Quality
JPEG Compressiillity
Samupees per Iteration (RWR)
5.2
512
Negative Flisste: (80))
-50
1,024
5.0
LAION Amdlheli
4,006
4.8
-100
8,
16.384
4.8
-150
4.4
42
Somples per Iteeation (DDPO,s)
Rewaed Queries
Roward Queeries
256
Figure 11 (RWR interleaving ablation) Ablation over the number of samples collected per iteration for RWR. The number of gradient updates per iteration remains the same throughout. We find that more frequent interleaving is beneficial up point, after which it causes performance degradation. However, RWR is still unable to match the asymptotic performance of DDPO at any level of interleaving.
Aesthetic Quality Quantitative Generalization
Finetuning Prompts (Aninals Ouly)
5.2
Unseen Animals
Ordinary Objects
5.0
Reward
4.8
4.6
4.4
4.2
10k
20k
30k
10k
50k
Reward Queries
Figure 12 (Quantitative generalization) Reward curves demonstrating the generalization of the aesthetic quality objective prompts not seen during finetuning. The finetuning prompts are list of 45 common animals, "unseen animals" a list of 38 additional animals, and "ordinary objects" is list of 50 objects (e.g. toaster, chair, coffee cup, etc.).
20

---

Prefrained
Aesthetic Quality
Compressibility
Incompressibility
Figure 13 (RWR samples)
a hedgehog riding a bike
dog riding bike
lizard riding bike
shark washing dishes
a frog washing dishes
a monkey washing dishes
Figure 14 (More image- mnage-prooptte alignment samples)
APPENDIX G MORE SAMPLES
Figure 13 shows qualitative samples from the baseline RWR method. Figure 14 shows more samples seen prompts from DDPO finetuning with the image-prompt alignment reward function. Figure 15 shows more examples generalization to unseen animals and everyday objects with the aesthetic quality reward function. Figure 16 shows more examples of generalization to unseen subjects and activities with the image-promph alignment reward function.
21

---

Pretrained (New Animals)
Aesthetic Quality (New Animals)
Pretrained (Non-Animals)
Aesthetic Quality (Non-Animals)
Figure 15 (Aesthetic quality generalization)
22

---

a capybara washing dishes-
snail playing chess
dog doing laundry
a giruffe playing basketball
parrot driving
duck trkinng exam
a robot fishing in a lake
a horse typing on a keyboard +
rabbit sewing clothes
riding bike
car eating sandwich
an apple playing soccer
Figure 16 (Image-prompt alignment generalization)
23


Reviews:
[{"review_id": "7QT1jjqoGe", "reviewer": "Reviewer_qqX7", "summary": "This paper presents a method for fine-tuning large-scale diffusion-based text generation models using reinforcement learning. It views the reverse process of diffusion models as a reinforcement learning process with time steps T (T is the denoising steps), where the output of the diffusion model at each step serves both as an action and the observation for the next moment. The authors describe how posing denoising as a multi-step decision-making problem enables a class of policy gradient algorithms, referred to as denoising diffusion policy optimization. And the authors also employ an open-source image quality evaluator as the reward function and also devise an automated prompt-image alignment method using feedback from a vision-language model.", "questions": "My main points have been outlined in the Strengths and Weaknesses sections. If the authors can provide satisfactory responses and additional experiments addressing these points, I would consider revising my review.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 1, "strengths": "This paper is well-written, with clear logic and beautifully crafted figures, making it easy to follow the authors' line of reasoning. Additionally, the paper is well-structured, presenting an easy-to-follow approach to fine-tuning diffusion models using reinforcement learning for alignment. The narrative is straightforward, and the methods described are, in my opinion, sensible.", "weaknesses": "1) In terms of image generation quality, the paper lacks a quantitative and qualitative comparison with recent works. It fails to provide experimental support for its effectiveness. Specifically, in the absence of comparisons in image quality with all methods related to \"Optimizing diffusion models using policy gradients,\" it is challenging to discern the improvements this paper offers over baseline approaches. This makes it difficult to evaluate the paper's contribution to the community.\n\n2) Regarding originality and community contribution, compared to DPOK and \"Optimizing ddpm sampling with shortcut fine-tuning,\" I did not observe significant differences between this paper and DPOK from the introduction to the Method section. The structural similarity in writing is quite evident. The most noticeable difference is the introduction of importance sampling in method 4.3. However, this alone does not sufficiently support the paper’s claims of innovation, especially without experimental evidence backing its effectiveness (if such evidence exists, please inform me during the Rebuttal phase). The second noticeable difference is the automated generation of rewards using BLIP, which is already a standard engineering practice and has been claimed by RLAIF. I do not believe the methodological contributions and community impact of this paper meet the acceptance standards of the ICLR conference.\n\n3) The experimental evaluation criteria are unreasonable. Most pretrained reward evaluation models are not designed for robust data manifold, especially for treating it as a \"fine-tuning teacher.\" Consequently, out-of-domain evaluation is necessarily needed. For example, the authors use Aesthetic Quality as a reward function to fine-tune the Diffusion model and employ the same Aesthetic Quality for scoring during evaluation. This approach does not allow for assessing whether an improvement in the Aesthetic Quality Score correlates with an enhancement in image generation quality. A more reasonable evaluation, as exemplified by DPOK, would involve fine-tuning with one Reward model and then evaluating it against both this model and a new, out-of-domain Reward model. DPOK's evaluating approach might provide a more substantial basis for assessment.\n\n4) Lack of code. The lack of open-source code, compounded by the absence of comparisons with recent works, makes it difficult to assess the practical feasibility of the approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BEujamuBbq", "reviewer": "Reviewer_sWSo", "summary": "This paper introduces Denoising Diffusion Policy Optimization (DDPO), where they fine-tune diffusion models by reinforcement learning (RL). In specific, given the scalar reward function that takes an image and conditions (e.g., text prompts), DDPO considers the reverse generative process as Markovian Decision Process (MDP), where the reward is only given at the zeroth timestep. The authors implement DDPO via policy gradient (i.e., REINFORCE) or together with importance sampling (i.e., PPO-like approach). Through experiments they show that DDPO can be used in improving aesthetic quality, image-text alignment, and compression (or incompression) of an image, and verify its effectiveness in comparison to reward augmented fine-tuning.", "questions": "- Regarding the capability of DDPO, is it possible to control the image generation to have predicted reward? For example, Classifier-Free guidance provides control over sample fidelity and diversity. It seems like DDPO improves prompt fidelity at the expense of sample diversity, so I wonder if there is a way to control the amount of reward during generation. \n- It seems like the reward function for different tasks have different ranges, and thus DDPO uses running mean and variance for normalization. I guess the difficulty of solving MDP varies across different prompts. Could the author elaborate on the optimization analysis with respect to the difficulty of prompts? \n- In implementation, the author chooses T=50 throughout experiments. What if the timesteps become smaller or larger? Definitely the denser timestep sampling would require more cost, but I believe it can reduce the variance of gradient. Could the author elaborate on this point? \n- Do the authors try different reward models for image-text alignment? How robust is DDPO algorithm with respect to different reward functions?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- This paper is clearly written and easy to follow.\n- The problem of solving diffusion generative processes as solving MDP is clearly stated, and the proposed method generalizes prior works (i.e., reward-weighted regression (RWR)) for the multi-step MDP case.\n- The experiments clearly validate the efficiency of DDPO over prior diffusion model tuning with reward functions as well as detailed algorithmic choices are provided.", "weaknesses": "- After RL fine-tuning, the generated images seem to be saturated. For example, the fine-tuned models generate images with high aesthetic scores, but they seem to generate images with similar backgrounds of sunset. For prompt alignment experiments, the models generate cartoon-like images. \n- I think one of the main contributions of the paper is on utilizing VLMs for optimizing text-to-image diffusion models. In this context, the discussion on the choice of reward function should be discussed more in detail. For example, I think different instruction prompts for the LLaVA model would make different results, but the paper lacks details in the choice of reward function.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wlw0IfLk6E", "reviewer": "Reviewer_EgHv", "summary": "The paper introduces a method that poses the diffusion process as an MDP and applies reinforcement learning to text-to-image diffusion models. It uses different reward functions, such as image compressibility, prompt alignment and image quality. The authors formulate three different approaches to fine-tune image diffusion models, two involving multi-step reinforcement learning and one that re-weights the original inverse diffusion objective. The authors evaluate their method on relevant datasets and discuss relevant shortcomings.", "questions": "- I am wondering why the lacking quantitative experiments have not been conducted (or why they cannot be conducted)\n- Similarly, I am wondering how the authors think about the early stopping mechanism", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- the paper proposes a novel work (other work mentioned in related works section is concurrent work as it will only be published at Neurips in December)\n- the approach is simple yet effective\n- a variety of reward functions are explored and all yield visually pleasing results", "weaknesses": "- The proposed method does not consider the problem of overoptimisation, instead the authors argue that early stopping from visual inspection is sufficient. This makes the method applicable to problems where visual inspection is possible (which is likely the case for many image tasks). However, it renders the method inapplicable to problems where the needed visual inspection is not possible. (E.g. one might not be able to apply this method to a medical imaging task where visual inspection by a human supervisor does not allow to determine when to stop the optimisation process).   \n- Further, the reliance on visual inspection negatively impacts scalability of the method.  \n- The evaluation is missing a simple classifier guidance baseline, where samples are generated from the diffusion model with classifier guidance according to the given reward function. (Classifier guidance is mentioned in the related work but not evaluated as a baseline)  \n - The hypothesis that cartoon-like samples are more likely after fine-tuning because no photorealistic images exist for the given prompt category could be verified by fine-tuning on categories for which photo-realistic images do exist .\n- the generalization study in 6.3. is purely qualitative, a quantitative analysis would be more adequate\n- similarly, overoptimization could be quantitatively analysed", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tXmkJsrWRj", "reviewer": "Reviewer_rYFt", "summary": "In this paper, the authors propose a policy gradient algorithm to finetune text-to-image diffusion models for downstream tasks using only a reward function. Specifically, they reframe the denoising process as a multi-step MDP and introduce two variants of denosing diffusion policy optimization. In addition, they validate the proposed method on different downstream tasks, such as aesthetics and prompt-image alignment. In particular, they propose a automated VLM-based reward function for prompt-image alignment.", "questions": "- In the related work, ‘Reward-Weighted Regression’ should be used before using its abbreviation (RWR). If not, it maybe a misleading phrase for the readers, especially ones without context.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+ The proposed method is effective and validated with various experiments. Besides, this paper is simple and easy to implement. \n+ The proposed automated prompt alignment is somewhat practical and provide an alternative for prompt-image alignment.", "weaknesses": "- The contribution of the proposed method is minor, though it is effective on different downstream tasks. Compared with DPOK, the modification of policy optimization is somewhat incremental. In the related work, the authors claims that training on many prompts is one of two key advantages over DPOK. But, DPOK also can train with multiple prompts and provides quantitative results for that experiment. It is better to conduct this experiment with the same setting to support this point. Besides, the authors only show that the proposed method outperforms simple RWR methods. It is essential to compare with previous works. For example, show different methods’ ability to handle prompts involving color, composition, counting and location. \n-  As shown in Appendix B, there is a subtle bug in the implementation. This bug maybe affect the quantitative comparisons and lower the support for the effectiveness of the proposed method. It is one of my major concerns about this paper. \n- In the experiment related to aesthetic quality, the finetuned model produce images with fixed style, as shown in Figure 3 and Figure 6. Is it only attributed to the bug in the implementation? Based on this phenomenon, I raise a concern: does the proposed method compromise the diversity of text-to-image diffusion models?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)