A Simple and Effective Reinforcement Learning Method for Text-to-Image Diffusion Fine-tuning
Authors: Shashank Gupta
Chaitanya Ahuja
Tsung-Yu Lin
Sreya Dutta Roy
Harrie Oosterhuis
Maarten de Rijke
Satya Narayan Shukla
Venue: Accepted by TMLR ()

Abstract: Reinforcement learning (RL)-based fine-tuning has emerged as a powerful approach for aligning diffusion models with black-box objectives. Proximal policy optimization ( PPO) is a popular choice of method for policy optimization. While effective in terms of performance and sample complexity, PPO is highly sensitive to hyper-parameters and involves substantial computational overhead. REINFORCE, on the other hand, mitigates some implementation complexities such as high memory overhead and sensitive hyper-parameter tuning, but has suboptimal performance due to high variance and crucially sample inefficiency, which is the primary notion of efficiency we study in this work. While the variance of the REINFORCE can be reduced by sampling multiple actions per input prompt and using a baseline correction term, it still suffers from sample inefficiency. To address these challenges, we systematically analyze the sample efficiency-effectiveness trade-off between REINFORCE and PPO, and propose leave-one-out PPO ( LOOP), a novel RL for diffusion fine-tuning method. LOOP combines variance reduction techniques from REINFORCE, such as sampling multiple actions per input prompt and a baseline correction term, with the robustness and sample efficiency of PPO via clipping and importance sampling. Our results demonstrate that LOOP effectively improves diffusion models on various black-box objectives, and achieves a better balance between sample efficiency and final performance.

Published in Transactions on Machine Learning Research (03/2026)
A Simple and Effective Reinforcement Learning Method for Text-to-lmage Diffusion Fine-tuning
Shashank Gupta*1
University of Amsterdam, The Netherlands
Chaitanya Ahuja Meta
chahuja@@eeta.co com
Tsung- Yu Lin Meta
tsungyulin@metta.com
Sreya Dutta Roy Meta
ereyoduttarmoy@@ett..com
Harrie Oosterhuis
harrie.costerfouis @ra.
Radboud University, Nijmegen, The Netherlands
Maarten de Rijke
m.derijke@uoo.. ml
University of Amsterdam, The Netherlands
Satya Narayan Shukla
sns. ins.ikgppgmmail.com com
Meta
Reviewed on OpenReview: https: // openreview.
/ete::///peerevvee net/ forum? id= t8WJhkn455
Abstract
Reinforcement learning (RL)-based fine-tuning has emerged a powerful approach for aligning diffusion models with black-hox objectives. Proximal policy optimization (PPO) is popular choice of method for policy optimization. While effective in terms of performance and sample complexity, PPO is highly sensitive to computational overhead. REINFORCE, on the other hand, mitigates some implementation complexitics such as high memory overhead and sensitive hyper-parameter tuning, but has suboptimal performance due to high variance and crucially sample inefficiency, which is the primary notion of efficiency we study in this work. While the variance of the REINFORCE be reduced by sampling multiple actions per input prompt and using baseline correction term, it still suffers from sample inefficiency. To address these challenges, we systematically analyse the sample allicienoy-efecctivenee trade-off between REINFORCE and PPO, and propose leave-one-on PPO (LOOP), a novel RL for diffusion fine-tuning method. LOOP combines variance reduction techniques from REINFORCE, such as sampling multiple actions per inpul prompt and a baseline correction term, with the robust ness and sample efficiency of PPO via clipping and importance sampling. Our results demonstrate that LOOP effectively improves diffusion models various black-box objectives, and achieves better balance between sample efficiency and final performance.
and involves substantial
"Corresponding author. Work doue during internship at Meta AL. &Correspondng author.

---

Published in Transactions on Machine Learning Research (03/2026)
1 Introduction
Diffusion models have emerged as powerful tool for generative modeling (Ho et al. 2020 Sohl-Dickstein et al. 2015), with strong capacity to model complex data distributions from various modalities, like images (Rom- bach et al. 2022). text (Austin et al., 2021). natural molecules (Xu et al. 2023 and videos Blattmann
ct al. 2023
Diffusion models are typically pre-trained on a large-scale datasel, such that they can subsequently generate samples from the same data distribution. The training objective typically involves maximizing the data distribution likelihood. This pro-training stage helps generate high-quality samples from the model. However, some applications might require optimizing custom reward function, for example, optimizing for generating hestalli pleasing images (Xu et al. 2024), semantic aligmment of image-text pairs based on human feedback (Schuhmann et al. 2022), or generating molecules with specific properties (Wang et al. 2024).
To optimize for such black- box objectives, RL-based fine- tunntun has been successfully applied to diffusion models (Black et al., 2023; Fan et al.} 2024 Gu et al.] 2024 Li et al. 2024 Wallace et al., 2024). For RL-based fine-tuning, the reverse diffusion process is treated as a Markov decision process (MDP), wherein prompts treated as part of the imput state, the generated image at each time-step is mapped to action, which receives reward from fixed reward model (environment in standard MDP), and finally the diffusion model is treated as a policy, which we optimize to maximize rewards. For optimization, typically PPO is applied (Black el al.] 2023 Fan et al. 2024).. In applications where getting a reward model is infeasible or undesirable, "RL-freee tuninuning (typically offlnn) can also be applied (Wallace et al. 2024). For this work, we only focus on diffusion model fine-tuning using "online" RL methods, specifically policy-gradice style methods, like PPO (Schulman et al. 2017).
An advantage of PPO is that it removes the incentive for the new policy to deviate too much from the previous reference policy, via importance sampling and clipping (Schulman et al. 2017), which, we discuss below, leads to superior sample efficiency over simpler policy-gradient methods such as REINFORCE (Williams) 1992 However, PPO comes with significant. implementation overhead: in practice, RL. fine-tuning for diffusion models via PPO requires concurrently loading three models in memory: (i) The reference policy: The base policy, usually initialized with the pre-trained diffusson model. (ii) The current policy: The RL fine-tuned policy, also initialized with the pre-trained diffusion model. (iii) The reward model: Typically a large vision-language model trained via supervised fine-tuning (Lee et al. 2023), which assigns a scalar reward to the final generated image. In addition, PPO is known to be sensitive to hyper- pyper-parameters (Engstrom et al., 2019 Huang et al.) 2024 Zheng et al. 2023). We note these costs as practical motivation for seeking simpler alternatives, though the primary axis of comparison in this work is sample efficiency, not computational cost.
Simpler approaches, such as REINFORCE (Willamms 1992p, avoid such additional implementation complexities by requiring neither a separate reference policy in memory nor sensitive hyperparameter tuning. However, this implementation simplicity comes at a cost: in practice, REINFORCE less sample efficient than PPO, and suffers from high variance and instability (Gupta 2023 Gupta et al. 2024b). A variant of REINFORCE, reinforce leave-one-out (RIO0) (Ahmadian et al. 2024 Kool et al.) 2019), samples multiple sequences per input prompt and applies a baseline correction term to reduce variance; however, it remains sample inefficient because it does not permit trajectory reuse across policy updates, unlike importance-sampling based methods.
This raises a fundamental question about the efficienyy-effettiveeess trade-off in RI-based diffusion fine- tuning. In this work, we systematically explore the tradc-off between sample efficiency, defined as requiring fewer training prompts to achieve good performance, and effectiveness, measured by stable training and final reward. We compare simple REINFORCE approach with the standard PPO framework, demonstrating that while REINFORCE greatly reduces implementation complexity, it comes at the cost of reduced performance and sample complexity.
Motivated by this finding, we propose a novel RL method for diffusion fine-tuning, LOOP, which combines the best of both worlds. To reduce the variance during policy optimization, LOOP uses multiple actions (diffusion trajectories) and a REINFORCE-style bascline correction term per input prompt. To maintain the stability, robustness, and sample complexity of PPO, LOOP uses clipping and importance sampling.
2

---

Published in Transactions on Machine Learning Research (03/2026)
We clarify an important distinction regarding the notion of efficiency in this work. When discussing the efficiency-eliectiveees trade-off, we primarily refer to sample efficiency, defined as an hm'' ability to achieve better performance with the same number of input prompts during training. LOOP exhibits superior sample efficiency compared to PPO. We emphasize this notion because directly translates into better performance for a fixed training dataset size, which is often the dominant constraint in pract when optimizing diffusion models with computationally expensive reward models. For a fixed number of training prompts, LOOP attains higher reward values by sampling multiple trajectories per prompt and employing a leave-one-out baseline correction term.
However, we note that while LOOP demonstrates superior sample efficiency by requiring fewer training prompts to achieve a given performance level, it requires K diffusson sampling passes per prompt, leading to an extra O(K) computational overhead relative to PPO. Future work could explore adaptive sampling strategies, asynchronous generation pipelines (Han et al., 2025), or distributed trajectory sampling (Bartoldson) ct al., 2025) to mitigate this computational cost while preserving sample efficiency gains. We report the total GPU runtime and memory usage in Appendix We leave the study and improvement of the computational efficiency of LOOP as part of future work.
Our approach is conceptually similar to the recently proposed GRPO method for RL fine-tuning of LLMs (Shao et al. 2024). The key technical differences are: (i) LOOP does not apply standard-deviation normalization in the advantage calculation. Recent work on LLM fine-tunin tinettunigg suggests that removing this normalization term can improve performance (Liu et al. 2025). D. (ii) Following this recent work, LOOP omits the KL penalty term. Prior studies indicate that explicit KL regularization has minimal practical effect on performance (Black) et al. 2023 and recent theoretical work shows that on-policy RL methods implicitly maintain KT. proximity to the base policy even without explicit regularization (Shenfeld et al.) 2025). (iii) In the diffusion setting, the reverse process has a fixed sequence length across all generations, making sequence-length normalization unnecessary. We provide an empirical comparison on GRPPO vs. LOOP in Appendix B
For the primary evaluation benchmark, we choose the text-to-image compositionality benchmark T2I- CompBench (Huang et al., 2023). Text-to-image models often fail to satisfy an essential reasoning ability of attribute binding, i.e., the generated image often fails to bind certain aftributes specified in the instruction prompt (Fu & Cheng 2024 Huang et al.] 2023 Ramesh et al., 2022). As illustrated in Figure 1 LOOP outperforms previous dillusion methods on altribule binding. As altribute binding is a key skill necessary for real-world applications, we choose the 121-CompBench benchmark alongside two other common tasks:
aesthetic image generation and image-text semantic alignment (Black et al. 2023
To summarize, our main contributions are as follows:
PPO vs. REINFORCE -fficiency-cfoocttieenss trade-off. We systematically study how design elements like clipping, reference policy, value function in PPO compare to a simple REINFORCE method, highlighting the seafeency-fffectvenees trade-off in diffusion fine-tuning. To the best of our knowledge, we are the first ones to present such a systematic study, highlighting the trade-offs in diffusion finc-tuning.
Introducing LOOP. We propose LOOP, a novel RL for diffusson fine-tuning method combining the best of REINFORCE and PPO. LOOP uses multiple diffusion trajectories and REINFORCE bascline correction term for variance reduction, as well as clipping and importance sampling from PPO for robustness and sample efficiency.
Empirical validation. To validate claims empirically, we conduct experiments the T2I- CompBench benchmark image compositionality benchmark. The benchmark evaluates the attribute binding capabilities of the text-to-image generative models and shows that LOOP succeeds where previous text-Lo-image generative models often fail. We also evaluate LOOP on two common objectives from the literature on RL for diffusion: image aesthetic and text- iestt-mmaeeaao semantic alignment Black ct al. 2023
The remainder of the paper is organized as follows. In the next section, we provide the necessary background and discuss related work. Section 3 revisits the cfficiency-efecttvvnee trade-off between REINFORCE, and
3

---

Published in Transactions on Machine Learning Research (03/2026)
SD v2
DDPO
LOOP
"White cat playing with black ball"
"Mechanical owl with cobalt blue feathers on rusted bronze lamppost.
"A hexagnnal watermelon placed besides a knife"
"Black horse with glowing eyan patterns in maze of floating golden rings"
"Neon pink cactus growing cobalt blue rock"
Figure 1: LOOP improves attribute binding. Qualitative examples presented from images generated via Stable Diffusion (SD) 2.0 (first row), DDPO (Black el al. 2023) (second row), and LOOP In the first prompt, SD and DDPÓ both fail to bind the color black with the ball in the image, whereas LOOP binds the color black to the ball. In the second example, SI) and DDPO fail to generate rusted bronze color lamppost, whereas LOOP manages to do that. In the third image, SD and DDPO fail to bind the shape bezagon to the watermelon, whereas LOOP manages so. In the fourth example, SD and DDPO fail to generate the black horse with flowing cyan patterns, whereas LOOP generates the horse with the correct color attribute. Finally, in the last image, SD and DDPO fail to bind cobalt blue color to the rock, whereas LOOP binds that successfully.
(third row).
PPO. Section introduces our proposed method, Leave-One-Ol PPO (LOOP) for diffusion fine-tuning. Section 5 describes the experimental setup, Section 6 details our hyporparametors, and Scctions 7 and present our results and qualitative examples. Finally, Section concludes the paper.
2 Background and Related Work
2.1 Diffusion Models
We focus on denoising diffusion probabilistic models (DDPM) as the base model for text-to-image generative modeling (Ho et al. 2020: Sohl-Dickstein et al.) 2015). Briefly, given conditioning context variable (a text prompt in our case), and the data sample Xo, DDPM models p(xo c) via Markov chain of length T,, with the following dynamic::
T c) II (x Xt, c).
Po (xpe
c) = p(xr
(1)
t=1
Image generation in diffusion model is achieved via the following ancestral sampling scheme, which is a reverse diffusion process:
XT N(O,I), x, N (x))))(xx..,1)))0))) e,
[0, 1).
(2)
4

---

Published in Transactions on Machine Learning Research (03/2026)
where the distribution at time-step is assumed to be multivariate normal distribution with the predicted mean (x0 c. C, and a constant variance.
2.2 Proximal Policy Optimization (PPO) for RL
PPO was introduced for optimizing a policy with the objective of maximizing the overall reward in the RL setup. PPO removes the incentive for the current policy иı to diverge from the previous policy яĄ-1 outside the range [1 - e, 1 + el,, where is a hyper-parameter As long as the subsequent policies are closer to each other in the action space, the monotonic policy improvement bound guarantees a monotonic improvement in the policy's performance as the optimization progresses. This property justifies the clipping term in the mathematical formulation of the PPO objective function (Achiam et al., 2017 Queency et al. 2021 Schulman 2015). Formally, the PPO objective function is:
J(0) )(0)=E(min (re(0) A., (lpprr(()),
33)
where r0( (ule) is the importance sampling ratio between the current policy Te(a c) and the previous [t-1(
reference policy TT-1(a c), A is the advantage function (utton Barto 2018), and the clip operator restricts the importance sampling ratio in the range [1 ee-e, 1ee.
2.3 RL for Text-to-lmage Diffusion Models
The diffusion process can be viewed as an MDP (S,A, P, P,R, where S the state space, is the action space, P is the state transition kernel, R is the reward function, and po is the distribution of initial state So- In the context of text-to-image diffusson models, the MDP is defined as:
St = (c, 1,,t,xx xt), (at st) = (x Xt, c), P(se+1 St, at) 8(c, (() c) if 0,
at = Xt-11
(4)
Po(so) = (o(c),,r,N(() D)), R(st, at)
otherwise.
The input state St is defined in terms of the context (prompt features), and the sampled image at the given time-step t: xt. The policy me is the diffusion model itself. The state transition kernel is dirac delta function 5 with the current sampled action x; as the input. The reward is assigned only at the last step in the reverse diffusion process, when the final image is generated. The initial state pn corresponds to the last. state in the forward diffusion process: XT.
2.4 PPO for Diffusion Fine-tuning
The objective function of RL fine-tuning for a diffusion policy can be defined as follows:
T
(T( (rmp(r)
Σ
R(s at)
Er-p(r me) [r(xo, c)],
(5)
where the trajectory (xT, ...,, xo) refers to the reverse diffusion process (Eq. 1]. and the total
reward of the trajectory is the reward of the final generated image Xo (Eq, 4.. We ignore the KL-regularized
version of the equation, which is commonly applied in the RLHF for LLM litcrature (Rafailov et al. 2023
Zeng et al. 2024 Zhong et al.] 2024), and proposed by Fan et al. (2024) in the context of RL for diffusion models. As shown by Black et al. (2023), adding the KT-regularization term makes no empirical difference terms of the final performance. The PPO objective is given as:
no Tx( c) 1 c-c, 1+ r r(xo, (t))) Tola (x c)
co).
ppP (T)
R
clip Σ
1=0
where the clipping operation removes the incentive for the new policy ne differ from the previous round policy wold (Black et. al. 2023 Schulman al. 2017)
5

---

Published in Transactions on Machine Learning Research (03/2026)
2.5 Relationship to Offline Preference Based Methods
Although LOOP operates in online reinforcement learning selting, similar to PPO, recent work has explored adapting direct preference optimization (DPO) (Rafailov et al. 2023 and related offiine methods to diffusion models.
DPO-based diffusion alignment: Several methods have exlended the DPO framework to diffusion models. Diffusion DPO (Rafailov et al., 2023 Wallace et al. 2024] applies the DPÔ objective to diffusion models by treating denoising a sequential decision making process and by training on preference pairs without explicit reward queries. Diffusion RPO (Gu et al. 2024 introduces pairwise preference optimization at each denoising timesteps, as opposed to pairwise preference at the trajectory level. Other work has explored noise conditioned preference learning (Gambashidze et al. 2024 as well as applications to diffusion based policy learning (Kagg et al. 2023).
We do not include direct empirical comparisons with DPO style methods because these approaches learn from pre-collected preference datasets rather than from online reward queries and thus represent complementary paradigm for aligning generative models with human preferences. DPO-based methods rely on pre-collected preference datasets and are therefore constrained by the coverage of their training data, whereas online methods such as LOOP query rewards during training and can explore beyond the initial policy distribution. In settings where large and high qualily preference datasets are available, DPO style methods may offer competitive performance with fewer online reward queries. Conversely, when online exploration feasible and preference data is limited, online methods may be more appropriate. Future work could investigate systematic comparisons between online and offline methods under controlled conditions, well as hybrid approaches that combine both paradigms.
3 REINFORCE vs. PPO: A Sample Efficiency-fffectiveees Trade-Off
In this section, explore the sample efficiency-effectiveesss trade-off between two prominent reinforcement learning methods for diffusion fine-tuning: REINFORCE and PPO. Understanding this trade-off is crucial for selecting the appropriate algorithm given constraints on training data or reward-query budgets.
In the context of text-to-image diffusion models, we aim to optimize the policy æ to maximize the expected reward R(xo:T; c) = r(xo, c). Our objective function is defined as:
Jo(T) Ee-p (o.- [r(xo,c).
REINFORCE for gradient calculation. For optimizing this objective, the REINFORCE policy gradient
(also known score function (SF) (Williams 1992) provides the following gradient estimate:
V V SF (T)
Il po (xx-1 c) (xo: c) e))
Exxor
Vo log
(7)
Σ Ve log pe (xx-1 Xx, c) (xo, e)
= Exxo
where the second step follows from the reverse dilfusion policy decomposition (Eq.
In practice, a batch of trajectorics is sampled from the reverse diffusion distribution, i.c., X ~ (x(o)))), and a Monte-Carle Carlo estimate of the REINFORCE policy gradient (Eq. is calculated for the model update.
REINFORCE with baseline correction. To reduce variance of the REINFORCE estimator, common trick to subtract constant baseline correction term from the reward function GGreensmith et al. 2004
Gupta et al. 2024b Mohamed et al. 2020
T
SFB
TT)
E
v log xe. (r(xx) c) -b))
(8)
6

---

Published in Transactions on Machine Learning Research (03/2026)
REINFORCE Leave-one-out (RLOO). To further reduce the variance of the REINFORCE estimator, RLOO samples K diffusion trajectories per prompt ((xir) ~ #(. [)), for better Monte-Care estimate of
the expectation (Ahmadian et al. 2024 Kool et al. 2019 The RLOO estimator is:
V VoJø
BLOO
n
R
K
Vo log (x x c) (r by)
However, REINFORCE-based estimatons have a siguificant disadvantagge they do not allow sample reuse (i.e., reusing trajectories collected from previous policies) due to a distribution shift between policy gradient updates during training. Sampled trajectorics can only be used once, prohibiting mini-batch updates. This makes it somple inefficienl.
To allow for sample reuse, the importance sampling (IS) trick can be applied
Owen 2013
Schulman 2015
Ta(a: ct) Told (a: Ce)
1 (T) =
Re
(10)
where my is the cwrrent policy to be optimized, and Vold is the policy from the previous update round. With the IS trick, we can sample trajectories from the current policy in a batch, store it in a temporary buffer, and re-use them to apply mini-batch optimization (Schulman et al. 2017).
Motivation for PPO. With the IS trick, the samples from the old policy can be used estimate the policy gradient under the current policy me (Eq. 7 in statistically unbiased fashion (Owen 2013 i.c., in expectation the IS and REINFORCE gradients are equivalent (Eq. 10 Eq. 70. Thus, potentially, we can improve the sample efficiency of REINFORCE gradient estimation with 1S.
While unbiased, the IS estimator can exhibit high variance (Owen to unstable training dynamics. Additionally, significant divergence weew the current policy my and the previous policy oold can result in the updated diffusion policy performing worse than the previous one (Achiam et al. 2017 Schulman 2015). Next, will prove this formally. We note that this result has previously been established by (Achiam al. 2017) for the more general RL setting. In this work, we extend this finding to the context of diffusion model fine-tuning.
2013 This high variance may lead
A key component of the proof relies on the distribution of states under the current policy, i.e., d'((s. In the case of diffusion models, the state transition kernel P(s++1 St, at) is deterministic, because the next state consists of the action sampled from the previous state (Eq. 4), i.c., P(s++1 St, at) = While the state transition kernel is deterministic, the distribution of states is stochastic, given that it depends on the action at time I, which is sampled from the policy (Fq. 4). We define the state distribution follows.
Definition 1. Given the distribution
conterts p(C), the (determinústic) distribution over time = öt), and the diffusion policy n, the state distribution at time t is:
p(s: σ) p(c)o(t) (xx) e, ().T(xxx1l1)..... dx++1 X1s1
Subsequently, the normalized discounted state visitation distribution can be defined as:
d' (s)= = (1- Σ
?' p(ss = IT)
(11)
t=
The advantage function is defined as: T (s. a) Q (s, - = (s) Sutton & Barto 2018 Given this, the monotonic policy improvement bound can be derived:
Theorem 3.1 (Achiam et al. 2017 Consider current policy Ti. Let C = max Ea [ (s, a)]],
and TV TT] [s)) represent the total variation distance between the policies TT( s) and mT( s)s and s be the current state. For any future policy म, we have:
J(T) J(Tx) 2 I-
1
E (s.aa)n
T(a s)
A TH (s)
mT.
(1 (1-7)) 71) Egna (TV(T() ()) T) [s)]]-
[2 (a

---

Published in Transactions on Machine Learning Research (03/2026)
A direct consequence of this theorem is that when optimizing a policy with the IS objective (Eq. 10 to guarantee that the new policy will improve upon the previous policy, the policies should not diverge too much. Therefore, we need to apply constraint on the currntt policy. This can be achieved by applying the clipping operator in the PPO objective (Eq. 3 (Achiam et al., 2017 Gupta et al., 2024a c Queeney et al.)
2021: Schulman el al. 2017)- We provide an empirical comparison of PPO with and without clipping
Appendix D demonstrating the practical importance of the clipping operator for stable and effective diffusion fine-tuning.
This gives rise to sample Afficiency-effecttoeee trode-off between REINFORCE and PPO. REINFORCE offers greater implementation simplicity, requiring fewer hyperparameters and less memory overhead, but at the cost of lower sample efficiency and suboptimal final performance. PPO, despite its implementation overhead, achieves superior sample efficieccy and reliable policy improvements through importance sampling and clipping.
We note that a similar trade-off analysis was performed in the context of RL fine-tuning for large language models (LLM) (Ahmadian et al.) 2024). However, their analysis was limited empirical study, whereas we present theoretical analysis in addition to the empirical analysis. To the best of our knowledge, we are the first to conduct such a study for diffusion methods.
4 Method: Leave-One-Our PPO (LOOP) for Diffusion Fine-tuning
We demonstraaed the importance of PPO in enhancing sample elficiency and achieving stable improvements during training for diffusion fine-tuning. Additionally, we showcased the RLOO method's effectiveness in roducing the variance of the REINFORCE mothod.
In this section, we introduce our proposed method, LOOP, a novel RI. for diffusion fine-tuning method combining the best of both worlds. While PPO achieves superior sample efficiency over REINFORCE, its objective is still estimated from single trajectory per prompt, which can result in high-variance gradient estimates. We exploit this remaining source of variance our starting point for improving upon PPO.
The expectation in the PPO loss (Eq. 3p is typically estimated by sampling a single trajectory from the policy in the previous iteration Toldi X0.T ~ Fotd for a given prompt c:
T Σ K==
To To(x c) Tola (x (x,, c) 1 - e, 1 + r(xo,c)
clip
(12)
Even though the single sample estimate is an unbiased Monte- Carlo approximation the expectation, it can have potentially high-variance (Owen 2013). Additionally, the IS term mx(x,
[|x,,c) can also contribute to
Tatal
xx,e)
high-variance of the PPO objective (Swaminathan & Joachins 2015 Xie el al. 2023 Both factors combined,
can lead to high-variance, and unstable training of the PPO.
Taking inspiration from RLOO (Eq. g we sample K independent trajectories from the previous policy for a given prompt e, and apply a baseline correction term from each trajectory's 's reward, to reduce the variance of the estimator:
1L (अ)
1 K
clip
X c) [ x c)
To [x:
e, 10) (r(x6)c)-10) b³]
(13)
t=0 Fold
where xb.r ~ nold; Vi E [1, KJ. The baseline correction term by reduces the variance of the gradient estimate, while being unbiased in expectation (Gupta et al.) 2023 2024b Mohamed et al. 2020). A simple choice
of bascline correction can be the average reward across the K trajectories. However, it results in biased estimator (Kool et. al. 2019). Therefore, choose the leave-one-out average baseline, with average taken across all samples in the trajectory, except the current sample i, ie.,
8¹
Σ r(x))-
(14)
k
8

---

Published in Transactions on Machine Learning Research (03/2026)
We note that the baseline DDPO (PPO) method for diffusion fine-tuning [Black et al. 2023) employs different variance-reduction strategy. Specifically, DDPO uses a running-mean baseline, where the baseline term is computed as running average of rewards for a given prompt across optimization steps. This running mcan is subtracted from the reward to reduce gradient variance. In contrast, LOOP adopts a leave-one-out baseline computed across the K independently sampled trajectories within the same optimization step. Unlike the running-mean bascline, which aggregates rewards across iterations and may introduce bias due to policy drift, the leave-one-out baseline remains unbiased within each batch of sampled trajectories and provides stronger variance reduction by leveraging multiple contemporaneous samples per prompt. We provide direct empirical comparison between the running-mean baseline used in DDPO and the leave-one-out baseline used in LOOP in Appendix
Originally RLOO sampling and baseline corrections were proposed in the context of REINFORCE, with a focus on on-policy optimization (Ahmadian et al. 2024 Kool et al., 2019), whereas we are applying these the off-policy step of PPO. We call this method leave-one-ous PPO (LOOP).
Our approach is conceptually similar to the recently popular GRPO method for RL fine-tuning of LLMs (Shao et al. 2024). Although our work was developed independently before GRPO gained widespread recognition, we do not include head-to-head comparison.
Technically, the distinction lies in following aspects: (i) unlike GRPO, our formulation does not apply standard-deviation normalization in the denominator, as this has been shown to potentially harm performance in recent LLM fine-tuning via RL studies (Liu et al., (025). (ii) similar to GRPO, we omit a KL penalty term, since our empirical experiments showed that it has little practical benefit. Furthermore, recent study showed that on-policy RL implicitly constrains the updated policy to remain close to the base policy under KL divergence measure, even without explicit KL penalty term (Shenfeld et al. 2025), and (iii) we ignore the generation-length normalization term. In the diffusion setting, this simplification is further justified by the fact that the sequence length of the reverse diffusion process is fixed across generations, rendering length normalization unnecessary. We provide an empirical comparison on GRPO vs. LOOP in Appendix B We further evaluate whether removing the KT. term from the objective reduces diversity in the generated images by examining the reward diversity trade-off; qualitative results are presented in Appendix F
Provenly, LOOP has lower variance than PPO: Proposition 4.1. The LOOP estimator j1 11.0P(()) (T) (Eq. 13) has lower variance than the PPO estimator
(=) (Eq.) 12
Var (1LO0((M) Var (p)
(15)
Proof. Since the sampled trajectorics are independent:
Var J [LOOP(
Var JPPO
< Var {P
K2
5 Experimental Setup
Benchmark. Text-to-image diffusion and language models often fail to satisfy an essential reasoning skill of attribute binding. Attribute binding reasoning capability refers to the ability of a model to generate images with attributes such as color, shape, texture, spatial alignment, (and others) specified in the input prompt. In other words, generated images often fail to bind certain aftributes specified in the instruction prompt (Fu & Cheng. 2024 Huang al. 2023 Ramesh et al., 2022). Since attribute binding seems to be a basic requirement for useful real-world applications, we choose the TelcChmpBench benchmark (Huang et al, 2023), which contains multiple attribute modding/image compositionality tasks, and its corresponding reward metric to benchmark text-to-image generative models. We also select two common tasks from prior RL for diffusion work: improving aesthetic quality of generation, and image-text semantic alignment. (Black et al.) 2023 Fan et al.) 2024h. To summarize, we choose the following tasks for the RL optimization: (i) Color, (ii) Shape, (iii) Texture, (iv) 2D Spatial, (v) Numeracy, (vi) Aesthetic, (vii) Image-text Aligument. For all tasks, the prompts are split into training/validatiin /validation prompts. We report the average reward on both training and validation split.
9

---

Published in Transactions on Machine Learning Research (03/2026)
Model. As the base diffusion model, we use Stable diffusion V2 [Rombach et al. 2022 which is latent diffusion model. For optimization, we fully update the UNet model, with a learning rate of 1e-5, We also tried LORA fine-tuning (Hu et al. 2021). but the results were not satisfactory, so we update the entire model instead.
6 Hyperparameter and Implementation Details
For REINFORCE (including REINFORCE with baseline correction term), PPO, and LOOP the number of denoising steps (T) is set to 50. The diffusion guidance weight is set to 5.0. For optimization, we use
AdamW (Loshchilov & Hutter, 2017) with learning rate le and the weight decay of le-4, with other parameters kept the default value. We clip the gradient norm 1.0. We train all models using 8 A100 GPUs with batch size of 4 per GPU. The clipping parameter for PPO, and LOOP is set to le-1.
For all experiments, apply dlassffir-frre guidance with guidance scale of 5.0.
DDIM sampler with 50 inference steps, set the noise parameter to 1 1.0, and
For details of the implementation and a complete pscudo-code of the LOOP algorithm, we refer the reader to Appendix
7 Results and Discussion
7.1 REINFORCE vs. PPO Efficiency--ffectiiveesss Trade-off
We present our empirical results on the efficiency-ellectivveess trade-off between REINFORCE and PPO. Our evaluation compares the following methods: the REINFORCE policy gradient for diffusion fine-tuning (Eq.
7
the REINFORCE policy gradient with a baseline correction term (REINFORCE w BC), detailed
in Eq.
where the baseline term the average reward for the given prompt (Black et al. 2023 and the
PPO
objective for diffusion fine-tuning, which incorporates importance sampling and clipping, as outlined
Eq.
3
This PPO objective equivalent the DDPO objective in the original RL for diffusion method (Black
al.
2023
Figure 2 shows the training reward over epochs for the attributes: Color, Shape, and Texture from the 12l-CompBench benchmark, and training reward from optimizing the aesthetic model. Results are averaged over 3 runs. clear that REINFORCE policy gradient is not effective in terms of performance, as compared other variants. Adding a baseline correction term indeed improves the training performance, validating the effectiveness baseline terms of training performance, possibly because of reduced variance. PPO achieves the highest training reward, validating the effectiveness of importance sampling and clipping for diffusion fine-tuning
Color
Shape
Texture
Aesthetic
6.5
Reward
6.75
0.53
0.69
6.5
6.42
0.00
659
5.0
440
Epochs
Epochs
Epochs
Epochs
Figure 2: Evaluating REINFORCE vs. PPO trade-off by comparing: REINFORCE (Eq. 7. REINFORCE with baseline correction term (Eq D. and PPO (Eq. B}. We evaluate on the T2I- 121-CompBench benchmark over three image attributes: Color, Shape, and Texture. We also compare on the acsthetic task. Y-axis corresponds the training reward, x-axis corresponds the training epoch. Results are averaged over three independent runs; shaded regions indicate +1 standard deviation around the mean, computed three independent runs.
10

---

Published in Transactions on Machine Learning Research (03/2026)
We also evaluate the performance on separate validation set. For each validation prompt, we generate 10 independent images from the dilfusion policy, and average the reward, finally averaging over all evaluation prompts, The validation results are reported in Table [ The results are consistent with the pattern observed with the training rewards, i.., REINFORCE with baseline provides a better performance than plain REINFORCE, suggesling that baseline correction indeed helps with the final performance. Nevertheless, PPO (DDPO) still performs better than REINFORCE.
Table I:
Comparing REINFORCE with DDPO the T21-CompBench benchmark over three image altributes: Color, Shape, and Texture. We report average reward on unseen test set (higher is better). For each prompt, average rewards over 10 independent generated images are calculated. We report the mean and standard deviation across 3 independent runs.
Method
Color
Shape f
Texture t
REINFORCE
0.6438 (0.0132)
0.5330 (0.0105)
0.6359 (0.0094)
REINFORCE w/ BC 0.6351 (0.0344)
0.5347 (0.0097)
0.6656 (0.0134)
DDPO
0.6821 (0.0030) 0.5655 (0.0185) 0.6909 (0.0138)
We now have empirical evidence supporting the efficienc--ffecctiveees trade-off discussed Section 3 From these results, we can conclude that fine-tuning lext-lo-image diffusion models is more ellective with IS and clipping from PPO, or baseline corrections from REINFORCE. This bolsters our motivation for proposing LOOP approach to effectively combine these methods.
7.2 Evaluating LOOP
Next. we discuss the results from our proposed RL for diffusson fine-tuning method, LOOP.
Performance during training. Figure shows the training reward curves for different tasks, against number of epochs. LOOP outperforms DDPO [Black et al.] 2023) across all seven tasks consistently throughout training. This establishes the effectiveness of sampling multiple diffusion trajectories per input prompt, and the leave-one-out baseline correction term (Fq. 9 during training. Training reward curve is smoother for the aesthetic task, compared to tasks from the 121-CompBench benchmark. We hypothesise that improving the attribute binding property of diffusion model is harder task than improving the aesthetic quality of generated images.
Table 2: Comparing the performance of the proposed LOOP method with state-of-the-ard baselines on the T2I-CompBench benchmark over image attributes such Color, Shape, Texture, Spatial relation, and Numeracy. The metrics reported are the average reward on an unseen test set (higher is better). For each prompt, rewards are averaged across 10 generated images. For DDPO and LOOP, we additionally report the mean and standard deviation across independent runs.
Model
Color
Shape
Texture 1
Spatial
Numeracy 1
Stable v1.4
Rombach al.
2022
0.3765
0.3576
0.4156
0.1246
0.4461
Stable v2 Rombach ct al. 2022
0.5065
0.4221
0.4922
0.1342
0.4579
Composable v2
[iuu et al.
2022
0.4063
0.3299
0.3645
0.0800
0.4261
Structured v2 Feng et al.
2022
0.4990
0.4218
0.4900
0318
0.4550
Attn-Exet v2
Cheler et
2023
0.6400
0.4517
0.3963
0.1455
0.4767
GORS unbiased
Huang et
2023
0.6414
0.4546
0.6025
0.1725
GORS Huang el al.
2023
0.6603
0.4785
0.6287
0.1815
0.4841
DDPO Black et al.
2023
0.6821
(0.0030)
0.5656 (0.0185)
0.6909
(0.0138)
0.1961
(0.0034)
0.5102 (0.0041)
LOOP 2)
0.6786
(0.0037)
0.5746 (0.0125)
0.6938 (0.0047)
0.1801 (0.0085)
0.5072 (0.0052)
LOOP (k: 3)
0.7516
(0.0097)
0.6220 (0.0173)
0.7354 (0.0071)
0.1966 (0.0058)
0.5242 (U.0062)
LOOP (A: = 4)
0.7859 (0.0114)
0.6676 (0.0021)
0.7519 (0.0036)
0.2137 (0.0073)
0.5423 (0.0014)
Table 2 reports the average rewards on the test set across various tasks from the T2I-CompBench benchmark.
For each prompt, we generate 10 different images and calculate the average rewards. LOOP consistently
outperforms DDPO (Black et. al. 2023) and other strong supervised learning-hased baselines across all
11

---

Published in Transactions on Machine Learning Research (03/2026)
Color
Shape
Texture
0.60
0.75
Reward
0.59
0.70
0.55
0.50
0.45
ITO
0.45
1007 3-
LOop
0:40
1OOP 2)
0.50
(6=3)
(x-2)
0.40
0.35
25
30
75
100
125
150
175
200
34
15
100
150
175
30
15
100
125
150
175
200
Spatial
Image-text alignment
Aesthetic
8.5
5.625
0.70
5.000
0.60
7.5
Reward
0.60
0.55
65-
0.50
PPO Logt (km)) LOST (c-3)
L001 S-z)
0.45
L031 kmz)
1037 2-
LOOP 3--1
5.
LDOP 00
LOOP 00
LOOP (6-4)
5.0
200
275
150
200
125
200
Figure 3: Comparing DDPO (referenced as PPO) with the proposed LOOP on the 121-CompBench benchmark with respect to image attributes: Color, Shape, Texture, and Spatial relationship. We also report results aesthetic preference and image text aligmment tasks (Black et al. 2023). The y-axis shows training reward, and the x-axis shows training epoch. Results are averaged over three independent runs; shaded regions indicate +1 standard deviation around the mean, computed over three independent. runs.
tasks. Notably, LOOP achieves relative improvements of 18.1% and 15.2% over DDPO on shape and color attributes, respectively.
For the aesthetic and image-text alignment objectives, the validation rewards are reported in Table 3 LOOP results in a 15.4% relative improvement over PPO for the aesthetic task, and 2.4% improvement over PPO for the image-texl aligument. lask.
Table 3: Comparing the performance of LOOP with DDPO on the aesthetic and image-text alignment tasks. Higher values are better. For each prompt, rewards are averaged across 10 generated images. We report the mean and standard deviation across independent runs.
Method
Aesthetic
Image Align. t
DDPO Black et al. 2023
6.6795 (0.0925)
20.456 (0.4383)
LOOP (A: 2)
6.7338 (0.0792)
20.773 (0.4383)
LOOP (I: 3)
7.1213 (0.0370)
20.623 (0.0963)
LOOP (k: 4)
7.7061 (0.1006) 20.912 (0.0787)
Impact of number of independent trajectories (k). The LOOP variant with number of independent
trajectories K 4 performs the best across all tasks, followed by the variant K 3. This is intuitive given that Monte-Carlo estimates get better with more number of samples (Owen 2013) Surprisingly, the performance of the variant with K = 2 is comparable to PPO.
8 Qualitative Examples
For a qualitative evaluation of the autribute-binding reasoning ability, we present some example image generations from SD, DDPO, and LOOP in Figures 1 and 5
In Figure D qualitative examples of the attribute binding task are presented. In the example in the first column of Figure [ the input prompt specifies a black ball with white cat. Stable diffusion (SD) and PPO fail to bind the color black with the generated ball, whereas LOOP successfully binds that attribute.
12

---

Published in Transactions on Machine Learning Research (03/2026)
SD v2
PPO
LOOP
A puppy dog"
"A cat"
"Butterlly"
"Bright yellow sunflower in green field"
"Crystal clear mountain lake reflecting snow-capped
peaks"
Figure 4: LOOP improves acsthetic quality. Qualitative examples are presented from images generated via: Stable Diffusion 2.0 (first row), PPO (second row), and LOOP = 4 (third row). LOOP consistently generates more aesthetic images, as compared to PPO and SD.
Similarly, in the third column, SD and PPO fail to bind the hexagon shape attribute to the watermelon, whereas LOOP manages to do that. In the fourth column, SD and PPO fail to add the horse object itself, whereas LOOP adds the horse with the specified black color, and flowing cyan patterns.
Figure highlights improvements in aesthetic quality the generated images. Compared to SD v2 and PPO, LOOP produces sharper, more coherent compositions with balanced lighting and color tone. For example, the second column ("a cat") and in the fourth column ("butterfly"), LOOP enhances realism and contrast while preserving overall artistic intent.
Finally, Figure 5 presents additional qualitative examples that emphasize both binding and aesthetics. LOOP accurately binds challenging color-object pairs (e.g., teal branch, pink cornfield) while producing more visually appealing and natural results. PPO and SD v2 often miss attribute alignment or produce dull, less cohesive scenes.
9 Conclusion
We have studied the efficiency-ellecctivenes trade-off between two fundamental RL methods for diffusion fine-tuning: REINFORCE and PPO. Our analysis, both theoretical and empirical, demonstrates that while REINFORCE is simpler to implement, requiring less memory and fewer hyperparameters, it suffers from high variance and poor sample efficiency compared to PPO. PPO, though more sample efficient and more effective in terms of reward optimization, comes with significant implementation overhead, requiring three models in memory simultaneously and sensitive hyperparameter tuning.
Building on these insights, have introduced LOOP, a novel RL method for diffusion fine-tuning that combines variance reduction techniques from REINFORCE (multiple trajectory sampling and leave-one-out.
13

---

Published in Transactions on Machine Learning Research (03/2026)
SD v2
PPO
LOOP
"A white feather on "A neon orange owl "Pink bridge over a "A pink cornfield" black velvet sitting teal branch"
"A yellow flower in green vase"
glowing blue river"
surface"
Figure Additional qualitative examples presented from images generated via Stable Diffusion 2.0 (first row), PPO (second row), and LOOP & = 4 (third row). LOOP consistently generates more acsthetic images, compared to PPO and SD (first, third, and fifth prompt). LOOP also binds the color attribute (teal branch in second example, and pink cornfield in the forth example), where SD and PPO fail.
baseline correction) with the robustness and sample elficiency of PPO (importance sampling and clipping). Our empirical evaluation on the T2I-CompBench benchmark demonstrates that LOOP achieves substantial improvements over both the base Stable Diffusion model and the state-of-the-art PPO method across multiple tasks, including attribute binding (color, shape, texture, spatial relationships), aesthetic quality, and image-text alignment.
Quantitatively, LOOP 4) achieves substantial improvements over PPO across all evaluated tasks. On the T21-CompBench benchmark, LOOP achieves relative improvements of 18. 18.1% shape binding, 15. 15.2% color binding, 8.8% on texture binding, and 8.9% spatial reasoning, LOOP also improves aesthetic
quality by 15.4% and image-text aligmment by 2.2%. Qualitatively, as shown in Figures 1 4 and 5 LOOP
successfully binds attributes that previous methods fail to caplure, while also producing more visually coherent. and aesthetic images.
A limitation of LOOP is the increased computational cost from sampling multiple diffusion trajectories per prompt, which leads to longer training times compared standard PPO. Future work could explore adaptive sampling strategies to reduce this overhead while maintaining LOOP's effectiveness, extend the method to other diffusion architectures and modalities, or investigate the integration of human preference modeling for better alignment with real-world objectives.
Acknowledgements
We Uhank our reviewers for their helpful leedback. This research was supported by Huawei Finland through DreamsLab, the Dutch Research Council (NWO), under project numbers 024.004.022, NWA.1389, 1.AA.1339.0..11.. 183,, KICH3.LTP.22 20.006, and VI. feni.222.220 222.222.26 26, and the Europcan Union under grant agreements No. 101070212 (FINDHR) and No. 101201510 (UNITE). Views and opinions expressed are those of the author(s) only and do not necessarily reflect those of their respective employers, funders and/or granting authorities.
14

---

Published in Transactions on Machine Learning Research (03/2026)
References
Joshua Achiam, David Held, Aviv Tamar, and Picter Abbeel. Constrained policy optimization. In International Conference on Machine Learning, pp. 22-31. PMLR, 2017.
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadace, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting REINFORCE style optimization for learning from human feedback in LLMs. arxiv preprint orXiv:2402. 14740, 2024.
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. Adeances in Neural Information Processing Systems, 34:17981-17993, 2021.
Brian R. Bartoldson, Siddarth Venkatraman, James Diffenderfer, Moksh Jain, Tal Ben-Nun, Scanie Lee, Minsu Kim, Johan Obando-Ceron, Yoshua Bengio, and Bhavya Kailkhura. Trajectory balance with asynchrony: Decoupling exploration and learning for fast, scalable LLM post-training. In The Thirly-ninth Annual Conference Neural Information Processing Systems, 2025. URL https://openneview.nee. net//
Forum?id-VaPtiwoDn
Kevin Black, Michael Janner, Yilun Du, llya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arviv preprinl or Xiv:2200 13301, 2023.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Soung Wook Kim, Sanja Fidler, and Karsten Kreis. Aligu your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 22563 22575, 2023.
Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. ACM Transactions on Graphics (TOG), 42(4):1-10, 2023.
Logan Engstrom, Andrew Llyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep RL: A case study on PPO and TRPO. In International Conference on Learning Representations, 2019.
Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammnad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Reinforcement learning for fine-tuning text-to to-image diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-fres structured diffusion guidance for compositional text-to-image synthesis. arXiv preprint arXiv:2212. 05032, 2022.
Huan Fu and Guoqing Cheng. Enhancing semantic mapping in text-to-image diffusion via galher-and-bind. Computers & Graphics, 125:104118, 2024.
Alexander Gambashidze, Anton Kulikov, Yuriy Sosnin, and Tyy Makarov. Aligning diffusion models with noise-conditioned perception. arXiv preprint arXiv:2406. 17636, 2024.
Evan Greensmith, Peter I. Bartlett, and Jonathan Baxter. Variance reduction techniques for gradient. estimates in reinforcement learning. Journal of Machine Learning Research, 5(9), 2004.
Yi Gu, Zhendong Wang, Yueqin Yin, Yujia Xie, and Mingyuan Zhou. RPP Aligning diffusion models through relative preference optimization. arXiv preprint arxiv:2406. 06382, 2024.
Shashank Gupta. Safe, Efficient, and Robust Reinforcement Learning for Ranking and Diffusion Models. PhD thesis, University of Amsterdam, 2025.
Shashank Gupta, Harric Oosterhuis, and Maarten de Rijke. Safe deployment for counterfactual learning rank with exposure-based risk minimization. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 249 258, 2023.
15

---

Published in Transactions on Machine Learning Research (03/2026)
Shashank Gupta, Philipp Hager, Jin Huang, Ali Vardasbi, and Harric Oosterhuis. Unbiased learning to rank: On recent advances and practical applications. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, pp. 1118 1121, 2024a.
Shashank Gupta, Olivier Jeunen, Harric Oosterhuis, and Maarten de Rijke. Optimal bascline corrections for off-policy contextual bandits. In Proceedings of the 18th ACM Conference on Recommender Systems, pp- 722 722-732, 2024b.
Shashank Gupta, Harrie Oosterhuis, and Maarten de Rijke Practical and robust safety guarantees for advanced counterfactusl learning to rank. In Proceedings of the 33rd ACM International Conference Information and Knowledge Management, pp. 737 747, 2024c.
Zhenyu Han, Ansheng You, Haibo Wang, Kui Luo, Guang Yang, Wenqi Shi, Menglong Chen, Sicheng Zhang, Zeshun Lan, Chunshi Deng, et al. AsyncFlow: An asynchronous streaming RL framework for efficient LLM post-training. orXiv preprint ar Xixx:2507.01666, 2025.
Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840 33:6840-6851, 2020.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LORA: Low-rank adaptation of large language models. arXiv preprint ar arXi::2106 00.00800 2021.
Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2--ComBennhh: comprehensive benchmark for open-world compositional text-to-image generation. Advances in Neural Information Processing Systems, 36:78723-78747, 2023.
Shengyi Huang, Michael Noukhovitch, Arian Hosscini, Kashif Rasul, Weixun Wang, and Lewis Tunstall. The N+ implementation details of RLHF with PPO: A case study TL; DR summarization. arXiv preprint ar arXiv:2403. 1 17011, 2024.
Bingyi Kang, Xiao Ma, Chao Du, Tianyu Pang, and Shuicheng Yan. Efficient diffusion policies for offline reinforeement learning. Advances in Neural Information Processing Systems, 36:67195-67212, 2023.
Wouter Kool, Herke van Hoof, and Max Welling. Buy 4 REINFORCE samples, get a baseline for free! ICLR 2019 Deep Reinforcement Learning meets Structured Prediction Workshop, 2019.
Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint XrX:::2302. 12192, 2023.
Shufan Li, Konstantinos Kallidromitis, Akash Gokul, Yusuke Kato, and Kazuki Kozuka. Aligning diffusion models by optimizing human utility, arXiv preprint arXiv:2404.0 044622444444 2024.
Nan Liu, Shuang Lú, Yilun Du, Antonio Torralba, and Joshua Tenenbaum. Compositional visual generation with composable diffusion models. In European Conference on Computer Vision, pp. 423-439. Springer, 2022.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding R1-Zero-like training: A critical perspective. arXio preprint ar Xiw:2503.20783, 2025.
llya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2017. URL
Shakir Mohamed, Mihaela Rosca, Michael Figurnov, and Andriy Mnih. Monte Carlo gradient estimation machine learning. Journal of Machine Learning Research, 21(32)):1-62, 2020.
Art B. Owen. Monte Carlo Theory, Methods and Faamples. https: sw.domainsmmc// 2013.
16

---

Published in Transactions on Machine Learning Research (03/2026)
James Queeney, Yannis Paschalidis, and Christos G. Cassandras. Generalized proximal policy optimization with sample reuse. Advances in Neural Information Processing Systems, 34:11909-1199 11919, 2021.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Newral Information Processing Systems, 36:53728 53741, 2023.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204 061200000665, 1(2):, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pallern recognition, pp. 10684 10695, 2022.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale datasot for training next generation imago-text models. Advances in Neural Information Processing Systems, 35:25278 25294,, 2022.
John Schulman. Trust region policy optimization. arXiv preprint arXi::1502.053 157,00070000 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arxiv preprint (rXXu::707.063// 2017.
Zhihong Shao, Pciyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowci Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits mathematical reasoning in open language models. arXiv preprint arXiv:2402.0 0330000 2024.
Idan Shenfeld, Jyothish Pari, and Pulkit Agrawal. RL's razor: Why online reinforcement learning forgets less. arXiv preprint ar Xiv: 04259, 2025.
Jascha Sohl-Dickstein Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibriun thernodynamics. In International conference on machine learning, pp. 2256 2265. PMLR, 2015.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, 2018.
Adith Swaminathan and Thorsten Joachims. The self-normalized estimator for counterfactual learning. In Advances in Neural Information Processing Systems, volume 28, 2015.
Siddarth Venkatraman, Moksh Jain, Luca Scimeca, Minsu Kim, Marcin Sendera, Mohsin Hasan, Luke Rowe, Sarthak Mittal, Pablo Lemos, Emmanuel Bengio, et al. Amortizing intractable inference in diffusion models for vision, language, and control. Advances in neural information processing systems, 37:76080 76114, 2024.
Bram Wallace, Maihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Sonthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shaliq Joty, and Nikhil Naik. Diffusion model aligmment using direct preference optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8228 8238, 2024.
Chenyu Wang, Masatoshi Uehara, Yichun He, Amy Wang, Tommaso Biancalani, Avantika Lal, Tommi Jaakkola, Sergey Levine, Hanchen Wang, and Aviv Regev. Fine-tuning discrete diffusion models via reward optimization with applications to dna and protein design. ar preprint arXiv:2,10.136433 2024.
Ronald J. Williams. Simple statistical gradient-followngg algorithms for connectionist reinforcement learning. Machine Learning, 8:229 256, 1992.
Zhengpeng Xie, Changdong Yu, and Weizheng Qiao. Dropout strategy in reinforeement learning: Limiting the surrogate objective variance in policy optimization methods. Xix preprint arXiv:2310.20380 2023.
17

---

Published in Transactions on Machine Learning Research (03/2026)
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
Minkai Xu, Alexander S Powers, Ron O Dror, Stefano Ermon, and Jure Leskovee. Geometric latent diffusion models for 3D molecule gencration. In International Conference on Machine Learning, pp. 38592-38610. PMLR, 2023.
Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang, Haiteng Zhang, and Jun Wang. Token-level direct preference optimization. Xiv preprint orXiv:2/04.114 119999 2024.
Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Yuhao Zhou, Limao Xiong, et al. Delve into PPO: Implementation matters for stable RLHF. In NewrIPS 2023 Workshop on Instruction Tuning and Instruction Following. 2023.
Han Zhong, Guhao Feng, Wei Xiong, Xinle Cheng, Li Zhao, Di He, Jiang Bian, and Liwei Wang. DPO meets PPO: Reinforced token optimization for RLHF, Xiv preprint arXiv:2404.1800 18922, 2024.
18

---

Published in Transactions on Machine Learning Research (03/2026)
A LOOP Algorithm Pseudo-code
This section provides a detailed pseudo-code implementation of the LOOP (Leave-One-Out PPO) algorithm for diffusion fine-tuning. The algorithm, shown in Figure 6 consists of four main steps. Step sampling multiple trajectories per prompl from the old policy. Step computing leave-one-out baseline advantages (Eq. [4p. Step updating the policy using clipped LOOP objective with importance sampling (Eq. 13). And Step performing gradient descent.
LOOP: Ceeve-One-uut PPO for Diffusson Fine-tuning
Input: prompts, old_policy, reward_funee function Hyperparameters: (trajectories per pronpt), epsilon (clipping)
prompt training.prompte::
# Step 1: Sample K trajectories from old policy trajectories
revards = [
for in range(K):
x_noise samplee_gausiannnoisel
traj ddffuslot_reveroo.prro wwt.ppoocccoo........oooo (old_policy, prompt, x_noise) trajectories. append (traj) (trj))
rewards... append (reward_functionn) (traj wtraa.tfnall.................i prompt))
# Step 2: Compute leave-one baseline advantages
advantages - for in range(K):
Baseline: average of all revards excluding i-th reward baseline_i for j renge(K):
if j baseline_i rewards [j] baseline_i baseline_i / (K 1)
Advantage: reward minus leave -ot baseline
advantages.. append (rewards [i] baseline_i)
# Step 3: LOOP update with inportance sampling and clipping loss 0 for in range(K):
for t in range (num coun_tinesteps):
# Compute importance ratio
Log_prob_new
current_polict prob (traj [i],
log_prob_old old_policy. 1og_ (traj t)
ratic exp (log_prob_new log_prob_old)
Clipped surrogate objective
clipped_ratio crip(ratio, 1-epsilon, Itepsilon) loss -min(ratio advantages clipped advantages [i])
Step 4: Gradient descent optimize (1oss)
Figure 6: Pseudo-code for LOOP. The algorithm samples K trajectories per prompt, computes leave-one-out advantages A R² E5 R,, and updates the policy with clipped importance sampling.
19

---

Published in Transactions on Machine Learning Research (03/2026)
B LOOP vs. GRPO
In this section, we compare LOOP with GRPO (Shao et al. 2024p. In the intrucutioon (Section 1 we provided theoretical comparison of LOOP with GRPO. In this section, we provide an empirical comparison on the aesthetic task.
Figure 7 presents results from small-scale experiment comparing GRPO with LOOP on the aesthetic fine-tuning task. We conducted this auxiliary study because the base model used in our main experiments, Stable Diffusion 2.0, has since been deprecated I As a result, we ran both methods using an unofficial fork of SD 2.0 2
68
58
GRPD
ecoch
Figure 7: Comparision of LOOP vs GRPO the aesthetic task.
The plot shows steady and stable reward improvement for both methods over training epochs. In the early phase, GRPO and LOOP exhibit closely aligned performance, indicating similar initial optimization behavior. As training progresses, LOOP consistently achieves slightly higher aesthetic rewards than GRPO, maintaining a persistent margin through convergence. Overall, while both methods demonstrate stable learning dynamics, LOOP attains better final aesthetic performance in this small-scale setting.
Overall, the figure empirically supports the claim that incorporating leave-one-out baseline correction within PPO-style clipped objective improves optimization effectiveness for diffusion fine-tuning, yielding better alignment with aesthetic reward models while maintaining stable training dynamics.
C Comparing Different Baseline Correction Terms
Figure presents & small-scale empirical comparison between the leave-one-out baseline used in LOOP and the ounning-mean baseline adopted in DDPO (PPO) (Black al. 2023). evaluated on the aesthetic task in the same toy setup used for the GRPO LOOP comparison (Appendix Bp. Both variants demonstrate stable and monotonic reward improvement over training, confirming that either baseline strategy supports effective optimization. However, the leave-one-out baseline consistently achieves higher rewards than the running-meat baseline, with the gap becoming more pronounced in later epochs.
A possible explanation for the weaker performance of the rumning running mean aggregates rewards across different optimization steps, it mixes samples generated by different. policies over time. As the policy evolves, older reward statistics may longer accurately reflect the current policy distribution, leading to misaligned baseline and suboptimal credit assignment. In contrast, the leave-one-out baseline is computed using multiple trajectories sampled from the same policy at the same iteration, making it more locally consistent and better aligned with the current optimization step. This likely contributes to the improved and more stable performance observed with the leave-one-out baseline.
meannng-mea mean baseline is the policy drift. Since the
http:///waghhheeeeeeeesaalllllld base httpe///wweeennneeeee////////hhd 2-1hhaase
20

---

Published in Transactions on Machine Learning Research (03/2026)
LOOP Juunning sverage)
LOOP
Figure 8: Comparision of LOOP vs GRPO on the aesthetic task.
D Importance of Clipping in PPO
Figure 9 presents ablation study evaluating the role of clipping in PPO for diffusion fine-tuning across three T21-CompBench tasks: Color, Shape, and Texture. We compare standard PPO with clipping against PPO without clipping under the same training configuration. Across all three tasks, PPO with clipping consistently achieves higher rewards and exhibits smoother, more stable training dynamics. While PPO without clipping initially improves, its performance is less stable and couverges to lower final reward values. The gap becomes more pronounced in later epochs, indicating that clipping plays an important role in maintaining controlled policy updates and preventing excessive deviation from the reference policy. These results empirically support the theoretical motivation for clipping and demonstrate its practical importance for stable and effective diffusion model fine-tuning.
Task: Color
Task: Shape
0.65
PPO w/o clip
PPO
0.60
PPO w/o clip
PPO
A
0.60
M
0.55
Reward Mean
Reward Mean
0.55
0.50
0.45
0.45
0.35
0.40
0.30
25
50
75
100
125
150
175
200
25
50
75
100
125
150
175
200
Epoch
Epoch
Task: Texture
PPO w/o clip
PPO
0.70
Reward Mean
0.65
0.60
0.55
M
0.50
0.45
0.40
0
25
50
75
100
125
150
175
200
Epoch
Figure 9: Effect of clipping in PPO for diffusion fine-tuning. PPO with clipping achieves more stable training and higher reward than PPO without clipping across Color, Shape, and Texture tasks.
21

---

Published in Transactions on Machine Learning Research (03/2026)
E
Runtime and Memory Usage
Figure 10 summarizes the computational cost of RL fine-tuning across tasks by reporting (a) mean wall-clock runtime and (b) peak GPU memory usage for PPO and LOOP with different numbers of trajectories K. Across tasks, runtime increases with K, reflecting L.OP'ss O(K) sampling overhead from generating multiple diffusion trajectories per prompt, while PPO is consistently the fastest. Peak GPU memory remains broadly comparable across methods, with only modest increases for larger K, suggesting that LOOP's primary cost is additional sampling time rather than substantially higher memory footprint. Overall, the figure highlights the expected efficiency trade-off: LOOP improves sample efficiency at the expense of increased wall-clock compute, with relatively small changes in peak memory.
Hoan Wall (ibtun)) and tiithed
Perfrmmnce by Hethed and Tud: (heac Hemnory
Figure 10: Runtime and memory overhead of LOOP relative to PPO across tasks, The figure on the left reports mean wall-clock runtime (hours) and the figure on the right reports peak GPU memory (GB) for each task and method variant.
F Reward-Diversity, Trade-off
Figure plots the reward (on a log-scale) against generation diversity for PPO and LOOP variants across four prompls (hippopolamus, octopus, snail, and cheetah) on the aesthetic task. Diversity quantified as the average pairwise CLIP cosine distance within a batch of generated images, following previous work (Venkatraman et al. 2024h.
Across all prompts, LOOP with & achieves the highest log reward while simultaneously maintaining high diversity, consistently occupying the Pareto-dominant (upper-right) region of the plot. In contrast, PPO attains the lowest diversity in every case, indicating pronounced mode collapse: optimization concentrates probability mass on narrow subset of high-reward samples.
Increasing in LOOP improves both reward and diversity jointly, rather than trading one for the other. This suggests that sampling multiple trajectories per prompt not only strengthens optimization but also promotes exploration of distinct generation modes. The effect consistent with the leave-one-out baseline, which reinforces relative improvements over average across diverse trajectories instead of amplifying single dominant mode.
22

---

Published in Transactions on Machine Learning Research (03/2026)
sppogctamuus
OcTopss
PRO
RP LOCP kod2
1coPk-2
2.00
2,02
2.00
(ogireeand)
1.46
1.98
1.92
1.52
1.90
1.90
0.075
0.055
0.090
0.11
0.12
oernly ELP distene))
Diversty CLPP contre cresanee)
creatah
2.05
PRD
LOOP
2.12
LOCP kwc
2.04
2.10
Lgjitenadi
2.58
2.04
1.95
2.02
1.94
2.00
1.98
0.06
0.67
0.00
0.10
o.06s. 0.055
0.090
Dvcrtity pairwise
Diversity
Figure 11: Reward vs. diversity trade-off for PPO and LOOP variants across four prompts on the hesth task. We report mean log reward and diversity, measured as the average pairwise CLIP cosine distance across a batch of generated images (Venkatraman et al., 2024). LOOP k=4 consistently achieves the highest log reward while maintaining competitive diversity across all prompts, whereas PPO achieves the lowest diversity scores, suggesting mode collapse towards a narrow set of high-reward images.
23


Reviews:
[{"review_id": "0L0aql8EdJ", "reviewer": "Reviewer_J2GG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Please excuse the very belated review.\n\nThe paper presents a study on RL based finetuning of text-to-image diffusion models. The authors evaluate different design decisions common to policy gradient methods, such as multiple trajectory sampling and clipping. They present results that show that improved performance can be obtained from the specific combination of multiple rollouts and leave-one-out baseline design common to modern REINFORCE variations such as RLOO (an LLM method) and PPO style clipping. They term this combination of design decisions LOOP.", "claims_and_evidence_yn": "No", "claims_explanation": "Note that overall the claims made in the paper are well supported. However, I believe one core part of the claim in the paper is missing, which is going to be the major focus of the method.\n\nWhile PPO does introduce the clipping mechanism to prevent large importance ratios, there is also the design dimension of the value baseline. While the LLM community is currently debating the merits of trained value functions vs multi-trajectory averages, there are several possible ways to introduce the baseline/value estimator into the algorithm. The LOOP method not only uses clipping plus multiple rollouts, it also uses a leave one out baseline. If I understand correctly, the main prior method uses (DDPO) uses a running mean and std deviation. So the presented algorithm changes multiple design choices, but does not fully evaluate where performance improvements come from. As leave-one-out is relevant enough to be chosen as the title of the method, I think this merits investigation.\n\nAs all of the presented design decisions are common practice in the literature, and none require novel exposition or additional theory, I think it is reasonable to ask for the comparison between design choices to be expanded. While all the results are technically correct, the paper boils down to a hyperparameter tuning over the number of rollouts for policy gradients in diffusion. TMLR specifically only asks that claims are substantiated, but I think in this case it is reasonable to also request thoroughness given the relative simplicity of the contribution, and the likely interest of the community in the question which parts of the changed algorithm are relevant for the performance increase.\n\nSome additional minor questions/nitpicks:\n\nHow are the error bars estimated to be at 80% prediction interval if only three runs were trained? What method and statistical assumptions lead to an 80% prediction intervals?\n\nThe results in Figure 2 and Figure 3 do not seem to align. The PPO method evaluated in Figure 2 seems to reach substantially different returns as the PPO method in Figure 3. i might have missed the differences, but I think the naming should be distinct if these are distinct methods. The table is again inconsistent with the numbers in the Figure. If the table is on a held-out set, it is also strange why the results would be better by a decent margin than those achieved on the training set (compare Figure 2 Color PPO with Table 1 Color DDPO).", "audience_yn": "Yes", "audience_explanation": "While the methods are well established in other parts of the literature, I believe there is merit and potential interest in seeing these insights tested and replicated in other parts of the literature.", "requested_changes": "Please add a comparison between the running mean and std deviation used as the baseline in DDPO, and the leave-one-out baseline. if possible, comparing against a trained value function would be even better, but since this seems to be non-standard in the related work, I accept if the authors declare this as out-of-scope.\n\nPlease clarify the statistical methodology and explain the different results in different figures.\n\nPlease add meaningful confidence intervals to the tables as well for clarity.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rO29PngpAw", "reviewer": "Reviewer_yJgL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies reinforcement learning (RL)–based fine-tuning of text-to-image diffusion models under black-box rewards, with a particular focus on the trade-off between simple REINFORCE-style policy gradients and Proximal Policy Optimization (PPO). The authors first formalize diffusion sampling as an MDP and re-derive the PPO objective in this setting, along with a diffusion-specific statement of the monotonic policy-improvement bound. They then empirically compare vanilla REINFORCE, REINFORCE with a baseline, and PPO (DDPO) across several reward functions (attribute binding tasks from T2I-CompBench, aesthetics, and image–text alignment), showing that PPO plus baselines improves training stability and final reward over REINFORCE variants. Building on this analysis, they introduce Leave-One-Out PPO (LOOP), which augments PPO with (i) sampling K>1 trajectories per prompt and (ii) a leave-one-out baseline $b_i$ when computing advantages, while retaining importance sampling and clipping. LOOP is shown to outperform DDPO and several non-RL baselines on T2I-CompBench (color, shape, texture, spatial, numeracy) and on aesthetic and image–text alignment rewards, with the strongest results for K=4. Qualitative examples suggest improved attribute binding and aesthetics compared to Stable Diffusion v2 and DDPO. \n\n\nKey strengths:\n1. Clear problem formulation: the paper gives a concise background on diffusion-as-MDP and shows how PPO objectives map to this setting.\n2. Practical, incremental algorithm: LOOP is conceptually simple (RLOO + PPO) and easy to implement on top of existing DDPO codebases.\n3. Strong empirical gains: consistent improvements over DDPO and previous compositionality methods on T2I-CompBench and on aesthetic / alignment tasks, including sizable relative improvements (≈15−18%) on shape/color binding.\n4. Ablation over K: the study of K=2,3,4 provides some insight into the effect of multiple trajectories per prompt and connects to Monte-Carlo variance reduction.\n\nKey weaknesses:\n1. Despite the stated focus on an “efficiency–effectiveness trade-off”, the paper does not provide quantitative compute / memory / sample-efficiency comparisons (e.g., wall-clock time, GPU memory, number of reward evaluations) between REINFORCE, PPO, and LOOP.\n2. The theoretical part largely reuses known results and adapts them to the diffusion setting; the novelty of the theory is modest and not clearly separated from prior work. \n3. LOOP still requires a reference policy and a reward model, and also multiple trajectories per prompt; it is not obvious from the experiments that it is more efficient than PPO for a fixed compute budget.\n4. Limited baselines: there is no comparison to newer RL-style diffusion methods (e.g., DPO-style, Diffusion-RPO, or GRPO-inspired variants), and even for GRPO, the authors explicitly avoid a head-to-head comparison while acknowledging conceptual similarity.\n5. Evaluation is restricted to Stable Diffusion v2 and specific reward models; generalization to other diffusion architectures / resolutions / domains is not demonstrated.\n6. Some claims (e.g., “first systematic study of REINFORCE vs PPO for diffusion,” or “KL regularization has minimal effect”) rest mainly on a narrow set of experiments and prior citations rather than broad empirical evidence within this work. In particular, removing the KL term can lead to significant reduction in the diversity of generation which is undesirable. (see [1][2]) It is necessary to show how the current method effects the sample diversity empirically. \n\n[1] S Venkatraman et. al. Amortizing intractable inference in diffusion models for vision, language, and control\n[2] Y Fan et. al. DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models", "claims_and_evidence_yn": "Yes", "claims_explanation": "For the core empirical claim—that LOOP achieves better reward than DDPO/PPO and REINFORCE-style baselines on T2I-CompBench, aesthetics, and image–text alignment—the evidence is generally convincing:\nThe experimental setup is described with reasonable detail (base model, reward functions, optimization hyperparameters).\nTraining curves (Figures 2–3) and test-set tables (Tables 1–3) consistently show LOOP outperforming DDPO and REINFORCE variants on the reported metrics.\nThe effect of multiple trajectories K is documented and aligns with the Monte-Carlo variance-reduction story (performance improves with larger K, especially moving from K=2 to 4).\n\nHowever, several secondary or stronger phrased claims are only partially supported:\n1. The paper emphasizes an efficiency–effectiveness trade-off and suggests that LOOP “achieves a better balance between computational efficiency and performance,” but there are no measurements of wall-clock time, GPU hours, peak memory, or number of reward calls per achieved reward level. Without such metrics, one can only conclude that LOOP is more effective, not that it is more efficient than PPO for a fixed budget.\n2. The theoretical section adapts known monotonic improvement bounds to the diffusion setting but does not empirically validate the specific bound (e.g., by measuring TV distances or verifying the bound’s tightness); the main role of this section is explanatory, not predictive. I think it might be better for the author's to clarify their contributions there or remove certain things from that section and focus on what additional novelty this work provides. \n3. The work leans on prior empirical observations to justify omitting KL regularization and standard-deviation normalization, but within this paper those design choices are not ablated, so the reader cannot tell how crucial they are in the current setting.\n\nOverall, I would say the central algorithmic and empirical claims are well-supported, while claims about efficiency, generality, and design choices are somewhat under-evidenced and would benefit from additional ablations and metrics. As highlighted before, removing the KL term can lead to significant reduction in the diversity of generation which is undesirable. (see [1][2]) Author should show how the current method effects the sample diversity empirically. Based my personal experience of training models with and without KL term for diffusion fine-tuning, the high rewards/improvements can come at the cost of mode collapse of the sampling distribution.", "audience_yn": "Yes", "audience_explanation": "TMLR’s readership includes researchers working on diffusion models, RLHF/RL-from-feedback, and generative modeling more broadly. This paper sits at the intersection of these topics and addresses a very practical question: how should we fine-tune diffusion models under black-box rewards in a way that is both performant and not overly fragile?\nMore specifically:\n1. RL-based fine-tuning of generative models (LLMs or diffusion) is currently a very active area; a method like LOOP, which is simple and empirically strong, is likely to be of practical interest to people building aligned T2I systems.\n2. The adaptation of RLOO-style variance reduction to diffusion PPO is conceptually straightforward yet non-trivial to implement correctly; a well-documented version with empirical validation is useful to practitioners.\n3. The focus on attribute binding (T2I-CompBench) addresses a widely-recognized weakness of current text-to-image models, so improvements here are directly relevant to many applications.\n4. Even if the theoretical contribution is modest, the empirical insight that “PPO-like methods benefit significantly from multiple trajectories + leave-one-out baselines in diffusion” is a concrete design lesson.\nIn short, this is an incremental but practically meaningful contribution that would interest the segment of TMLR’s audience working on diffusion models, RL for generative models, and text-to-image alignment.", "requested_changes": "important for acceptance;\n1. Quantify the efficiency–effectiveness trade-off.\n Provide comparisons of compute and sample efficiency between REINFORCE, REINFORCE+baseline, PPO, and LOOP for at least one task.\nSuggested metrics: wall-clock time to reach a target reward, total number of reward model evaluations, GPU hours, and peak GPU memory. This is essential to support the framing that LOOP offers a better balance between efficiency and performance.\n2. Clarify what is theoretically novel vs prior work. \nThe monotonic policy improvement bound closely follows Achiam et al.; please clearly state what is formally new and what is a direct specialization of existing theorems.\nConsider moving some of the proof details to an appendix while focusing the main text on the intuition and the implications specific to diffusion.\n3. Ablations on LOOP design choices.\nAdd ablations isolating the contributions of (a) multiple trajectories K, (b) leave-one-out baseline vs simple mean baseline vs no baseline, and (c) clipping vs no clipping, ideally on at least one T2I-CompBench task and one aesthetic/alignment task.\nThis would help demonstrate that the combination proposed in Eq. (13–14) is actually necessary, rather than any one component being sufficient.\n4. Clarify and, if possible, justify the “efficiency” claim for LOOP. \nAs written, LOOP still uses a reference policy, a current policy, and a reward model—like PPO—while also sampling multiple trajectories per prompt. On its face this seems more expensive than DDPO.\nEither (a) show that LOOP reaches higher reward in fewer optimization steps or with less compute, or (b) revise the narrative to emphasize performance and stability rather than computational efficiency.\n5. Strengthen or reframe the relationship to GRPO and other contemporary methods.\nSince the method is “conceptually similar” to GRPO, the absence of any empirical comparison is a noticeable gap. If possible, include at least a small-scale comparison or justification why a direct comparison is infeasible (e.g., implementation issues, domain mismatches).\nAlso consider situating LOOP relative to more recent diffusion-alignment methods (e.g., Diffusion-RPO, DPO-style approaches), even if only at the level of discussion and limitations.\n6. Plot the diversity of samples. As the KL term has been removed it'll make sense to report the sample diversity of the RL-finetuned model for sanity check whether the improvements come at the cost of mode collapse around high reward samples.\n7. Expand experimental coverage or explicitly narrow the claims.\nAll experiments use Stable Diffusion v2; the claims sometimes read more generally (“diffusion models” in general). Either (a) add results on at least one additional base model (e.g., SD v1.5 or a different latent diffusion architecture) or (b) soften the claims to “for SD v2 under the studied rewards” and discuss potential generalization.\n\nWould strengthen the work but not strictly required:\n1. Clarify reward models and evaluation protocol. [Optional]\nProvide more detail on the reward networks used for T2I-CompBench, aesthetic, and image–text alignment tasks (architectures, training data, normalization).\nClarify how training/validation/test splits are constructed for each task and whether any prompts overlap with the pre-training data of SD v2 or the reward models.\n2. Hyper-parameter sensitivity and robustness. [Optional]\nPPO is known to be hyper-parameter sensitive; it would be useful to see whether LOOP is comparably sensitive or more robust (e.g., to learning rate, clipping parameter ϵ, and K).\nEven a small study varying such hyperparameters would add credibility to the “simple and effective” claim.", "broader_impact_concerns": "The paper focuses on RL-based fine-tuning of text-to-image diffusion models, improving attribute binding and aesthetic quality. This has the usual dual-use implications of stronger generative models:\n\n1. Positive: Better alignment with prompts can improve accessibility (e.g., clearer visualizations, educational material), creative tools, and downstream applications like design or scientific illustration.\n\n2. Negative: Improved controllability and aesthetics can also make it easier to generate persuasive misinformation, deepfakes, or harmful content that more faithfully matches malicious prompts.\n\nI do not see novel ethical risks unique to LOOP beyond those already present with text-to-image diffusion and RL-based fine-tuning, but the paper would benefit from a short, explicit discussion of these issues.", "additional_comments": "Overall, I find the paper to be practically useful contribution with very little theoretical novelty. The empirical section is the main strength; which however looks incomplete right now. Tightening the claims around efficiency and adding a few targeted ablations would substantially improve the paper.\nFrom a reproducibility standpoint, releasing code and training scripts for LOOP (or at least pseudo-code plus detailed hyperparameters) would make it much easier for practitioners to adopt the method.", "extra_scores": {}, "extra_text": {}}, {"review_id": "IQsz9OMfJZ", "reviewer": "Reviewer_3DSC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the efficiency–effectiveness trade-off between REINFORCE and PPO for RL-based post-training on text-to-image diffusion models. REINFORCE is simple and cheap, but known as suffering from high-variance and sample-inefficient; PPO is good at performance while expensive and hyperparameter-sensitive.\n\nBased on their theoretical analysis (leveraging results from prior works like Achiam et al., 2017),  and empirical analysis, the authors propose LOOP (Leave-One-Out PPO), which combines 1) PPO’s importance sampling + clipping; 2) REINFORCE/RLOO’s multi-trajectory sampling and leave-one-out baseline variance reduction.\n\nExperiments on T2I-CompBench (containing multiple attribute binding/image compositionality tasks), with evaluation on image aesthetics, and image-text alignment show that LOOP provides stronger performance than standard PPO (DDPO). The key claims from the paper include:\n\t•\tLOOP consistently outperforms baselines in attribute binding (color, shape, texture, spatial).\n\t•\tLOOP improves aesthetic and alignment scores.\n\t•\tLOOP smooths training curves and is more stable.", "claims_and_evidence_yn": "Yes", "claims_explanation": "1. The paper provides a clean theoretical and empirical argument showing limitations of REINFORCE and PPO in diffusion RL settings, establishing a credible motivation for LOOP, using clear articulation (e.g., presentations in main text, visual aids in figures and organized results in tables)\n\n2. The method is intuitive yet nontrivial with the following reasons:\n* Multi-trajectory sampling reduces PPO variance.\n* Leave-one-out baseline avoids bias introduced by simple baselines.\n* Cliping + Importance Sampling to maintain stability.\n\n3. LOOP achieves substantial improvements across all tasks:\n* +18.1% (shape), +15.2% (color), +8.8% (texture) over DDPO (Table 2)\n* +15.4% improvement in aesthetics (Table 3)\n\nThe improvements are significant and appear robust (K=4 best).\n\n4. The paper’s experiments are comprehensive, including:\n*  7 benchmarks (color, shape, texture, spatial, numeracy, aesthetic, alignment)\n*  Multiple baselines, including PPO, stable diffusion baselines, and compositional methods.\n*  3-run averages with prediction intervals.\n\n5. The paper adapts Achiam et al.’s monotonic improvement theorem for diffusion MDPs, which strengthens its theoretical justification for PPO-style clipping in diffusion RL.", "audience_yn": "Yes", "audience_explanation": "The paper studies the efficiency–effectiveness trade-off between REINFORCE and PPO for RL-based post-training on text-to-image diffusion models. The paper leveraging results from prior works like Achiam et al., 2017, proposes a method that empirically works well  attribute binding and improves aesthetic and alignment scores.", "requested_changes": "1. The authors reference GRPO (Shao et al., 2024) and claim conceptual similarity but do not perform empirical comparisons. Considering LOOP is extremely close to GRPO (multi-samples, leave-one-out baseline, etc.), the comparison is non-trivial in my opinion.\n\n2. The authors only study the effect of different $k$. The proposed LOOP still involves several key components that need more careful exploration. For example\n* Effect of leave-one-out\n* Effect of clipping\n* How does the removal of standard-deviation normalization compare with GRPO\n* The sensitivity to diffusion sampler (maybe quantify the relation of stochasticity in diffusion reverse, and the number of sampling steps?)\n\n3. As LOOP requires multi-trajectory sampling, the compute cost should be quantified. The authors  need to further study GPU hours,  memory usage etc. for comprehensive analysis, otherwise it is  hard to support the \"efficiency–effectiveness\" claim. \n\n4. The experiment  settings are not detailed, which may cause misunderstanding from readers who are not fully familiar with diffusion fine-tuning. Please clarify the settings such as:\n* What specific reward models are used per task?\n* Are the pretrained models such as text encoder fixed or also affected by the reward?\n* What sampler the diffusion model uses, do they apply ODE solver, any type of scheduler, fixed noise schedules, etc.\n* How the guidance is applied in the sampling?\n\n5. The monotonic improvement bound (Theorem 3.1) is restated from Achiam et al.\n\n6. The evaluation does not study FiD, perceptual scores and CLIP scores. \n\n7. Math notations are not consistent. For example, Eq. 10 uses normal font for the variables  while other places the variables use bold notations. Are they different? If so, the authors need to define them in the text.\n\n### References\n\n- Shao, Zhihong, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang et al. \"Deepseekmath: Pushing the limits of mathematical reasoning in open language models.\" arXiv preprint arXiv:2402.03300 (2024).", "broader_impact_concerns": "The paper studies principle method for RL, thus is not  significantly concerned on the ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)