PCPO: Proportionate Credit Policy Optimization for Preference Alignment of Image Generation Models
Authors: Jeongjae Lee
Jong Chul Ye
Venue: ICLR 2026 Poster (2026)
Keywords: Text-to-image generation
Reinforcement Learning
Preference Alignment
Primary area: generative models

Abstract: While reinforcement learning has advanced the alignment of text-to-image (T2I) models, state-of-the-art policy gradient methods are still hampered by training instability and high variance, hindering convergence speed and compromising image quality. Our analysis identifies a key cause of this instability: disproportionate credit assignment, in which the mathematical structure of the generative sampler produces volatile and non-proportional feedback across timesteps. To address this, we introduce Proportionate Credit Policy Optimization (PCPO), a
framework that enforces proportional credit assignment through a stable objective reformulation and a principled reweighting of timesteps. This correction stabilizes the training process, leading to significantly accelerated convergence and superior image quality. The improvement in quality is a direct result of mitigating model collapse, a common failure mode in recursive training. PCPO substantially outperforms existing policy gradient baselines on all fronts, including the state-of-the-art DanceGRPO. Code is available at https://github.com/jaylee2000/pcpo/.

Published as a conference paper at ICLR 2026
PCPO: PROPORTIONATE CREDIT POLICY OPTIMIZA- TION FOR ALIGNING IMAGE GENERATION MODELS
Jeongjae Lee & Jong Chul Ye KAIST (jayle22000, jong. ye @kaist.ac..k
ABSTRACT
While reinforcement learning has advanced the alignment of text-to-image (T2I) models, state-of-the-art policy gradient methods are still hampered by training in- stability and high variance, hindering convergence speed and compromising image quality. Our analysis identifies a key cause of this instability: disproportionate credit assignment, in which the mathematical structure of the generative sampler produces volatile and non- proppoorrtii feedhack across timesteps. To address this, we introduce P Proportionate Credit Policy Optimization (PCPO), framework that enforces proportional credit assignment through stable objective reformulation and principled reweighting of timesteps. This correction stabilizes the training process, leading to significantly accelerated convergence and superior image quality. The improvement in quality is direct result of mitigating model collapse, com- mon failure mode in recursive training. PCPO substantially outperforms existing policy gradient baselines all fronts, including the state-of-the-art DanceGRPO. Code is available at wtttt::t//// ///::///it /ttpt//gggtbttt/////gg thub. com Seaylee2000/pcte
INTRODUCTION
Modern T2I generation, dominated by powerful diffusion and flow models (Esser
2024
Labs et al. 2025 Podell et al., 2023). still struggles to create outputs that consistently align with
human preferences (Google 2024). Group Relative Policy Optimization (GRPO), form of Re- inforcement Learning from Human Feedback (RLHF) highly successful in large language models
(LLMs) (DeepSeek-Al et al., 2025 Shao et al., 2024). has emerged as the state-of-the-at online policy gradient framework for aligning image generation models (He et al.) [2225] Liu et al. [025] Xue et al. 2025). Despite their success, GRPO methods often encounter training instability and model collapse, limiting their performance and reliability.
In this work, we found that these issues stem from two fundamental limitations that arise when applying policy gradients to generative samplers. First, the standard objective is susceptible to numerical precision errors that skew gradient magnitudes. Second, and more critically, the mathe- matical structure of these samplers leads to disproportionate credit ussignment. This manifests high-variance learning signal with volatile, non- non-proportional feedback across timesteps- a primary source of instability that highly detrimental to the training process.
To address this, we introduce Proportionate Credit Policy Optimization (PCPO), a framework that targets both limitations. PCPO first enhances numerical stability by reformulating the objective and, more importantly, ensures proportional credit assignment with principled reweighting schedule. These targeted modifications accelerate convergence and produce superior samples by mitigating model collapse.
Our work concurrent to several others aiming to improve alignment efficacy by addressing subopti- mal credit assignment. For instance, TempFlow-GPPe (He et al. (025) uses trajectory branching and MixGRPO (Li et al. 2025) employs a sliding SDE window to focus optimization on high-impact timesteps, primarily for training acceleration. While TempFlow-GRPO also proposes proportional reweighting scheme, our proportionality principle (eg. Proposition 2 offers a more fundamental explanation for these improvements, successfully accounting for cases where simpler, empirical heuristics fail. As such, PCPO stabilizes the training process, leading to significantly accelerated convergence and superior image quality, and mitigating mode collapse. Experimental results confirm

---

Published as a conference paper at ICLR 2026
DDPO
Ours
(8)
DanceGRPO
(b)
Figure 1: Qualitative comparison of baseline methods (top) and PCPO (bottom) on identical prompts and seeds. PCPO mitigates model collapse in baselines across different frameworks. (a) DDPO (SD1.5, Aesthetics): At matched reward level, PCPO preserves diversity and fidelity while DDPO collapses into blurry, homogenous style. (b) DanceGRPO (FLUX, HPSv2.1): After training for 200 epochs, PCPO achieves both higher reward and superior image quality, avoiding artifacts observed in the baseline.
that PCPO substantially outperforms existing policy gradient baselines on all fronts, including the state-of-the-ar DanceGRPO.
Related Work. Aligning LLMs with human preferences is predominantly achieved through RLHF [Christiano al. (017). Early methods popularized Proximal Policy Optimization (PPO) for this purpose (Ouyang et al. 2022 Schulman et al. 2017). Subsequently, Direct Preference Optimization (DPO) emerged as simpler, reward-free alternative that gained widespread adoption by reframing alignment supervised learning problem on pairwise preferences (Rafailov et al.) 2023). However, recent advancements have demonstrated the superior performance of volicy-gradiem methods: notably, GRPO has surpassed previous techniques complex reasoning tasks, establishing a new of-te-of-the-ar (DeepSeek-AM AT et al. 2025 Shao et al.] 2024).
The evolution of preference alignment in T2I models has mirrored the trends in LLMs. Initial efforts adapted PPO to the diffusion process but were often plagued by training instability, limiting their
scope to constrained vocabularies (Black al. 2024 Fan al. 2023). The subsequent adaptation
2

---

Published as a conference paper at ICLR 2026
of DPO improved stability and broadened vocabulary coverage (Wallace al. 2024 Yang et al.
2024).
Most recently, GRPO-based frameworks have achieved state-of-the-arr performance (Liu et al.
2025
Xue et al. 2025). This progression was theoretically anticipated: as the optimal policy of DPO
is upper-bounded by that of policy-gradient methods (Xu et al. 2024). stabilizing policy-grade seiccggrrddeet training was expected yield superior results.
primary obstacle to enhancing stability and performance of policy-gree aaiiyggr training is model collapse (Shumailov al., 2024). a degenerative process where model trained recursively on its own outputs progressively degrades. In the context of online RL for T2I models, we observe this phenomenon manifesting in two key failure modes. The first is classic mode collapse, a loss of sample diversity that is also well-documented in LLM alignment, where the policy's entropy is exhausted in pursuit of high rewards (Cui et al. 2025 Park al., (025). The second is image quality degradation, form of reward hacking where the model over-optimizes for the reward signal (e.g., aesthetic score) the expense of general fidelity, producing artifacts and unrealistic outputs (Wang & Yu 2025). In this work, we use "model collapse" the umbrella term to refer to this overall process where both sample diversity and fidelity are compromised.
2 PCPO: PROPORTIONATE CREDIT ASSIGNMENT POLICY OPTIMIZATION
2.1 PRELIMINARIES
Diffusion and Flow Matching. Conditional diffusion probabilistic models (Ho et al. 2020) learn to create data by reversing Markovian forward process that gradually adds Gaussian noise to a clean sample Xg. The model, co(x,, typically by minimizing a weighted meean-squrred error objective. Flow matching models (Lipman cct al. 2023) simplify this process by learning the velocity us (x,, t, c). which is typically applied to follow the straight-line path between Gaussian noise and the data sample (Liu et al. 2023). This allows for efficient generation via solving a deterministic ordinary differential equation (ODE).
trained to predict the noise added at an intermediate state Xt-
Policy Gradient Alignment. We frame the image generation process Markov Decision Process (MDP) (Bellman (957)) following the formulation of Black et al. (2024). The T-step reverse process prompt A terminal reward r rxxo: c) is assigned at the final step. To improve sample efficiency, PPO performs multiple
is defined by states 8t (,) and actions 0r X conditioned
optimization steps on trajectories from an older policy Te... clipping the importance sampling ratio
(0) = Pg /Pem to stabilize updates (Fan et al. 2023
Schulman et al. 2017):
T
(P)O(() == -Eropoas max (-prA, -clipp
where A is the normalized terminal reward and clip. == clip(p:, 1 1,, E) for clipping threshold g. The state-of-the-art GRPO (Shao al. 2024 employs an analogous objective, enhancing stability by performing group-relative reward normalization calculate the advantage, i.c. Á' (r* PG)) /aa/aa. from a group of G samples. Since our contribution, PCPO, focuses on modifying the policy ratio At and its underlying credit sesinmmnnt-mechanisee proceed using the simpler PPO notation for our derivation. We omit the KL penalty term for simplicity, following prior work (Black et al. 2024 Xue et al. (025).
common to both frameworks- -e
2.2 PCPO DERIVATION
PCPO for Diffusion Models. Following Huang et al. (2024), we note that the gradient of Eq. 11 equivalent to that of hinge loss,
Lringe
E
max max [|A - A(p: - 1)}].
We stabilize this objective by replacing the numerically unstable term - with the more robust log Pt. This choice is justified in two ways. First, under the hinge loss interpretation (Huang et al. 2024], this term acts as a swappable "classifier," allowing us to substitute different functions while maintaining the core mechanism. Second, it is sound Taylor approximation for small policy updates
3

---

Published as a conference paper at ICLR 2026
(log Pt = Pc-(), 1). a condition enforced by the small clipping range in our experiments; empirically confirmed that this approximation error never exceeded 1.2% during training. More justification is provided in Appendix D This leads to our stable log- wiingegg objective:
T
LPCPO-base (0) :- E
max(0, E|A| - A log
The core issue of disproportionate credit, however, lies within the log Pr term itself. We decompose this term in the following proposition, with the full derivation provided in Appendix(A.I
Proposition 1. For DDIM sampling schedule, the log policy ratio log Pt. is given by:
C(t) w(()
log Pt
())()e))) (ldd) cold
(e) + =)(l((()(e)))
(4)
ot
where
VI- V 11 ā o3 0.
C(t)
a
Here, é denotes the denoiser's noise prediction | whereas cold is the Gaussian noise sampled during reverse sampling under the old policy. Substituting Eq. (4) into Eq. (3) transforms the PPO objective into an equivalent e-matching loss
E--matching (0) =
IE
max 0, E|A|+ + AD(w(t),
_e(0) old) }
(5)
where
D(w(t), e( e(()) E (00) == (t) - -((() - old + {()(((t)())))))))))))))))
(c)
=((i) 11?
(6)
This decomposition reveals that the gradient contribution of each timestep is scaled by native weight w(t) that is highly non- non-uniform, spanning orders of magnitude (Figure 2b). This variance is primary source of training instability, as both causes gradients from different timesteps to be scaled inconsistently and leads to the most amplified gradients being clipped disproportionnaely often.
We argue that for proper credit assignment, these weights should be uniform. This principle is justified by direct analogy to the foundational REINFORCE policy gradient algorithm (Sutton & Barto 2018: 8: Williams 1992). that framework, parameter updates are proportional to the eligibility vector (the policy gradient term), which is scaled by each action's 's contribution (often assumed to be uniform). Our analysis (see Appendix E) shows that the diffusion sampler's 's gradient formulation is analogous, but with critical distinction: it scales this "cligibility vector" by non-uniform, arbitrary weight wt)). This (t) is an artifact of the sampler's 's mathematics, not deliberate credit assignment strategy. This native scaling introduces high variance by weighting the credit A based the noise schedule rather than the step's actual importance.
PCPO restores credit assignment proportional the integration interval by re-engineering the DDIM variance schedule, o,, to produce a constant weight, w(() = w², for all timesteps. To do this, we use the definition of the weight from Proposition 1 w()) C())/a,- For each timestep 4, we the weight on the left-hand side to our target constant w". With the standard DDIM schedule (ax) fixed on the right- side, the only free parameter remaining in the equation is the variance o,. We can therefore solve for the precise value of at each step that yields our desired constant weight. To ensure fair comparison and isolate the effect of this uniform weighting, we rescale wn to match the mean of the original, non-uniform weights (see Figure a b)).
PCPO for Flow Models. Applying policy gradients to flow models requires introducing stochasticity via reverse SDE with the same marginal probability densities the original ODE (Liu et al, 2025 Xue et al. 2025):
1 0 (x.)) + a.dw.
dx, (u;
2
Unless otherwise specified, is shorthand notation for noise prediction with classifier-free guidance Ho & Salimans 2021] (1 (
(e; 0) wcrcē(:;c)
4

---

Published as a conference paper at ICLR 2026
(a)
(b)
(c)
(d)
5
0.5
2,5
2
wi
x
0.0
0.0
0
Figure 2: Weight rescaling by PCPO. DDIM Sampler: (a) Volatile native weights w(t) (blue) are replaced with uniform, rescaled weight (orange). (b) This achieved by computing a new variance signal oN that remains close to the original (corresponding to w* = 4.5 (olive)), then rescaling. Light blue corresponds to w = 4.0, gray to w" 5.0. SDE Sampler: Native (blue) and rescaled (orange) weights for (c) DanceGRPO SDE, (d) Flow-GRPO SDE.
Integrating this SDE using first-order Murer--aruyama discretization allows for trajectory sampling, where the log policy ratio for single step can be simplified form analogous to Eq.
(tz) log Pt. (tt)) (us uala) old + "
u I?]
w(tt)
V (1
(1
41)0f
o
2t,
(8)
The challenge of disproportionate credit is particularly complex in modern flo-mmtchhig models. We illustrate this issue using the DanceGRPO SDE, where ot, is constant, 7) (Xue et al.) (025). For high-resolution synthesis, the timestep shifting technique (Esser al. 2024) creates non-uniform integration intervals At,, making native weights highly non-proportional to the interval length (wo(t) 2 Vt).). While we restored proportionality in diffusion models with minor, non-degrading adjustment to its variance schedule, an analogous modification for flow models requires drastic changes to the variance schedule or the timestep shifting strategy. Both options are problematic, as significantly deviates from the original, well-optimized sampling procedure (Esser et al.) 2024 Xue jet al., 2025). Therefore, PCPO takes a different approach for flow models: it enforces proportionality by directly reweighting the training objective, defined in the following proposition.
For flow matching SDE in the form of Eq. 77) the weight schedule w(t,) that ensures credit is proportional to the integration interval At, is given by:
Proposition 2.
N
(1 t
w(t:) = CAtis
Tt
24
The proof for Proposition 2 is in Appendix A.3
Figure c d) compares the default (vanilla) and improved (proportional) timestep weights for both the DanceGRPO and Flow-GRPO SDEs. The plots are generated using hyperparameters from the original works: N 16, 17 0.3 for DanceGRPO (Xue et al. 2025). and N = 10. 0.7 for Flow-GRPO (Liu et al. 2025). Corresponding values of ( are 13.343 and 4.315, respectively. For follow its official implementation and approximate the final weight wN (at 1) avoid a divide-by-zers error the variance schedule, o; = ///t/(1-t)
the Flow-GRPO visualization,
EXPERIMENTS
3.1
METHODS
Our main analysis focuses on applying PCPO to two policy gradient frameworks: DDPO (Black ctt al. 2024) on Stable Diffusion 1.5 (SD1.5; Rombach et al. (2022)), and the state-of-the-art Dance- GRPO (Xue al. 2025) on both Stable Diffusion 1.4 (SD1.4 .4) and the FLUX. (FLUX) flow- matching model (Labs) 2024). We train DDPO on two reward models: Aesthetics (Schuhmann 2022) and BERTScore (Zhang* et al. 2020), and DanceGRPO on 1PSv2.1 (Wu et al.] 2023). Our evaluation twofold: we first analyze training dynamics by tracking reward acceleration and clipping fractions throughout the learning trajectory. We then assess sample quality at matched reward levels using
5

---

Published as a conference paper at ICLR 2026
(a)
x x10-2
(b)
x10-2
(c)
(d)
7.82
0.36
0.40
0.24 0.37
0.48
Reward
Clipfrac
0.00
0.00
Epoch
200
Epoch
200
Epoch
300
Epoch
200
Baseline
PCPO
Figure 3: Reward and clipping fraction traces for PCPO (orange) vs. baselines (blue): (a) DDPO, Aesthetics, (b) DDPO, BERTScore, (c) DanceGRPO (SD1.4), HPSv2.1, (d) DanceGRPO (FLUX), HPSv2.1.
Table Training Efficiency of Baseline vs. PCPO. Speedup epochs translate directly to wall-clock time savings (see Figure
Baseline
Reward
Target Level
EpochS
EpochSycro
Speedup
DDPO
Aesthetics
6,90
147
118
24.6%
DDPO
BERTScore
0.52
191
146
30.8%
DanceGRPO (SD1.4)
HPS
0.370
236
188
25.5%
DanceGRPO (FLUX)
0.360
209
41.2%
Table 2: Raw scores for (a) DDPO (Aesthetics) and (b) DanceGRPO (HPS) experiments. PCPO was
put disadvantage for the DanceGRPO evaluation. Statistically significant differences in bold.
(a)
(b)
Batch
Method
FID
FDDINO
18*
LPIPS
Model
Method
FID
FDNIN
IS*
I.PIPS
Baseline
31,72
473.17
26.35
0,6208
Baseline
90.34
1078.42
7.61
0.4948
256
SD1.4
PCPO
27.86
461.69
24.12
0.6262
PCPO
84.74
1035.45
7.50
0.4894
Baseline
24.09
451.19
25.67
0.6321
Baseline
46.23
$39.83
12.66
0.5736
512
PCPO
22.06
391.30
25.65
0.6525
FLUX
PCPO
40.38
438.88
11.90
0.5708
Fréchet Inception Distance (FID) (Heusel al., 2018 Fréchet Distance DINOv2 (FDDiNo) (Stein
et al. 2023), Inception Score (IS) (Salimans et al. 2016). and LPIPS Diversity (Zhang et al. (0188) 11
To account for per- prompt variance in these quality metrics, we validate our findings using Linear Mixed Model (LMM). All main results are from experiments using main configurations from Table Appendix B with the exception of the half-sized batch configurations used for LMM analysis. All qualitative results are also generated from experiments using main configurations, with the exception of Figure (b).
To further validate PCPO's robustness, we benchmark performance unseen prompts from the MSCOCO-2017 validation (5K) (Lin et al. 2014) and MJHQ-305 33K (Li al. 2024) datasets, evaluating
on diverse suite of alignment metrics: 1,Sv2.1, Aesthetics, CLIPScore (Hessel et al. 202 I). PickScore (Kirstain et al. 2023), and ImageReward (Xu et al. 2023). Next, we test PCPO's generalizability by applying it to the Flow-GRPO SDE framework (Liu al. 2025) the SD3.5- M model (Esser et al. 2024)-a substantially different architecture and training setup that uses different rewards (OCR (Chen et al. (2023). PickScore) and auxiliary KI. divergence penalty. Full experimental details are provided in Appendix
3.2 RESULTS
PCPO Improves Training Efficiency and Stability, PCPO's principle of proportionate credit assignment translates directly to enhanced training stability. As shown in Figure PCPO consistently maintains lower and more stable clipping fraction than the baselines. This stability is the key to its faster convergence, leading to substantial training acceleration across all experimental settings (Table D.. A detailed breakdown of speedups is available in Table 10 in Appendix
PCPO Mitigates Model Collapse. The stability from PCPO translates to significant improvements in fidelity and diversity. Specifically, for the DDPO experiments, PCPO achieves statistically

---

Published as a conference paper at ICLR 2026
significant improvement in sample fidelity (FID). While FDDINo showed similar trends, the effect was not statistically significant (p = 0.247). We attribute this lack of significance to limitations of the base model or task setup, especially since the FDDIO metric also failed to register significant effect for batch size (p = 0.468). This suggests the metric may be insensitive to improvements within this specific experimental context, possibly due simplicity of prompts or limitations of the base model's capacity.
The effect on LPIPS diversity is more nuanced: while the main effect of PCPO alone was not statistically significant, the LMM analysis re- veals strong, positive interaction between the algorithm and batch size (3m 0.016, P = 0.008). This indicates PCPO's benefit to di- versity becomes prominent when synergizing with larger batch sizes, which on their own were found to significantly increase diversity (Soach 0.010. p 0.016).
Table 3: LMM analysis for DDPO (Aesthetics).
Effect
Metric
Coeff. (3)
p-value
FID
-7.750
0.047
FDDNo
-52831
0.247
Algorithm
IS*
-0.241
0.021
LPIPS
0.004
0.401
FID
-135509
0.001
FD
-66.811
0,468
Batch Size
IS*
-0,266
0.011
The most compelling evidence for PCPO's role in mitigating model collapse, however, comes from the Inception Score (IS) analysis. We acknowledge the common interpretation that, given similar FID, a higher IS can indicate bet- ter quality. However, this metric's behavior can be task-dependent, and it is known that non-diverse models can "artificially achieve high IS" (Sadal al.] (024). To determine the correct interpretation for our specific task, we first established an empirical ground truth. We found that increasing the batch size- known technique to reduce model collapse by preserving data diversity (Shumailov et al. 2024) causes a statistically significant decrease in IS (Bbatch = 0.2666, p 0.011).
LPIPS
0.010
0.016
FID
4.053
0,463
FDINO
-55.720
0.222
Interaction
IS*
0.181
0.221
LPIPS
0.016
0.008
This finding strongly suggests that, in this con-
Table 4: LMM analysis for DanceGRPO (HPS).
text, high IS is not an indicator of quality but
rather a pathological artifact of mode collapse, rewarding low- diweriiersity. high-confidence out- puts. We therefore treat IS for our task as a metric be minimized, given lower parable FID. With this understanding, PCPO's statistically significant reduction in IS (Batg -0.241, 0.021) is not a sign of degradation, but strong evidence that achieves a desirable stabilizing effect, similar to using larger batch.
Model
Metric
Coeff. (3)
p-value
FID (1)
-6.730
0.026
FDDINO (L)
-64.482
0.128
SDL44
IS (L)
-0.029
0.499
L.IPS (1)
-0.005
0.102
FID (4)
-13.884
0.001
FDDINo (L)
-175.592
0.001
FLUX
IS (L)
-0.184
0.014
LPIPS (1)
-0.002
0.629
PCPO's fidelity improvements (FID, FDDINO)
also hold in the DanceGRPO experiments. This is particularly notable because the comparison was handicapped; due to noisy reward trajectories, we evaluated PCPO at a higher-reward checkpoint than the baseline, a state that typically harms FID. Nonetheless, PCPO still delivered significantly better FID for both models. This result, along with a lower IS for the FLUX model, reinforces that PCPO effectively mitigates model collapse. Conversely, no significant effect on LPIPS was observed for the DanceGRPO experiments. We hypothesize this is due to two confounding factors: the batch sizes may have been too small to activate the synergistic effect, and the comparison made at a checkpoint where PCPO had already achieved a higher reward, potentially masking underlying diversity improvements.
Qualitatively, PCPO avoids the severe model collapse that affects the baselines, producing clear and diverse images instead of the blurry, repetitive outputs characteristic of unstable training (Figure []. Further comparisons showing PCPO's superior visual fidelity throughout training are available in Appendix H To validate these gains with human perception, we conducted a formal preference study. To ensure fair comparison, we evaluated PCPO (epoch 120) against the baseline at two beward-bracketin checkpoints (epochs 180 and 240). The results were decisive: human evaluators robustly preferred PCPO's is outputs over the DanceGRPO baseline in all categories (Figure This
7

---

Published as conference paper at ICLR 2026
(a)
(b)
General
36
10
53
Preference
Visual
33
12
54
33
17
50
Quality
Text-Image
27
30
4.3
28
26
44
Alignment
60
40
20
0
20
40
60
60
40
20
20
40
60
Preference (%)
Preference (%)
Figure 4: Human evaluation results for DanceGRPO (blue) vs. PCPO (orange) on FLUX. To ensure a fair comparison that accounts for faster convergence, our model (120 epochs) was evaluated against the baseline at two reward-ber -araakb-bbckiingggggg checkpoints: (a) 180 epochs and (b) 240 epochs.
Table 5: Human preference alignment metrics on 5K unseen prompts from (a) MSCOCO-2017 Validation set, (b) MJHQ-30K, PCPO outperforms DanceGRPO on the trained reward (HPSv2.1) and unseen metrics, even an earlier epoch (FLUX), confirming less reward hacking.
Model
Method (Epoch)
HPSv2.1
Aesthetic
CLIPScore
PickScore
ImgRwd
(a) MSCOC0-2017 Val
Base Model DanceGRPO (E200)
0,252
5.18
0.365
21.53
0.17
SD1.4
0.317
5.69
0.357
22.13
0.87
PCPO (E200)
0.326
5.69
0.365
22.26
0,90
Base Model
0.293
5.70
0.382
22.94
0,95
FLUX
DanceGRPO (E180)
0.330
6.15
0.373
23.08
1.07
PCPO (E120)
0.337
6.21
0.369
23.12
1.14
(b) MJHQ
Base Model DanceGRPO (E200)
0.247
5.56
0.370
19.84
0.17
SD1.4
0.347
6.14
0.374
20.95
1.12
PCPO (E200)
0.353
6.18
0.379
21.14
1.16
Base Model
0,306
6.37
0.398
21.96
1.14
FLUX
DanceGRPO (E180)
0.345
6.56
0.387
22.14
1.27
PCPO (E120)
0.350
6.56
0.398
22.28
1.32
preference was strong even on mobile devices where the baseline's subtle artifacts (Figure 20 were less visible, suggesting the true performance gap may be even wider.
Evaluation with Unseen Prompts on Diverse Reward Metrics. We evaluated checkpoints at matched reward levels from
IPPSv2.1-trainee DanceGRPO experiments. We test generalization on simpler prompts from the MSCOCO-2017 validation set and more complex prompts from the MJHQ-30K dataset. The results, presented in Table b), demonstrate that PCPO not only excels on the metric was trained on (HPSv2. 1) but also outperforms the baseline across a wide suite of metrics. For SD1.4, ., PCPO outperforms DanceGRPO across all metrics at an equivalent training point (200 epochs). For the FLUX model, PCPO at 120 epochs outperforms the DanceGRPO baseline at 180 epochs. This confirms that PCPO converges substantially faster more generalizable policy. Notably, PCPO maintains text-image alignment (CLIPScore) better than the baseline, suggesting it is less prone reward hacking.
Generalization. To test generalizability, we applied PCPO in significantly different training regime using the SD3.5-M model with the Flow-GRPO framework. This setup involved a distinct noise schedule, different rewards (OCR, PickScore), auxiliary KL penalty, and separate prompt dataset (see Appendix B). Despite these substantial changes, PCPO's benefits remained clear. As shown in Figure 5| PCPO consistently outperformed the baseline, achieving a higher reward and lower (better) KL divergence, while maintaining significantly less clipping. This confirms PCPO's s stabilizing properties robust across different models, noise schedules, and training configurations.
8

---

Published as a conference paper at ICLR 2026
(a)
(b)
(c)
x10
(d)
x10
0.85
0.30
22.80
0.30
0.0
Eval OCR
Clipfrac
Eval PS
Clipfrac
-KL
-KL
0.57
0.00
21:50
0.00
-2.
-1.5
250
0
250
0.57
0.85
21.63
22.80
Epoch
Epoch
Eval OCR
Eval PS
Flow-GRPO
PCPO
Figure 5: PCPO demonstrates robust generalizability on SD3.5-M using the Flow-GRPO framework. PCPO (orange) consistently outperforms the baseline (blue), achieving higher reward, lower KL divergence, and a lower clipping fraction. Plots explained in depth in Appendix B.3.4
3.3 ABLATION STUDY
PCPO Component Analysis. Our ablation study dissects the contribution of each PCPO component by sequentially adding them to the DDPO (Aesthetics) baseline. 6 shows this progression from DDPO (blue), through the log Pr objective (green) and e-matching (red). to our final proportional weighting
a steady reduction in clipping. While each component contributes to stability, only the full PCPO framework achieves zero on-policy clipping ratio, theoretical result confirming its superior design. This final step of adding proportional weighting was also the most critical factor for accelerating training.
PCPO vs. Heuristic Acceleration Methods. To validate our principled approach, we con- trast PCPO against two heuristic acceleration strategies. First, we compare against rimestep subsampling, heuristic where policy updates use only 50% of timesteps. This reduces wall- clock time per epoch to just 58% of the standard training, but as Figure 7b) shows, the speedup comes at significant cost to final image quality,
(a)
(b)
x10
Aesthetic
Clipfrac
200
20
Epach
Step
DDPO
log(s)
PCPO
Figure 6: Ablation study of sequentially adding PCPO's components. Results show rewards and clipping fractions over (a) 200 epochs, (b) close- up of the initial 20 steps (unsmoothed).
Second, we contrast our proportionality prin- ciple with a heuristic inspired by the empiri- cal observations of concurrent work (He et al.) 2025): prioritizing high-noise timesteps due to their higher reward variance. The DanceGRPO SDE provides a decisive test case, as its native weights (X vDt.) already disproportionately favor the short, high-noise integration steps, unlike the Flow-GRPO SDE used by TempFlow-GRPP GRRO FFigure2ic. d)). The empirical heuristic would suggest amplifying these weights even further. In direct opposition, our proportionality principle requires reweighting in the opposite direction. Our main experiments confirm that applying our principle remarkably accelerates training. Conversely, further emphasizing high-noise steps with uniform weighting schedule ultimately harms performance compared to the baseline (Figure d)). Together, these results highlight the advantage of our prineipled approach: PCPO consistently achieves signifi- cant training acceleration without the degradation in sample quality that often accompanies simple heuristic methods.
DISCUSSION
Our results strongly suggest that PCPO effectively mitigates model collapse. This phenomenon, where models degrade when trained on recursively generated data, arises from several error sources accumulating over time. The primary driver is the statistical approximation error from using finite training samples, which causes progressive loss of the data distribution's tails (Shumailov et al.) 2024). Standard policy gradient methods aggravate this issue by aggressively clipping and discarding these crucial tail-end samples. While the most direct method to reduce this specific error is to increase the batch size, this approach incurs significant computational overhead and does not address other contributing factors, such as numerical precision errors, which also implicated in model collapse.

---

Published as a conference paper at ICLR 2026
x x10-2
(b)
(c)
(d)
7.14
Clipfrac 8
2.5
0.37
0.48
Aesthetic
Clipfrac
HPS
5.50
0.0
0.29
0.00
Epoch 200
Epoch
190
Baseline
PCPO
Heuristic
Figure 7: (a) The timestep subsampling method (purple) yields similar reward gain per epoch, which translates to significant reduction in wall-clock time per reward gain (58% of the DDPO baseline, blue). (b) But this speedup degrades sample quality and diversity, shown here for the prompt "fish" using 4 consecutive seeds at matched reward level (Aesthetics 6.90). Top row: DDPO, bottom (c) Purple: A uniform reweighting scheme that emphasizes timesteps. (d) The uniform scheme shows marginal initial speedup but is ultimately outperformed by the vanilla & VAi, schedule of DanceGRPO (blue).
row: DDPO + timestep subsampling.
high-noise
PCPO offers a more comprehensive and efficient solution by targeting multiple error sources. It directly counters the statistical error by preserving more tail-end data through drastically reduced clipping. Simultaneously, its log formulation addresses the secondary issue of numerical precision errors. Our LMM analysis confirms that the combined result of these targeted corrections is stabilizing effect paralleling that of doubling the batch size (Table PCPO therefore provides the benefits of larger batch training without the associated computational overhead, offering an efficient, multi-faceted defense against model collapse.
5 CONCLUSION AND FUTURE WORK
In this work, we addressed the critical instability in policy-gradient based alignment for T2I models, identifying the root cause disproportionale credit assignment. Our proposed framework, Propor- tionate Credit Policy Optimization (PCPO), corrects this fundamental flaw by ensuring the feedback signal from each timestep is proportional to contribution. This principled correction was shown to accelerate convergence and improve sample quality, achieving state-of-the oftt-offthh-art performance that surpasses strong baselines, including DanceGRPO.
This work's ' focus on providing a stable, proportional feedback signal for alignment opens several exciting avenues for future research. A promising direction is to explore the synergy between PCPO's ' credit proportionality and other stabilization techniques, such as dynamic clipping. temporal localization, and KL regularization. While PCPO reduces the need for PPO and GRPO's internal timestep clipping, the training of flow models often still relies on downstream gradient clipping to prevent divergence. Interestingly, aggressive gradient clipping has been observed to destabilize training (Xue & xingzhejun 2025). Therefore, a deeper investigation into the underlying mechanics that produce these large, unstable gradients presents fruitful research direction, potentially leading to alignment methods that stable by design.
particularly interesting avenue relates to gradient clipping.
Reproducibility Statement. For every proposition and mathematical derivation, we provide proofs in Appendix A Hyperparameters and resources required to reproduce experiments are stated in Appendix B Code is available at https: //t:://itttt github........o com/ jay ee2000/ pcpp//
Acknowledgementt This research was supported by the AI Computing Infrastructure Enhance- ment (GPU Rental Support) User Support Program funded by the Ministry of Science and ICT (MSIT), Republic of Korea (RQT-25-120217) This work was supported by the National Research Foundation of Korea under Grant RS 20344444433,444 This work was supported by the Institute of Information & Communications Technology Planning & (nliiuatiionIIIPP) grant funded by the Korea (MVT))
Star Fellowship(KAISST), This work was supported by Institute for Information & communications Technology Planning & Exaluation(IITP)) grant funded by the Korea governmen((MST)) 11910059 Artificial Intelligence Graduate School Program(KAIST)
005-222550030499
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang. Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025. URL
https://arxtie obg 2202 13923
Richard Bellman. markovian decision process. Journal of Mathematics and Mechanics, 6(5): 679-684, 1957.
Kevin Black, Michael Janner, Yilun Du, Ily Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024.
net/ Ltt.//ooooomm./CCcCCCC//
Jingye Chen, Yupan Huang, Tengchao Lv, Lei Cui, Qifeng Chen, and Furu Wei. Textdiffuser: diffusion models text painters. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023. Curran Associates Inc.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL [attps::/ www.:ugs..euuu.r...oo......... wttpouriips.cpppppppeeeeeepppeee
cc/paper miles/paperr2011 pdf
Hyungjin Chung, Jeongsol Kim, Geon Yeong Park, Hyelin Nam, and Jong Chul Ye. CFG++: Manifold-constrained classifier free guidance for diffusion models. In The Thirteenth International Conference on Learning Representations, 2025. URU/attpss//pppeeeeiies htt////pppeeviiie/e net forum?
id=E77uvbOTtp
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, et al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.22611 2025.
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang. Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue. Bingxuan Wang, Bochao Wu, Bei Feng. Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang. Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. Cai, Jiaqi Ni, Jian Liang. Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang, Liang Zhao, Litong Wang. Liyue Zhang. Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li. Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang. R. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei. Tianyu Sun, T. Wang, Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, W. L. Xiao, Wei An. Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang. Xinyuan Li. Xuecheng Su, Xuheng Lin. X. Q. Li. Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang. Xinnan Song, Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang, Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma. Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li. Yi Zheng, Yuchen Zhu, Yunxian Ma. Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu WU, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li,
11

---

Published as a conference paper at ICLR 2026
Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepscek-rl: Incentivizing reasoning capability in llms via reinforcement learning, 2025.
/tttps:///axxii wttpxxiw.rr//a////1...oo 12948
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, al. Scaling rectified flow transformers for high-resolution image synthesis. arXiv preprint arXiv:2403.03206 2024. URL https/
//
03206
Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Reinforcement learning for fine-tuning text-to-image diffusion models. In seirth-seventh Conference on Neural Information Processing
Systems, 2023. URL httpt:://///..... httpttt://opeeeeeieeeeee wtt.:eww.eeefotoooeeoeeoe wtw.ooooommnnoooeeeeee d
Google. Gemini image generation got it wrong. we'll do better., 2024. URL https //blog. gwogle cessed: 2025-05-21.
image generation issue/ Ac-
Xiaoxuan He, Siming Fu, Yuke Zhao, Wanli Li, Jian Yang, Dacheng Yin, Fengyun Rao, and Bo Zhang. Tempflow-grpo: When timing matters for grpo in flow models, 2025, URL https
04324
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: reference-free evaluation metric for image captioning. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7514-7528, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/202/ emnlp mmnlp-main.595. URL httts::/aacaatthlllllaooooo org/2021 www.ememmmmmmmmm.mmmmmmmmmmmmmmm 595/
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge local nash equilibrium, 2018. URL
https://arxxi http://aaxxvvvoooo org/ http:///ss/////.../../ 08500
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NewrlPS 202/ Workshop Deep Generative Models and Downstream Applications, 2021. URL /ttpps://pperreeiee
net/ www.enoiduuwwwaee.....
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, volume 33, pp. 6840-6851, 2020.
Nai-Chieh Huang, Ping-Chun Hsieh, Kuo-Hao Ho, and I-Chen Wu. Ppo-clip attains global optimality: Towards deeper understandings of clipping. In AAAI, pp. 16600-1607, 2024.
Luozhijie Jin, Zijie Qiu, Jie Liu, Zijie Diao, Lifeng Qiao, Ning Ding, Alex Lamb, and Xipeng Qiu. Inference-time time aligmment control for diffusion models with reinforcement learning guidance, 2025.
URL http::tts://// /wwp///rxvvvvv/am org/ /508 21016
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a- pic: an open dataset of user preferences for text-to-image generation. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY,
USA, 2023. Curran Associates Inc.
Black Forest Labs. Flux.
forest labs/flux 2024.
Black Forest Labs, Stephen Batifol, Andreas Blattmann, Frederic Boesel, Saksham Consul, Cyril Diagne, Tim Dockhorn, Jack English, Zion English, Patrick Esser, Sumith Kulal, Kyle Lacey, Yam Levi, Cheng Li, Dominik Lorenz, Jonas Müller, Dustin Podell, Robin Rombach, Harry Saini, Axel Sauer, and Luke Smith. Flux.1 kontext: Flow matching for in-context image generation and editing in latent space, 2025. BRLLhttp::////
org/ abs /2006 15742
Daiqing Li, Aleks Kamko, Ehsan Akhgari, Ali Sabet, Linmiao Xu, and Suhail Doshi. Playground v2.5: Three insights towards enhancing aesthetic quality in text-to-image generation, 2024.
12

---

Published as a conference paper at ICLR 2026
Junzhe Li, Yutao Cui, Tao Huang, Yinping Ma, Chun Fan, Miles Yang, and Zhao Zhong. Mixgrpo: Unlocking flow-basee grpo efficiency with mixed ode-sde, 2025. URL https:://axxiv..tt
abs/2500 21802
Zhanhao Liang, Yuhui Yuan, Shuyang Gu, Bohan Chen, Tiankai Hang, Mingxi Cheng, Ji Li, and Liang Zheng. Aesthetic post-training diffusion models from generic preferences with step-by- step preference optimization. In 2025 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 15199-13200 2025. doi: 10. 00.1099CVPP327442225003333
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In David Fleet, Tomas Pajdla, Bert Schiele, and Tinne Tuytelaars (eds.), Computer Vision ECCV 2014, pp. 740-755, Cham, 2014. Springer International Publishing. ISBN 978-3-319-10602-1
Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Repre- sentations, 2023. URL http::::///peeeeeeeeeee net/ PeeeruuiidPPqvMDDD..
Jie Liu, Gongye I. Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching models via online rl, 2025. URL.
https: http://axxvv.rrro wttp:r/g/a..///////////////////e 2505.. 05470
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. International Conference on Learning Representations (ICLR), 2023. URL https: http://oppveeiieeeeee ew. net/forun's 1==VVjTT 1nw5z
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Jaesung R. Park, Junsu Kim, Gyeongman Kim, Jinyoung Jo, Sean Choi, Jaewoong Cho, and Ernest Ryu. Clip-low increases entropy and clip-high decreases entropy in reinforcement learning of large language models, 2025. URL http::: wttp://v.rrr/ab///eeeeee
org. htlp//////////
26114
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis, 2023. htthttts::/ pttttpp:://riiivi arxiv. www.wi.wwwww.a....a..eooeee 2307. 952
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https
// wttp://ggbbb.....oaaaa/// 18290
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Seyedmorteza Sadat, Jakob Buhmann, Derek Bradley, Otmar Hilliges, and Romann M. Weber. CADS: Unleashing the diversity of diffuuson models through condition-annealed sampling. In The Twelfth International Conference on Learning Representations, 2024. U http::ttp///// //opnrevi/peereiee
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, and Xi Chen. Improved techniques for training gans. D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29. Cur- ran Associates, Inc., 2016. htap://66..aaaaaaaaa222222226661
Chrisoph Schuhmann. Laion aesthetics, Aug 2022.
URL https://laion.ai///
seionnaesthetics///
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL htt.://se//arc.....o//////////./
13

---

Published as a conference paper at ICLR 2026
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. 2024.
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson, and Yarin Gal. Ai models collapse when trained recursively generated data. Nature, 63 631(8022)7555755)) jul 2024. ISSN 1476-4687, doi: 10. 10.008844156--02--0766-6-
URL
001586-024-07555
George Stein, Jesse Cresswell, Rasa Hosseinzadeh, Yi Sui, Brendan Ross, Valentin Ville- croze, Zhaoyan Liu, Anthony I. Caterini, Eric Taylor, and Gabriel Loaiza-Ganem. Expos- ing flaws of generative model evaluation metries and their unfair treatment of diffusion mod- cls. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Ad- vances in Neural Information Processing Systems, volume 36, pp. 3732-3784. Curran Asso- ciates, Inc., 2023. URL URL.httpp:://prrceedd.gg. neurips cc/ wappaerrpppppppp.pepeeeeeeee...e files/paper/
Haoyuan Sun, Bin Liang. Bo Xia, Jiaqi Wu, Yifei Zhao, Kai Qin, Yongzhe Chang, and Xueqian Wang. Diffusionn-ainbooPA:: Improvements integrated preference alignment for diffusion-basse text-to-image generation. Transactions on Machine Learning Research, 2025. ISSN 2835-8856.
URL
net. http://ww..../..ooooocccoo//SSS
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018 URL http:://incommpleeiddesss. wtw.:/idea...etbooooo/ttttt -ooe/kteeee-bekk- 2nd
html
Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 8228-8238, June 2024.
Feng Wang and Zihao Yu. Coefficients--rrssevinng sampling for reinforcement learning with flow matching, 2025. URL http:/ss/acxx...oo////ee///
05952
R. J. Williams. Simple statistical gralient algorithms for connectionist reinforcement learning. Machine Learning, 8:229-256. 1992.
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Lai. Human preference score v2: solid benchmark for evaluating human preferences of text-to-image synthesis, 2023. JRLhttps: arxiv. org
absp//b//////eeee/e/gggg/.aaeeba abs 200.2320660990 09341
Jaazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: learning and evaluating human preferences for text-to-image generation. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pp. 15903-15935, 2023.
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is DPO superior to PPO for LLM alignment? a comprehensive study. In Forty-first International Conference on Machine Learning, 2024. htttp:t
ew, n
Zeyue Xue and xingzhejun. Please tell me why the grad_ suddenly explodedd during training, causing the reward to drop.,
htg.lttpp.///itnb...oo/////ee/pp Accessed: 2025-09-24.
1ssues/55]
Zeyue Xue, Jie Wu, Yu Gao, Fangyuan Kong, Lingting Zhu, Mengzhao Chen, Zhiheng Liu, Wei Liu, Qiushan Guo, Weilin Huang, and Ping Luo. Dancegrpo: Unleashing grpo on visual generation, 2025. BRLpttps: http://axvvvv.. org/ abs /2055 07818
Kai Yang. Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human feedback to fine-tune tee-tune diffusion models without any reward model. In Proceedings of the IEFF/CV Conference Computer Vision and Pattern Recognition, pp. 9941-8951 2024.
14

---

Published as a conference paper at ICLR 2026
Huizhuo Yuan, Zixiang Chen, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning of diffusion models for text-to-image generation. Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS '24, Red Hook. NY, USA, 2024. Curran Associates
Inc. ISBN 9798331314385.
Daoan Zhang, Guangchen Lan, Dong-Jun Han, Wenlin Yao, Xiaoman Pan, Hongming Zhang, Mingxiao Li, Pengcheng Chen, Yu Dong, Christopher Brinton, and Jiebo Luo. Bridging sft and dpo for diffusion model alignment with self-sampling preference optimization, 2025a. URL
https: http:///xxvvvv.rg./eee /2400. .05555
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2018.
Tao Zhang, Cheng Da. Kun Ding, Kun Jin, Yan Li, Tingting Gao, Di Zhang, Shiming Xiang, and Chunhong Pan. Diffusion model noise-aware latent reward model for step-level preference optimization. arXiv preprint arXi::2502.01035 2025b.
Tianyi Zhang*, Varsha Kishore Felix Wu", Kilian Q. Weinberger, and Yoav Artzi Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2020. https:://operrevi..e wett/forrmm..d-SSeeeaeee
Kaiwen Zheng, Huayu Chen, Haotian Ye, Haoxiang Wang, Qinsheng Zhang, Kai Jiang, Hang Su, Stefano Ermon, Jun Zhu, and Ming- Yu Liu. Diffusionnft: Online diffusion reinforcement with forward process. arXiv preprint arXiv:2509.66111 2025.
15

---

Published as a conference paper at ICLR 2026
A DERIVATIONS FOR PCPO OBJECTIVE
A. 1 PROOF OF PROPOSITION
Proposition a DDIM sampling schedule, the log policy ratio log Pt. is given by:
log Pt - (
)
C(t) oldd EEld + [[[((t)(E - w(t)
[t)
(4)
where
-I- ā C(t)
v 11-av ā 07 0.
Va
Proof.
N
x 1:
11- (20e 0
log Pt = log
N
x,- 1i
(2- (2+)), 0/21
1
(||8-1
140
(e- (2x))
||x P (xx) (1²)))
207
where
(1-1) (x.) =
(x,))(x))
VI
0g69 (xt)
v 0,-
ofE0 (xx).
var
(1-1) (xx) + 0 Told-
and
x = Hold
Substituting this into the equation above yields
x Mb
(t-1) (xx)]l 1
1 -Pold (xx)
(t-1)) (x(=))((e))
20?
(1 - :/. (((-2)(x)) (xx) + (0)1 l1o,e (0)
20?
[I((t)(E) (x,) -x.) + 2002 ||,, (t))
207
[[((t)(E!((s)) (x.) - -E( (xx) + c
oldl l| - ||
= 2
- (() (xx) (x:))
(1) Cold
+ [tt) (xx) =0ld (x:))))
A.2 PROOF OF EQ.
Here, we provide detailed derivation for the log policy ratio in the flow model setting, analogous to the proof for the diffusion model case.
We begin with the definition of the log policy ratio for single timestep &. which is the log- likelihood ratio the one-step transition probabilities. Both pe and Poua define a Gaussian distribution for the next state It, - At, given the current state Nt..-
(o 1 At,
Zt.)
log Pt. = log
Pold (1,,-110 xt..)
The SDE in Eq. 7 implies that the one-step transition is sample from N (p 1)(x))) Er,), where the variance is E,, = o? (At,) I and the mean is p
1) (xx,) = Xx, ug + o
so Sti where
(1
Sg
80 Xt. unti
(10)
16

---

Published as a conference paper at ICLR 2026
Using the formula for the log- log-ratio of two Gaussians with the same variance, get:
log Pt.
2(0 AT,)² (1xt--21
M4
(#:-1) (xxx)
||xx,
(1 1) (xxx) (1²)
(11)
Hold (1111)) (xee) Jz. v At, (4))
The next state Xt, Ar, is sampled from the old policy, so Xt,-Ar, Substituting this into the equation yields:
Cold
(II Ala ") (xx,) - Pb (xx,) + or v Als old
ald |o,,, v Ali old 11²)
log Pr,
207, A4;
(12)
Substituting Eq. (T)) into the mean difference, we have:
(1 -itt)
(ta- Pl (0(=))(x.))) = (uo( t() Louu t4)) (1;
(xx.)
(13)
2t:
Substituting this back and expanding the squared norm, we get:
log Pt.
[1(u0- Uola) uola)
At,()
(1 -1)o,
Tee At, oldd
(1)] 11o, v Dei
(1:)
112
-00 At;
2t,
||(B0 Udld)
1+ (1 - (1-t))o
20% م|i
21i
1 (1 - 11))
(or, Ali Dii (1:)
2(uu Alld
21,
At
+ (1 t.)o2,
021111 121lug- Uolall
Vdt,
(1
tz)o
(up Uola)
Cold
(1)
20
2f;
21;
By defining the weight w(t;)
viei ore
(1 +
(1-ti)o2,) 24,
we can rewrite the expression in a form
analogous the diffusion case:
log Pt. (tt))(us uold) e + I unw)l
(14)
This matches the structure Eq. confirming the derivation.
A.3 PROOF OF PROPOSITION
2
Proposition 2, For a flow matching SDE in the form of Eq. 7 the weight schedule w((t) that ensures credit is proportional the integration interval At, is given by:
N Vāt,
(1 t
w(t)) = CAts,
<=
(1 +
(9)
Tt.
2f:
Proof. To derive wtt)) x At, with the same mean weight as Wori we define normalizing coefficient such that:
N
Wongg (t,)
Σ w(tt),
where
VT.
-t) )
Wongg (t,)
(1
w(t,) = -At
2t:
Substituting this into the equation above yields Ç = Σn Worig (t:) = DN
(1 + 21.
(1-1)07
17

---

Published as a conference paper at ICLR 2026
Table 6: Index of Training Configurations. This table links the configuration index (from Table 7 its results in the paper. Main configurations in bold.
Config
Brief Description
Figures
Tables
DDPO Aes, Half Batch Size
b).
10
2 DDPO Aes
1 1a
10
11
a);
14
15
16
3
DDPO BERT, Half Batch Size
8(b)
10
4
DDPO BERT
b
17
5
DanceGRPO SD1.4
3
C),
13
[b),
6
DanceGRPO FLUX
1
3
10
b).
(b).
10
b).
20
7 DanceGRPO, DPO Comparison
8
FlowGRPO SI3..5, OCR
a.
9
FlowGRPO SD3.5, PickScore
5 [b, d)
10
DanceGRPO FLUUX, Naive Ace.
d
B
EXPERIMENT DETAILS
This section outlines the setup for all experiments, including training configurations, evaluation metries, and computational resources.
B. TRAINING SETUPS
We test PCPO across ten distinct training configurations, spanning three baseline algorithms, four base models, five reward functions, and five prompt datasets.
All experiments were conducted using fully online training, where each iteration generates fresh trajectories from the current policy. We specify important configurations and hyperparameters for each setup in 7 Apart from the learning rate, which was lowered from 3 10-4 to 1 x 10-1, and LoRA parameters for DDPO experiments, all hyperparameters follow that of the original codebases. For DDPO and Flow-GRPO, we used our memory-efficient reimplementations, following the logic of the official codebases.
B.1.1 TABLE LEGEND
Prompts. "animal" refers to the simple_amiie animals dataset from the DDPO codebase, consisting of 45 simple animal nouns (e.g.. "cat", "dog". "fish"). "activity" refers to the nouns_a i t es dataset from the same codebase, consisting 135 prompts combining animal nouns with activity verbs ("washing the dishes", "riding bike". "playing chess"), "HPD" refers to the open-vocabulary prompts from the HPDv2 dataset used in Xue et al. (2025). "Pick" refers to the Pick-a-Pic dataset, and "Pick-SFW"" refers the safe-for-wont (SFW) subset from the Flow-GRPO codebase. "OCR" refers to the OCR prompts from the OCR dataset.
Reward. "Aes", "BERT", "HPS". "CLIP", "Pick", and "OCR" are abbreviations for "Aesthetics". "BERTScore", "HPS-v2.1". "CLIP score", "PickScore", and "OCR" reward models, respectively.
Timestep subsampling. DanceGRPO(FLUX)) default settings the heuristic acceleration by subsampling 60% of timesteps for policy updates.
SKL. This is the coefficient for the KL penalty term for KL-regularized policy gradient algorithms, as in Equation (9) of et al. (2023) or Equation (3) of et al. (2025). An empty entry indicates no KI. penalty used.
18

---

Published as a conference paper at ICLR 2026
DanceGRPP
FLUX.1-lee
" 0.3
10
HPD
HPS
0.6
16
1.0
le.5
1.0
32
12
384
4
le-4
60
Table 7: 10 Training Sctups. Each column (1-10) indexes a unique setup. Configurations main experiments are in bold.
Flow-GRPO
SD3.5-M
Pick-SFW
n= 0.7
(32,64)
Pick
10
4.5
0.01
3e-4
1.0
32
16
512
le-4
20
Flow- GRPO
F SD3.5-M
OCR
OCR
1J ==0.7
10
4.5
0.04
3e.4
1.0
8
64
1e-4
(32,64)
20
DanceGRPO
SD1.5
Pick
Pick
50
5.0
le-5
1.0
32
16
512
2
le4
50
Dance
FLUX.1-dex
(128,256)
HPD
HPS
0.6
12
1.0
3e-4
1.0
32
12
384
1c-4
60
"
Dance
D SDL4
HPD HPS
50
5.0
1c-5
1.0
32
16
512
1c-4
50
DDPO
SD1.5
activity BERT
50
5.0
1.0
576
1c-4
(4,4)
DDPO
SDLS5
activity BERT
50
5.0
1.0
288
le-4
(4,4)
2
DDPO
SD1.5
animal Aes
50
5.0
1e4
1.0
512
1e-4
(4,4)
DDPO
D
SD1.5
animal
Acs
50
5.0
1e44
1.0
256
le-4
(4.4)
Sampling Hyperparume
Timestep subsampling
Training Hyperparans
Framework Model
Baseline algorithm
Prompts per iteration
Images per prompt Images per iteration Gradient updates Clip range (E) LoRA (r. ())
Noise level (SDE)
Checkpoint freg.
Sampling steps
Max grad norm
Diffusion/Flov
Learning rate
Parameter
Base model
Prompts
Reward
Task
3kc.
19

---

Published as a conference paper at ICLR 2026
Prompts/iteration & Images/prompt. These specify the number of unique prompts sampled per training iteration, and the number of images generated per prompt. For DDPO, Images per iteration random prompts are sampled each iteration, it does not require group-level normalization.
LoRA, and the rank and scaling factor for LoRA fine-tuning, respectively. An empty entry indicates full fine-tuning.
B.2 EXPERIMENT-SPECIRE RATIONALE LIMITATIONS
B.2.1 DDPO
This experiment replicates the original DDPO setup to establish a direct baseline. The goal is to evaluate PCPO's impact stability and sample fidelity within simple, well-understood environment. The lightweight nature of the task allows for checkpointing every epoch, enabling us to pinpoint the exact epoch where reward levels converge on validation of 50K seed-controlled images. This facilitates a rigorous statistical analysis using LMMs to isolate PCPO's effect from prompt-level variance.
Rationale:
Crucially, this setup is intentionally limited to simple prompts (45 animal nouns for Aesthetics, 45 animal nouns 3 activity verbs for BERTScore). As noted in Wallace al. (2024), the DDPO framework does not generalize well to complex, open-vocabulary prompts. Therefore, ability. Open-
Limitations:
this experiment evaluates performance on a narrow task, not general vocabulary experiments based on DanceGRPO and Flow-GRPO are presented later in the paper to address this.
Miscellaneous Details: For BERTScore, we replace the LLaVA VLM used in the original implemen- tation with Qween255VL--3B-Immiie (Bai et al. 2025).
B.2.2 DANCEGRPO
This experiment evaluates PCPO in more complex, open-vocabulary setting, using the DanceGRPO framework (Xue et al.) 2025) and the HPDv2 prompt set. This setup tests PCPO's ability to maintain fidelity and diversity while optimizing for high-level human preferences (HPS).
Rationale:
Limitations: The DanceGRPO framework is computationally intensive, limiting the number of ablations and repetitions can perform. Most ablations are conducted the simpler DDPO setup. For FLUX, we performed LoRA fine-tuning on GPUs, using the official scripts provided by the authors. This differs from the 16-GPU full fine-tuning in the original paper, and results in slightly lower performance ceiling. DanceGRPO's configurations for FLUX performs 4 gradient updates per iteration and uses timestep subsampling, making the training dynamic more fragile.
Miscellancous Details: For SD, we mainly use SD1.4 following Xue et al. (2025). For additional experiments, we SD1.5 as the base model align with other DPO-based frameworks. Training was terminated at 240 epochs, as further training degraded image quality for the baseline with minimal reward gains (Figure 20). Note that the naive acceleration ablation (Figure (d)) was run using full fine-tuning on 8 H100
B.2.3 FLOW-GRPO
Rationale: To demonstrate maximum generalizability, this experiment applies PCPO to com- pletely different regime: a different model (SD3.5-M), noise schedule (Flow-GRPO), rewards (OCR, PickScore), and training objective (has a KL penalty). Success here shows that PCPO is a model- agnostic principle, not quirk of one specific setup.
Due to resource constraints, we conduct experiments on the resource-light pickscore_ss _d__d_______ gpu and ocr__d3 per_ad3__gpu setups from the Flow-GRPO codebase, not the full-scale setups in the original paper.
Limitations:
To code GPUs with 24GB VRAM, modified the prompt sampler. In this process, we improved upon the original implementation by guarantecing the number of unique prompts per iteration match the configured value. For PickScore experiments, used the Pick-a-Pic
Miscellaneous Details:
20

---

Published as a conference paper at ICLR 2026
SFW sub-dataset instead of the Pick-a-Pic dataset used in the original paper, to avoid generating NSFW content. KL divergence was calculated as:
DKL (To) [Ter)
At
or t)
)
(uoo(x())) (xe t)l?
21
Fe.
following Liu et al. (see Appendix Dj.
2025), which avoids exp () operations and thus maintains numerical stability
B.3
EVALUATION DETAILS
B.3.1 IMAGE QUALITY METRICS
For controlled evaluation on image quality metrics, we generate fixed set of 50,000 images (45 prompts x 1112 seeds). This set is produced by the original base model to serve as reference, and by each fine-tuned policy for comparison. We evaluate these images at two levels of granularity:
We compute FID, FDDINO and IS across the entire 50k generated samples to measure overall fidelity and quality.
Overall Metries:
Per-Prompt Analysis: For more fine-grained statistical analysis, we compute metries on per-pro per-prompt basis (i.e., on the 1112 images generated for each prompt). We use a LMM on these per-prompt scores to isolate our method's true impact from prompt-level variance. Coefficients were estimated via Restricted Maximum Likelihood (REML). At this level, we also compute LPIPS Diversity [Zhang et al. 2018) to measure intra-prompt variety.
It important to note that FID and FDDINO scores computed smaller sample sets (e.g., FD per 1.1k images) are expected to have a higher magnitude than scores computed on the full dataset (FD-50k). Consequently, the effect size (Baig) estimated by the LMM, which is based these per-prompt scores, may appear larger in magnitude than the simple difference observed the overall FD-50k results. Since FD scores caleulated against the outputs of the original base model, they can be interpreted as a measure of distributional drift; a lower score signifies less deviation from the original policy. This indicates a successful mitigation of mode collapse and serves as a strong proxy for the preservation of the base model's 's inherent fidelity and diversity.
B.3.2 ALIGNMENT & GENERALIZATION METRICS
In addition to quality metrics, we evaluate on a suite of preference alignment metrics (HPSv2.1, CLIP- Score, PickScore, ImageReward, Aesthetics). These used in Table to measure generalization to unseen prompts and reward models. The MJHQ-30K dataset contained 3.3K prompts that do not fit in the CLIP tokenizer (> 77 tokens). As such, filtered these and used the first 5K prompts of the remaining set for evaluation. For Table we use HPSv2 instead of HPSv2. I to align with prior works. We used the 2048 prompts from the Pick-a-Pic evaluation set provided by the Flow-GRPO codebase. For all prompt datasets, we generated 1 seed-controlled image per prompt.
B.3.3 HUMAN PREFERENCE STUDY
To ensure a fair comparison that accounts for PCPO's accelerated convergence, we conducted a "bracketing" human preference study. Our model (epoch 120) was compared against two baseline DanceGRPO checkpoints that bracket its reward level: epoch 180 (lower reward) and epoch 240 (higher reward). The evaluation used pool of 450 unique image pairs for each comparison (45 prompts x 10 seeds). To climinate bias, web interface presented these pairs by fully randomizing the comparison set, prompt, specific image, and left/right display position for each question. We collected 297 responses against epoch 180 and 332 responses against epoch 240.
B.3.4 PLOTTING AND VISUALIZATION DETAILS
All training curves and clipping fractions, unless specified otherwise, are smoothed with a moving average window of epochs to improve readability.
Figure Subfigures (a, b) plot the validation reward, unlike other plots that show the reward computed training samples each iteration. This leads clearer trends, the training reward is
21

---

Published as a conference paper at ICLR 2026
Table 8: Computational resources used for each experimental setup,
Experiment
Hardware
DDPO (Aesthetics)
2x NVIDIA RTX 4090 (24GB)
DDPO (BERTScore)
3x NVIDIA RTX 4090 + Ix VLM GPU
DDPO (Ablations)
2x NVIDIA RTX 3090 (24GB)
DanceGRPO (SD1.4)
8x NVIDIA A100 (40GB)
DanceGRPO (SD1.5)
8x NVIDIA A40 (48GB)
DanceGRPO (FLUX)
8x NVIDIA A5000 (24GB)
DanceGRPO (Naive Acc.)
8x NVIDIA H100 (80GB)
Hlow-GRPO (OCR)
1x NVIDIA RTX3090
Flow-GRPO (PickScore)
4x NVIDIA RTX3090
noisy due to randomness from prompts sampled at each iteration. Validation rewards are computed every 50 epochs fixed set of 1024 prompts that not in the training pool.
Subfigures (c, d) plot the Pareto frontier between reward and KL divergence with respect to the reference policy (base model; not fine-tuned). To reduce noise in the visualizations, the KL divergence is plotted using a rolling mean of all estimated values up to the current epoch. While this approach yields low-variance estimate, likely underestimates the true KL. divergence that specific epoch. Both the baseline and PCPO start at the top-left corner (low reward, low KL divergence), and move towards the bottom-right corner (high reward, high KL divergence) as training progresses.
B.4 COMPUTATIONAL RESOURCES
RL-based fine-tuning is known be sensitive to hardware and implementation details. We report our resources for full reproducibility in Table
C COMPARISON WITH DPO AND SELF-PLAY BASELINES
For completeness, we attempt to provide comparisons with reward-free methods. While this attempt is meaningful, we emphasize that direct, fair comparison between reward-based methods (such as PCPO) and reward-free methods presents significant methodological difficulties that render head-to-head evaluation inconclusive.
Difficulty 1: Establishing Fair Evaluation Protocol. The core challenge lies in the choice of evaluation metric and the fairness of access to human preference data.
Limiting reward-based methods single model (c.g., PickScore, trained on subset of Pick-a-Pic prompts) creates an unfair disadvantage. Reward-based methods inherently rely on "distilled" knowledge from human annotations, whereas reward-free methods learn directly from preference data. Moreover, it restricts reward-based methods from leveraging their ability to aggregate preference knowledge from multiple sources (e.g.. linear combinations of reward models, explored in Xue et al. 2025): Zheng et al. (2025)).
Restricting Reward Model Choice:
Unrestricted Reward Model Choice: Allowing reward-bard based methods to use a linear combination of multiple reward models custom reward functions then argued to be unfair to reward-free methods, which must "zero-shot" against a richer evaluation metric they were not trained optimize. also allows reward based methods to indirectly access a wider set of human preference data distilled into various reward models.
Similar to prior work comparing DPO methods against policy-gradie methods (Liang al. 2025 Zhang et al.) 2025b), we set our experimental protocol to place PCPO at a comparative by using a single, fixed reward model (PickScore).
Difficulty 2: Reproducibility and Closed-Weight Models. A second major hurdle the lack of open-source checkpoints for SOTA DPO or self-play baselines, including SSPO (Zhang al. 2025a)
22

---

Published as a conference paper at ICLR 2026
Table 9: Comparison with DPO and Self-Play baselines on SD1.5, evaluated on preference metrics from the Pick-a-Pic dataset. Scores of other works are cited from each paper. f: trained on Pick-a-Pic v2. More details in text.
Method
HPSv2 (1)
Aesthetic (f)
PickScore (1)
ImgRwd (1)
SD1.5 (Base)
0.261
5.43
20.09
0.09
PCPO (E200)
0.271
6.14
22.42
0.99
gains (E200)
+0.010
+0.71
+2.33
+0.90
SDI.5 (Base)
0.238
5.37
20.53
-0.16
SSPO f
0.272
5.94
21.90
0.70
gains
+0.034
+0.57
+1.37
+0.86
SDI.5 (Base)
0.271
5.77
21.18
0,92
SPIN-Diffusion Iter3
0.276
6,25
22,00
1.12
gains
+0.005
+0.48
+0.82
+0.20
SDI.5 (Base)
0.265
5.47
20,56
0.08
LPO gains
0.276
5.95
21.69
0.66
+0.011
+0.48
+1.13
+0.58
SPIN-Diffusion (Yuan et al., 2024), and RainbowPA (Sun et al. 2025). Furthermore, while most work report similar evaluation protocols, i.e. slicing the Pick-a-Pic dataset into training and validation sets, the exact details of implementation differ significantly across published works. Notably. the reported base model (SD 1.5) performance varies widely, suggesting substantial variation in the difficulty and composition of the validation prompts used. Moreover, some works (Zhang et al. 2025a) use the richer Pick Pic v2 dataset, while others apply filtering (Zhang et al. 2025b).
A head-to-heer head comparison is thus difficult, and reported results must be interpreted with caution. While we attempt to normalize for prompt differences by comparing relative gains (final score minus reported base score), this metric can still favor models that report an atypically low base score. For instance, SSPO reports a significant gain of 0.034 in HPSv2, but its final performance commensurate with that of other methods, suggesting their validation set was uniquely challenging for the base model, inflating the relative gain metric.
Comparison Results. Nonetheless, we present quantitative comparison between PCPO and leading DPO/self play baselines Table | PCPO was trained using the standard DanceGRPO setup on SD1.5, using Pick-a-Pic prompts and the PickScore reward model. PCPO demonstrates highly competitive performance, underscoring its efficacy in human preference alignment. While CLIPScore comparisons are omitted due their absence in prior literature, we note that PCPO preserves semantic fidelity, maintaining a score of 0.387 (identical to the base model).
D ADDITIONAL RESULTS
Speedups for Different Reward Thresholds. Table details the training efficiency gains PCPO across all experimental settings. Speedup is calculated based on the number of epochs required to reach specific reward thresholds during training. training involves stochastic sampling of prompts and seeds, speedup values are subject to noise from the stochastic sampling of prompts and random seeds per iteration.
More Convergence Plots. Figure 8 shows the reward optimization traces and clipping fractions for the default, smaller batch size of 256 in DDPO.
Wall-Clock Time and Computational Cost. PCPO introduces computational overhead compared to the bascline (Figure 9). While baselines require an additional backward SDE sampler step to compute po for policy updates, PCPO only requires the network forward pass for noise/velocity prediction, which baselines also compute. By saving this SDE sampler step, PCPO achieves slightly lower wall-clock time per epoch.
Soundness of log Pr se pr - Approximmtiion We justify our choice using the log-ratio log Pt in
the PCPO objective (Fq. 3 via Taylor approximation from four aspects.
23

---

Published as a conference paper at ICLR 2026
Table 10: Training efficiency comparison across all experimental settings. We report the number of epochs for the baseline and our method (PCPO) to reach various target reward levels. Speedup is calculated as (EpochS pppochspcro 1) 100%. Highlighted rows correspond to the primary reward targets discussed in the main text.
Framework
Setting
Reward
Baseline
PCPO
Speedup
6.90
260
131
98.5%
Aesthetics (256/128)
6.60
210
100
110.0%
6.30
165
70
135.7%
6.00
108
48
125.0%
6.90
147
118
24.6%
Aesthetics (512/256)
6.60
107
89
20.2%
6.30
61
19.7%
DDPO
6.00
47
44
6.8%
(SD1.5)
0.52
246
198
24.2%
BERTScore (288/144)
0.51
188
175
7.4%
0.50
145
155
-9.4%
0.49
90
112
-19.7%
0.52
191
146
30.8%
BERTScore
0.51
159
124
28.2%
(576/288)
0.50
125
89
40.4%
0.49
86
74
16.2%
0.37
236
188
25.5%
DanceGRPO (SD1.4)
0.36
173
153
13.1%
HPSv2.1
0.35
153
124
23.4%
0.34
116
102
13.7%
0.36
209
148
41.2%
DanceGRPO (FLUX)
0.35
148
93
59.1%
HPSv2.1
0.34
127
83
53.0%
0.33
50
102.0%
(a)
(b)
x x00
(a)
(b)
7.82
0.56
2100
Reward
Clipfrac
Time
5.50
0.44
0
585
2050
Epoch
200
Epoch
200
Epoch
300
Epoch
200
DDPO
PCPO
DanceGRPO
PCPO
Figure 8: Training trajectories smaller batch for PCPO (orange) and DDPO (blue) on (a) Aes- thetics, (b) BERTScore rewards.
Figure 9: Wall-clock time per epoch for Dance- GRPO (blue) and PCPO (orange) for (a) SD1.4, (b) FLUX training.
1. Empirical Distribution of p:: We tracked the distribution of Pt during training (from scratch and near convergence) for both diffusion (DDPO, SD1.5) and flow models (DanceGRPO, FLUX). Figure 10 shows that Pe remains close to 1.0, with a standard deviation on the order of 10 10-4 This confirms that the policy operates precisely in the region where the approximation valid, making the Taylor approximation error negligible.
2. Error Bounding: The clipping mechanism of PPO and GRPO provides a strong bound on the upproximation error. In T2I alignment, the clip range & is typically set to very small value (e.g.. 10 for all of our experiments). The gradient in our objective (Eq. (3) only flows when log Pel is within this clipping range. This means the Taylor approximation is
24

---

Published as a conference paper at ICLR 2026
(a)
(b)
Policy Ratio
1.000
0.999
4 148
152
4 181
185
Epoch
Epoch
Figure 10: Empirical distribution of the policy ratio Pr during training for (a) DanceGRPO (FLUX) and (b) DDPO (SD1.5). In all cases, the mean, min, and max values remain exceptionally close to 1.0, validating the Taylor approximation log Pt Pt 1. Shade heights correspond to 2 std(pr).
Algorithm Ratio Computation
Baseline (PPO/GRPO)
PCPO (Ours)
1: log Pg < policy.evallsteei.e action)
1: log & policy.eval(state action)
2: log Pold 6- old oldooodpppoo www.ppy.clllllseeee.. (state, action)
log Pold <- old old__poicyyeeall (state, action)
3: log <- log po
log Pold
3: log p <- log pa
log Pold
4: p + exp(log p) Unstable step
4: // No exp() step
5: loss A)
5: loss + fllog A) // Stable
only ever utilized in the exact region where is most accurate (1p--1) log p making the effective approximation error negligible (on the order of O(
< < 1),
3. Numerical Stability: From numerical perspective, we argue that more significant source instability- which which our log Pt formulation also fixes-is the numerical precision error in computing Pe itself. As shown in Algorithm [ standard PPO/GRPO must compute exp(log Th log Tota). PCPO, in contrast, operates directly in log-space by using log P: = log n log Vold in the objective. While oe should be strictly 1.0 for an on-policy update, our per-step measurements typically show mean value of 0.99997 to 0.99998. This suggests that the floating point error larger than the Taylor expansion error /2/2.
4. Theoretical Justification: Lastly, even if the approximation were to fail, our formulation remains sound. As shown by Huang et al. (2024). the objective from Eq. (2) is a hinge loss where P. acts as "classifier. Their work demonstrated that this term can be swapped with log Pr (among other variants /pi me Tota) while yielding similar performance. This validates our log-hinge objective as a robust choice on its own merits, independent of the approximation.
Gradient Stability Analysis. To provide quantitative analysis of training stability, we measured the mean of the absolute gradient, B[]gr]]. and the variance of the absolute gradient, Var(Ig:)). for the first layer of the U-Net transformer. This analysis performed both DDPO (SD1.5) and DanceGRPO (FLUX) frameworks during two training phases: (i) the initial 5 epochs (starting from scratch) and (ii) 5 epochs near convergence (loaded from checkpoints at target reward levels).
The results, shown Figure TT] confirm PCPO's superior stability. While the gradient statistics are similar for both methods at the very start of training, significant gap emerges as training progresses. Near convergence, the baseline models exhibit gradients with larger mean and variance. This provides direct, quantitative evidence that PCPO mitigates gradient instabilities, leading to smoother and more stable training dynamics.
E CONNECTION TO REINFORCE ELIGIBILITY VECTOR
To formalize the connection between proportionality principle and established RL theory, analyze the REINFORCE algorithm (Sutton & Barto 2018 Williams 1992). REINFORCE provides first-principles, Monte Carlo estimator for the policy gradient.
For an episodic, non-discounted (y = 1) task with terminal reward, the parameter update is sum over all timesteps f. In our setting, the return G, is the same terminal advantage A for all steps. The
25

---

Published as a conference paper at ICLR 2026
(a)
(b)
x10
x10
1.2
l9d
0.0
-
Epoch
Epoch
DanceGRPO
PCPO
Figure 11: Plots of I9el for the network's first layer during training. Shades indicate std((ge))
5
0
100
10a
102
Magnitude Ratio
Figure 12: Magnitude ratio of the dot product term to the squared norm term in Equation 0 for the first off- off-policy data point DDPO. The dot product term is dominant even the off off-policy case, justifying our simplification.
update rule is therefore:
T
A (0), e,(0) V log (eg)s()
(15)
As noted by Sutton & Barto 20188. the term (#) is the eligibility vector. This vector is the component containing the policy's 's parameter gradients, and is the "only place that the policy parameterization appears the algorithm." Assuming each action contributes equally to the final reward, REINFORCE assigns credit scales each eligibility vector by the same advantage for fair credit assignment.
We now draw direct analogy our on-policy case (justified in Figure 12 From analysis, the gradient contribution from single timestep t is proportional to:
Ad, & A wt) [V
(16)
Analogous to (0)
The term Voe (t) Cod our core policy gradient, which is analogous the eligibility vector (#). However, as Equation 16]shows, [qustionnT6]shows,, this "eligibility vector" is multiplied by w(l) before being scaled by the advantage A. Assuming the expected statistics of the dot product term are constant across timesteps, the overall gradient magnitude for each step becomes directly proportional to its weight, w()).
This (t) is arbitrary scaling factor that arises from the sampler's mathematics, not a deliberate credit assignment choice: iee. proportional to the integration interval At. It breaks the uniform credit assignment of the REINFORCE framework by non-uniformly and inconsistently scaling the gradient contribution from each step. PCPO restores sound credit assignment by ensuring that w(t) proportional the timestep Al, its true contribution to the trajectory.
26

---

Published as a conference paper at ICLR 2026
F IMPLICIT REWARD GUIDANCE
We introduce Implicit Reward Guidance (IRG), an inference-time scaling mechanism that mitigates reward overoptimization by interpolating between base model and one more RL fine-tuned models. This principle was concurrently developed in the Reinforcement Learning Guidance (RLG) framework (Jin al., 2025); ; our work further extends the concept by demonstrating its application to dynamically compose multiple learned rewards.
The theoretical motivation for this stems from D3PO (Yang et al. 2024). which interpreted the learned policy as implicitly encoding reward signal log (Xo x xx)). Following CFG++ (Chung et al. 2025), IRG frames the sampling process as an optimization problem where, given a noisy state X,, we aim to find sample Xg that maximizes the implicit reward function.
To guide the sampling process, we can take gradient step in the direction that maximizes this reward. Approximating the predicted denoised sample &o projection of x,, the gradient of the reward with respect to & simplifies to:
Po (x Pro x()
Xx)
1 o
So log
(Ho(Xe)
(xx((xx)).
where 14 and peet the means of the one-step reverse transition distributions for the target and reference policies, respectively. This gradient term represents the direction of maximal reward increase.
While CFG++ adds this guidance term of the denoised sample Xn before renoising, this can be unstable for IRG. In IRG, the guidance term comes from a separate model that is distinct from the generative model being used for sampling. Using different models to compute separate noise vectors the Tweedie mean estimation and the renoising step can destabilize the DDIM inversion process.
Therefore, we propose an anlogous but more stable formulation where the guidance is applied directly to the noise prediction. By defining guided noise estimate @ (x,) == (x;) + (x))(x)) (x;(( we use this single, consistent noise vector for both the Tweedie mean estimation and the renoising step of the DDIM update. This ensures stability while effectively steering the generation process toward higher-reward outcomes.
Furthermore, IRG can compose multiple learned rewards. Given models (0.) each aligned to distinct reward, IRG forms mixed predictor:
é
IRG (0.)
Exef +
Σ Aeeo - -reet).
1=1
This principle extends directly flow models by replacing the noise predictor è with the velocity predictor ú:
JRG (0,)
úre +
Ae (ūo. -úret).
As shown in Figure 13 IRG effectively mitigates reward overoptimization by tuning the guidance scale À inference time. also enables the flexible composition of multiple objectives, providing smooth interpolation between competing rewards like aesthetics and text-alignment (Figure
27

---

Published as a conference paper at ICLR 2026
-=
==
== 0.6
= 1.0
== 1.2
== 1.5
Figure 13: Test-time scaling via IRG at various À values: Top rows: Reducing the weight to À < interpolates between the base model and fine-tuned model, alleviating reward overoptimization at inference-time. Bottom rows: Increasing the weight to A > 1 extrapolates the internal reward, thus can be used to enhance visual appeal. Full prompt list in Appendix G
(a)
(b)
0.53
\
A1 + A2
1.5
TERTe
A1
1.0
X2
0.47
5.5
Aesthetics Score
7.0
Figure 14: IRG tradeoff. (a) Aesthetics-BERRRSSCoo curves for different guidance scales, showing that increasing aesthetic guidance A₁ boosts visual quality but reduces semantic alignment, and vice versa for semantic guidance X2. (b) Generated samples for two prompts (top: a bear riding bike; bottom: dog washing the dishes) arranged by increasing A₁ (left arrow) and X2 (right arrow) for X1 + Xa 1.5, demonstrating smooth interpolation between aesthetic and semantic objectives.
28

---

Published as a conference paper at ICLR 2026
G FULL PROMPTS FOR FIGURES
Figure 1
(a) Aesthetics reward: cat, bear, butterfly, chicken, bee, camel.
(b) HPSv2.1 reward:
* A Thangka painting depicting the devil controlling group of people in circular formation.
* A painting of a monkey wearing gold headphones and sunglasses looking up at starry night sky.
* Matador challenging a bull in desert-filled ramen bowl.
Figure 13
Minimalist sticker art featuring abstract designs by Victor Ngai, Kilian Eng, and Lois Van Baarle.
The kitchen has a stove and refrigerator.
A close-up portrait of cute anime girl with extremely detailed eyes, featured as key visual in official media.
A green 4D-printed Biblically accurate angel.
Figure 17
a bat playing chess
bird riding bike
fish lion
Figure 18
A portrait of a lion goddess wreathed flame. posing in full body.
A portrait painting of Stephany Eisnor.
The kitchen has a stove and refrigerator.
Figure 19
Several people waiting at a bus stop in a dark city night, depicted in a digital illustra- tion.
Side-vies portrait of a knight with a skull helmet adorned with spikes, depicted tenth century stained glass window.
The album cover for the band Underealm features an evil entity in sophisticated suit, with dark and intricate details.
Figure 20
A Thangka painting depicting the devil controlling group of people in a circular formation.
A painting of a monkey wearing gold headphones and sunglasses looking up at a starry night sky.
Matador challenging a bull in desert-filled ramen bowl.
Several people wailing at a bus stop in a dark city night, depicted in a digital illustra- lion.
Side-view portrait of a knight with skull helmet adorned with spikes, depicted in tenth century stained glass window.
The album cover for the band Underealm features evil entity in sophisticated suit, with dark and intricate details.
The image features a copper material with references to various art platforms and artists.
Anor Londo, Dark Souls, with dragon flying the distance at night.
Two Tyrannosaurus rexes engaged boxing match.
The image depicts Yoshi Island created by Beeple.
-Digital painting featuring Soviet realism and grunge elements with a range of artistic influences, created by multiple artists and showcased on ArtStation.
- An artwork from Dan Mumford collection featuring mage invoking divine gods during storm with lightnings.
29

---

Published as a conference paper at ICLR 2026
H
MORE QUALITATIVE RESULTS
DDPO
Oun
DDPO
Ours
Figure 15: PCPO mitigates mode collapse high reward levels. For the prompts "hedgehog" and "mouse" (6 consecutive seeds), DDPO (top) exhibits noticeable mode collapse. In contrast, PCPO (bottom) maintains high visual diversity and sharpness the same reward level.
Figure 16: PCPO generalizes better to unseen prompts than DDPO. For prompts that were not in the fine-tuning prompt dataset (" ("armadillo". "colored pencils", "envelope", "microwave"), PCPO retains the generation capabilities much better than the baseline DDPO.
30

---

Published as a conference paper at ICLR 2026
0
50
100
150
200
Figure 17: PCPO demonstrates superior training stability, avoiding premature collapse. This figure tracks the qualitative progression per epoch for DDPO (top rows) and PCPO (bottom rows). The baseline DDPO shows early image degradation across both BERTScore (first two groups) and Aesthetics (last two groups) rewards. PCPO consistently maintains higher fidelity, although it can also exhibit collapse extremely high reward levels (e.g., Aesthetics epoch 200).
31

---

Published as a conference paper at ICLR 2026
100
200
Figure 18: DanceGRPO SDL.4, progression per epoch. Within each group. the top row shows DanceGRPO (SD1.4) results, the bottom row shows PCPO. PCPO better preserves fidelity and text-image alignment, DanceGRPO exhibits noticeable model collapse.
32

---

Published as a conference paper at ICLR 2026
60
120
180
UNIC
Figure 19: DanceGRPO FLUX, progression per epoch. Within each group, the top row shows DanceGRPO (SD1.4) results, the bottom row shows PCPO. PCPO better preserves fidelity, Dance- GRPO exhibits noticeable visual artifacts.
33

---

Published as a conference paper at ICLR 2026
Figure 20: PCPO is more robust to quality degradation from prolonged training. A comparison epoch 240 on the FLUX model. While extensive training for minimal reward gains causes the bascline (top row) to suffer significant image degradation and artifacts, PCPO (bottom row) maintains its quality much more effectively.
34

---

Published as a conference paper at ICLR 2026
1 USE OF LARGE LANGUAGE MODELS
We utilized Large Language Models (LLMs) as assistive tools throughout the research and manuscript preparation process. For research ideation, Google's Gemini was instrumental in suggesting the use of LMMs to control for prompt-level variance in our analysis. OpenAl's ChatGPT, Github's Copilot, and Gemini were employed to help draft and refine code implementations. In preparing the manuscript, ChatGPT and Gemini were employed to compose initial paragraph drafts from detailed research notes, In addition, these LLMs assisted restructuring sentences for improved readability.
35


Reviews:
[{"review_id": "OSy27l2zj7", "reviewer": "Reviewer_4hys", "summary": "The paper identifies disproportionate credit assignment across timesteps as a major cause of instability in the alignment for text-to-image models. \nIt proposes Proportionate Credit Policy Optimization (PCPO), which reformulates the PPO objective into a numerically stable log-hinge view and then enforces proportional credit over the sampling trajectory. \nFor diffusion models, PCPO makes per-step weights uniform by improving the variance schedule; for flow models, it restores proportionality by reweighting the training loss by each step’s integration interval, avoiding invasive sampler changes.\n\nAcross DDPO and DanceGRPO, PCPO reduces clipping and variance, speeds convergence, and improves fidelity, as indicated by lower FIDs, while mitigating collapse. \nA small human-preference study favors PCPO over strong baselines. Overall, the method offers a simple fix that stabilizes training and yields samples of better fidelity at similar reward levels.", "questions": "- Could you please report the $E[\\|g_t\\|]$, $Var(\\|g_t\\|)$, and clip rate w.r.t. t for basline and PCPO? Having two visualizations/reports in the early stage, and near convergence, would be better.\n- Why does it cause a negative effect when over-weighting the higher noise region?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Clarity. This paper is well written in terms of motivation and technical content. The paper clearly states the core problem, training instability in T2I alignment due to disproportionate credit across timesteps. The motivation and scope easy to follow. Figures helps understand the volatile and modified weighting clearly. \n- Technical contributions. Given the replace of $\\rho_t-1$ with $\\log \\rho_t$, the motivation is clear. This work further provides fix to the weighting under time shift for diffusion and flow models.\n- Improvements. PCPO offers convergence acceleration to the target reward levels from 24.6% to 41.2%, while keeping better fidelity, as indicated by FID scores.", "weaknesses": "- Metrics. FID provides a reasonable measurement of the shift from the base distribution, though it is less accurate for images of higher resolutions. It would be better to have modern metrics such as FD-DINOv2 for fidelity evaluation. \n- Regarding IS, it would be more scientific to claim that given similar FID levels, a higher IS indicates better quality; or given a similar IS score, a lower FID indicates less mode collapse.\n\nTo be clear, this paper is of good quality regarding a detailed question in the alignment of diffusion and flow models, and meets the bar of ICLR. The following questions and suggestions could improve the quality. \n\nSuggestions:\n- Plot per‑timestep gradient contributions and clipping rates before/after PCPO; report variance reduction in those contributions.\n- Quantify the Taylor error during training with a time‑series plot; show whether the stability gain comes primarily from the proportionality fix or from the log‑hinge.\n\nTypos:\n- model collapse -> mode collapse", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AQKvhYlLzP", "reviewer": "Reviewer_71VF", "summary": "This paper reveals that disproportionate credit assignment is  a key reason of instability when training T2I models using reinforcement learning. Proportionate Credit Policy Optimization (PCPO) is therefore introduced to deal with this problem. PCPO improve training stability of RL in training T2I models, leading to convergence acceleration and superior image quality.", "questions": "Table 1 employs only a single target level for each reward. It is important to investigate whether the reported conclusions remain consistent across different target levels. A discussion on the robustness and generalizability of the results with respect to this parameter would strengthen the paper.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper proposes Proportionate Credit Policy Optimization to deal with  training instability of RL for T2I fine-tuning. PCPO not only enhances numerical stability, but also ensures proportional credit assignment. The author also provide detailed theoretical derivation.\n\n2. Compared to DanceGRPO, PCPO introduces  convergence acceleration and superior image quality.", "weaknesses": "1. It would be beneficial to include an introductory overview of diffusion models and GRPO in the preliminary section, as their absence hinders the paper's accessibility for researchers new to the field.\n\n2. FID can not measure the human preference score of the generated images and text-image consistency, CLIP Score, ImageReward, and HPSv2  should be included in the  Tables  for comparison\n\n3. The text condition for image generation in Figure 13 and 14 is too simple,  I would like to see more complex qualitative comparisons in Appendix G, and text condition should be given so that I can observe text-image alignment.\n\n4. The experiment is not conducted on widely used datasets, e.g., MSCOCO-2014 30K validation  dataset, MSCOCO-2017 5K validation  dataset, MJHQ-30K dataset. It is benefical to convince me via reporting the results on above datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W7yI6Szd3l", "reviewer": "Reviewer_oEo3", "summary": "This paper proposes PCPO (Proportionate Credit Preference Optimization), a new framework to address the credit assignment problem in diffusion model preference optimization. While previous methods (e.g., DPO, LPO, TAPO) apply uniform or stepwise weighting across diffusion steps, they fail to proportionally assign reward signals according to the relative contribution of each latent step to the final image quality.\nPCPO introduces a proportionate credit weighting mechanism that computes latent-wise contributions via gradient-based sensitivity analysis, enabling consistent and efficient optimization across diffusion trajectories. The authors provide a theoretical justification linking PCPO to policy gradient credit assignment and demonstrate its empirical advantage on text-to-image (SDXL, SD3.5) and text-to-video (ModelScope) benchmarks, improving both alignment and fidelity metrics.", "questions": "1. Add more metrics?\n2. Add more ablations?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novel problem formulation: The paper clearly identifies the overlooked issue of disproportionate credit assignment in trajectory-based diffusion alignment.\n2. Technically novelty: The gradient-based proportional credit estimation is simple yet effective, bridging reinforcement learning and diffusion optimization perspectives.\n3. Strong empirical validation: Experiments on multiple backbones (SDXL, SD3.5, ModelScope) show consistent improvements over DPO, LPO, and TAPO.", "weaknesses": "1. Missing comparisons on metrics such as HPS, CLIP, Pick-a-Pic(Not reward), etc. FID and IS is not enough to evaluate the model quality.\n2.  Sensitivity & hyperparameters: Sensitivity plots exist but more systematic sweeps (schedules, batch sizes, guidance scales, LoRA ranks, SDE noise levels) can strengthen claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hbJk1ZagLx", "reviewer": "Reviewer_gCoT", "summary": "This paper proposes Proportionate Credit Policy Optimization (PCPO), a reinforcement learning framework for aligning text-to-image (T2I) generation models. The authors identify disproportionate credit assignment across diffusion or flow timesteps as a core source of instability in PPO/GRPO-style methods. This issue leads to high-variance gradients, excessive clipping, and eventual model collapse. Empirical results across diffusion and flow models: Stable Diffusion 1.4/1.5 and FLUX, demonstrate: faster convergence, improved image fidelity, and reduced model collapse.", "questions": "1. **Computation vs. Speedup.** It reports per-epoch acceleration, but omits detailed wall-clock comparisons, leaving open questions about added per-step computation.\n\n2. **Variance Quantification.**\nBeyond the clipping fraction, could the paper include explicit variance or gradient norm distributions to quantify the stability gains?\n\n3. **Baselines.** More baselines are encouraged to compare with, such as SPPO [1] and RainbowPA [2]. On the other hand, can PCPO be integrated with reward-model-free methods (e.g., self-play [3]) to further stabilize alignment training?\n\n4. $\\log \\rho_{t} \\approx \\rho_{t} - 1$ only holds when $\\rho_{t}$ is about $1$. This may fail in high-variance regions or off-policy updates, especially when sampling trajectories from older policies or noisy prompts. Could you please show the distribution of $\\rho_{t}$?\n\n\n[1] Bridging SFT and DPO for Diffusion Model Alignment with Self-Sampling Preference Optimization. arXiv:2410.05255, 2025.\n\n[2] Diffusion-RainbowPA: Improvements Integrated Preference Alignment for Diffusion-based Text-to-Image Generation. Transactions on Machine Learning Research, 2025.\n\n[3] Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation. NeurIPS 2024.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. **Quality**\n\nAblations convincingly demonstrate the cumulative contribution of PCPO’s components. Detailed appendices cover proofs, hyperparameters, and resource requirements for reproducibility.\n\n\n2. **Experimental Rigor**\n\nThe inclusion of Linear Mixed Model (LMM) statistical analyses adds robustness to the evaluation, while the human preference study further validates perceptual improvements.\n\n\n3. **Clarity**\n\nThe paper is well-organized and clearly written, despite the technical density. Visualizations, e.g., Figures 2–4, effectively illustrate instability sources and PCPO’s corrective mechanism. Mathematical derivations are complete and logically consistent.", "weaknesses": "1. **Theoretical Intuition.**\n\nWhile the proportionality argument is sound, the paper could better connect it to established temporal credit assignment or variance reduction principles in RL.\n\n\n2. **Lack of baselines.**\n\nIt is encouraged to compare with other SOTA methods. See Q3.\n\nWhile results on SD1.4/1.5 and FLUX are compelling, broader testing, e.g., on SDXL (Podell et al., 2023), is recommended to establish generality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)