Half-order Fine-Tuning for Diffusion Model: A Recursive Likelihood Ratio Optimizer
Authors: Tao Ren
Zishi Zhang
Jinyang Jiang
Zehao Li
Shentao Qin
Yi Zheng
Guanghao Li
Qianyou Sun
Yan Li
Jiafeng Liang
Xinping Li
Yijie Peng
Venue: ICLR 2026 Oral (2026)
Keywords: diffusion model
post-training
stochastic gradient estimation
Primary area: generative models

Abstract: The probabilistic diffusion model (DM), generating content by inferencing through a recursive chain structure, has emerged as a powerful framework for visual generation. After pre-training on enormous data, the model needs to be properly aligned to meet requirements for downstream applications. How to efficiently align the foundation DM is a crucial task. Contemporary methods are either based on Reinforcement Learning (RL) or truncated Backpropagation (BP). However, RL and truncated BP suffer from low sample efficiency and biased gradient estimation, respectively, resulting in limited improvement or, even worse, complete training failure. To overcome the challenges, we propose the Recursive Likelihood Ratio (RLR) optimizer, a Half-Order (HO) fine-tuning paradigm for DM. The HO gradient estimator enables the computation graph rearrangement within the recursive diffusive chain, making the RLR's gradient estimator **an unbiased one with lower variance** than other methods. We theoretically investigate the bias, variance, and convergence of our method. Extensive experiments are conducted on image and video generation to validate the superiority of the RLR. Furthermore, we propose a novel prompt technique that is natural for the RLR to achieve a synergistic effect. The implementation is available at https://github.com/RTkenny/RLR-Optimizer.

Published as a conference paper at ICLR 2026
HALF-ORDER FINE-TUNING FOR DIFFUSION MODEL: A RECURSIVE LIKELIHOOD RATIO OPTIMIZER
Tao Ren'
Zishi Zhang Jinyang Jiang' Zehao Li
Shentao Qin
Yi Zheng' Yijie Peng 166
Guanghao Li Qianyou Sun' Yan Li
Jiafeng Liang*
Xinping Li'
Peking University Tsinghua University
OriginFlow
HKUST
Harbin Institute of Technology
Xiangjiang Laboratory
A woutnn seving soking machine her cafft room.
As illattutionn ofa najestio withwit parsingg eyes.
A giil playnng with is joydal aimomplace
An antiete permntt of's presd profe
realitic photo spiler weasing the arai denib.
Matorcycle parked sicewak
Stable Diffasinn
After fime-tnnin
with frecklec har anilng rnder
Soubtii carsacos loough doctt kicking cloads ofsand.
RI RL optinized Text2Voo generatin
qpaco pinat, gaiing
ccallty rijss lops srmss nofups under moots
Figure Image and video samples generated from the model fine-tuned by our RLR optimizer.
Please refer to Appendices and for more qualitative examples.
ABSTRACT
The probabilistic diffusion model (DM), generating content by inferencing through recursive chain structure, has emerged powerful framework for visual gen- eration. After pre-training on enormous data, the model needs to be properly aligned to meet requirements for downstream applications. How to efficiently align the foundation DM is a crucial task. Contemporary methods are either based on Reinforeement Learning (RL) truncated Backpropagation (BP). However, RL and truncated BP suffer from low sample efficiency and biased gradient estimation, respectively, resulting in limited improvement or, worse, complete training fail- ure. To overcome the challenges, we propose the Recursive Likelihood Ratio (RLR) optimizer, Half-Order (HO) fine-tuning paradigm for DM. The HO gradient esti- mator enables the computation graph rearrangement within the recursive diffusive chain, making the RLR's gradient estimator an unbiased one with lower variance than other methods. We characterize the bias, variance, and convergence behavior of our method. Extensive experiments are conducted on image and video generation to validate the superiority of the R.R. Furthermore, we propose novel prompting technique that natural for the RLR to achieve a synergistic effect. The implemen- tation is available at ht http::///gttuuae http:/hwww....eeeeeeeeeeee/t//ae
Equal contribution. Project lead: rtkenny@stu.c www.puu.co... Corresponding author: pengyi ji jiekk. edu.

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Probabilistic diffusion model (DM) (Sohl-Dickstein al. 2013 Ho al. 2020: Zhang al. 2024b Gao & Li, 2025) has emerged as a transformative framework in high-fidelity data generation, demonstrating unmatched capabilities diverse applications such as image synthesis (Podell et al. 2023), video generation (Wang et al. 2023a and multi-modal data modeling. These models operate by recursively denoising latent representations, as shown in Figure 2 effectively capturing complex data distributions. However, fine-tuning DMs in the post-training phase remains daunting challenge, since the gradient estimation through the recursive structure imposes excessive computation overhead (Clark et al., (023). This challenge has limited the broader deployment of DM in dynamic and ensource-constraine environments.
It is natural way to fine-tune DMs via full backpropagation (BP) through all time steps (Rumelhart et al.) 1986), which is theoretically functional. providing precise gradient estima- tion over the entire diffusion chain. However, the computa- tional and memory overhead of BP scales prohibitively with the model size and the number of diffusion steps (Prabhudesai al.
I Latent variables
X
Parameter pertubbtiion
X
2023 Yuan et al., 2024a Clark et al. 2023 Prabhudesai
et al. 2024). ). making full BP impractical for most real-world
Naise generator
Lateuz
scenarios. Specifically, training Stable Diffusion 1.4 by full BP with batch size of 1 and 50 time steps would require approx- imately 1TB of GPU RAM (Prabhudesai al. (023). Thus, truncating recursive differentiation becomes common practice to alleviate memory overhead. But truncated BP suffers from structural bias, it terminates gradient computation before sourcing to the input, only considering a limited subset of the diffusion chain. Fine-tuning based on biased gradient esti- mation can inadvertently impair the optimization performance, resulting in model collapse, i.e., contents generated reducing to pure noise. Empirical evidence, in Figure 3 shows that truncated gradients result in significant drop in reward scores during training: the fewer the truncated time steps, the more severe the model collapse. Moreover, truncated BP fails to capture the multi-scale information across all time steps due the absence of differentiation on early steps. The hierarchical feature of generation imposes a thorough gradient evaluation to retain fidelity from pixel to structural level.
perturbttion
Figure 2: The recursive structure of diffusion models for gradient esti- mation.
24
Rowand
33
1100
120
130
175
200
Step
Figure Model collapse caused by the truncation: training SD 1.4 on the aesthetic reward model by truncated BP. dient computation trick has enabled another branch of DM fine-tuning (Lee et al. 2023 Black et al. 2023 Wallace et al. 2024; Fan al. 2024 It typically ignores the differentiable connection between steps and recovers the gradient by estimation. RL avoids caching intermediate activations, requirement requirements. It also supports gradient computation in divided manner under extreme circumstances to accommodate insufficient memory. The cost to pay is the high variance of the estimated gradient. Even if the estimator is unbiased, the variance can result in wild sample-inefficien updates, demonstrated by the slow convergence during training.
Reinforcement learning (RL) (Schulman et al. 2017
gra-
significantly reducing memory
These limitations of BP and RL underscore the necessity of more efficient, scalable, and stable fine- tuning approach that harmonizes computational tractability with optimization efficacy. To address this, we first investigate the recursive architecture of the DM and propose the problem of finding the minimal variance gradient estimator. Informed by perturbation-bate
estimation using Likelihood Ratio (LR) techniques (Tiang et af., 22024, Ren et al. 2025) (detailed reviews of LR techniques are provided in the Appendix A). we propose the Recursive Likelihood Ratio (RLR) optimizer. By utilizing the inherent noise in the DM, a local computational graph is enabled for pathwise gradient estimation shown in Figure 4 which can reduce the variance and better capture the multi-scale information. The RLR estimator is unbiased and has lower variance under the same computation budgets as other methods. Through perturbation baatd based computational graph rearrangement, the RLR mitigates the structural bias of truncated BP and the high variance RL, capable of better capturing multi-scale visual information. Our optimizer shares similarity with zeroth-order optimizer since both
2

---

Published as a conference paper at ICLR 2026
II. Panameter perturbation
Half-order Estimation
Parameter perturbation
Direct differentiation
Xy
8
R(X))
Inherent noise
Forward pass
Perturbation
Differentiation
Figure 4: The computation paradigm of the RLR optimizer.
need perturbation to estimation gradient. but the RLR utilize a local BP chain to enable low-variance estimation. Therefore, we name it as Half-Order optimizer. Our contributions are threefold:
We provide systematic analysis of gradient estimation in DMs, identifying structured design space of estimators. Then, we formulate the RLR optimizer in the design space under the protocol of minimizing variance with a limited computational budget.
Extensive evaluation of mmxe and Texr2 tasks are conducted. RLR consistently achieves higher reward scores across multiple human preference reward models and outper- forms SOTA video models on the VBench benchmark. Furthermore, we propose prompt technique for our RLR, validating the intuition and applicability of our method.
We conduct rigorous theoretical analysis of RLR's design, proving its unbiasedness, bounding its estimator variance, and establishing convergence guarantees. These results formally justify the empirical success of RLR and explain the deficiency of prior methods.
2 RELATED WORK
Diffusion probabilistic models have achieved ofate-of-the-ar performance in multi-modal generation (Ho et al., 2020 Rombach et al. (022). However, aligning pre-trained DMs for downstream tasks remains challenging. RL and supervised post-training have been widely explored to incorporate human preferences task-specific objectives, with recent methods including R[HF-inspired ap- proaches, DPO variants, and reward-moder -uidded fine-tuning (Ziegler et al.) 2019) Fan et al.) [2024] Wallace et al.) 2024 Prabhudesai al., 2023). Yet, RL-based fine-tuning often suffers from high variance and sample inefficiency, while truncated BP reduces memory cost but introduces structural bias that can lead model forward-leanee learning methods based on stochastic gradient estimation offer a promising alternative with lower computational and memory costs (Salimans et al., 2017 Peng et al. 2022, Chen et al. 2023). Please refer to Appendix A for extended related works.
al., 2023 2024 Xu et al. 2024b). Recent
3 UNBIASED MINIMAL VARIANCE GRADIENT ESTIMATOR FOR DIFFUSION MODEL
3.1 PROBLEM FORMULATION
DMs generate data by transforming a noise sample FT into a data sample By through a recursive This generation proceeds through T steps of stochastic
denoising process, as shown in Figure 2 updates, each governed by parameterized mapping éi, where the implementation of d, may vary across steps, yielding the transformation:
x1-1 = 6e 0)
(1)
where 0 denotes the model parameter, and 2, = Oztr, ~ NO /) is a stochastic perturbation. The noise term o can be either the inherent noise in the DM, i.e., _T-1 (p(x;00) + 2t, or the injected noise to the parameter, i.e., Tt-1 = ( f + 21). The function yo denotes the shared backbone network of the DM, reused across different steps, while oi should be used with the subscript to indicate which time-step it operates on. The full generation chain can be represented recursively as:
Ip (IT, = Φi o th Φ (12 22:; 0), 215 0) = ( -;
ΦT 2):;;)-
(2)
3

---

Published as a conference paper at ICLR 2026
where 21:7 independent noise terms injected at each step. Once pre- pre-trained model is available, a reward model R(zo) is introduced to guide post-training towards generating samples of higher semantic or perceptual quality. The resulting fine-tuning objective becomes:
max E[[(2))) max E [R( (1:7; 0))]
Fine-tuning DMs through the recursive structure poses a critical challenge: how to efficiently and accurately estimate the gradient of the expected reward with respect model parameters. A natural objective is to construct a gradient estimator for [[[(r))] that is unhiased and has minimal variance, under given computational and memory budget. We formulate the problem the following constrained optimization problem over the space of gradient estimators:
min GEO
Var(G)
S.t.
VeLE[R(Zo)] = [[G],
C(G) S B,
(4)
where G denotes the space of all unbiased gradient estimators G for the objective ([](( with the sample zy being obtained via generative chain defined by 3o Φ1: (#r; 21:7: f). The term Var(G) refers to the variance of the estimator, which we aim to minimize. The cost function C(G) quantifies the total computational and memory overhead incurred by the estimator G. typically measured in terms of the length of backward passes or the volume of intermediate activations stored. Finally, B represents the computational budget available for gradient estimation.
3.2 G: THE UNBIASED GRADIENT ESTIMATOR DESIGN SPACE IN DIFFUSION MODELS
We now characterize the feasible design space G for unbiased gradient estimators in DMs. Due to the recursive structure of the generative chain, the gradient estimator must propagate through all T steps. At each step I, we may choose one of three gradient estimation strategies:
First-Order (FO) uses exact backpropagation through di. Zeroth-Order (ZO) perturbs the parameters directly, e.g., (x, 0 + ofGe), and estimates the gradient via [( (((eee))))))))) Ct. based solely on function evaluations (Salimans et al. 2017).
We propose unbiased Half-Order (HO) gradient estimator which utilizes the inherent noise (rather than extrinsic perturbation) and applies the Likelihood Ratio technique, producing an estimator of the form: R(zu) DJ du the Jacobian of a local sub- sub-chain of length h. and f()) denotes the noise density. The HO
log f2)), where DoQu.+hh11 denotes
allows a h-length sub-chain, starting from any the diffusive chain. RI is special case of HO method with h = 1.
The full estimator design space thus defined as:
Grmll : (G (g) 191......77. .9)) ge E (FO, HO, Zo} STST),
(5)
where each sequence G E Grm represents a com- posite estimator composed of local choices at each time step. Table summarresss the variance, unbiasedness, and memory costs of each strat- egy: FO has the lowest variance but highest cost, ZO is the cheapest but suffers from the highest variance, and HO balances the two. The theoretical underpinnings of these variances. unbiasedness, and memory comparisons are formalized in Section 6 and constitute an independent contribution of this work.
Table 1: Comparison of gradient estimators.
Method
Unbiasedness
Variance
Memory
First-order Half-order Zeroth-arder Truncated BP
Small Large
Medium
Medium
Large Small
Small
Medium
SOLVING THE VARIANCE MINIMIZATION PROBLEM: RECURSIVE LIKELIHOOD RATIO OPTIMIZER
4.1 RECURSIVE LIKELIHOOD RATIO OPTIMIZER
Having characterized the decision space Grall- now turn to solving the variance minimiza- tion problem (4. Due to practical constraints, we further reduce the full design space by en- forcing the following structure: (i) All HO estimators should be connected in the chain, since
4

---

Published as a conference paper at ICLR 2026
separating the HO path would incur high variance. (ii) FO should be directly connected to the reward model, according to its definition. This way, the decision space Paul is reduce to - .... 9T | jisj S T h), which consists of one FO at the first step, HO sub-chain of length h starting at step and ZO estimators for all remaining steps to ensure unbiasedness. Under this specific structure, the decision variables are reduced to the length h and the starting index j. Each solution in GRLK is referred estimator (see Figure 4). which takes the form
GRlR {(g .9
Recursive Likelihood Ratio (RLR)
G = p(x1, dR(
dro
One One-stop order estimator
(6)
Da 25gth 2g9h h lenghh oordrr estimator
Σ Vn f(z); iEC
zeroth order estimator
where j E - 1 Mnz, C (0,,o,1) foriEC.
T {5.j + 1,
+ h), ~ N(0, and
Differentiating the reward model. In the first part of the RLR estimator, we apply the FO estimator to the first time step to directly backpropagate through the reward model, avoiding black-box treatment (e.g., RL and ZO) and better leveraging its structure, as shown in Figure
Fixed-horizon half-order optimization. The generation process of the DM follows coarse-to-fine structure, with every time step in the chain controlling different scale of generation. Incorporat- ing precise gradient information from every time step essential, but full BP is computationally prohibitive. Truncated BP introduces bias, while ZO and RL lead to high variance by ignoring structural information. To address this, the RIR optimizer incorporates an HO h-length sub-chain, capturing multi-scale information while minimizing variance. Specifically, the starting index of HO, j ~ J (1, -' h). is randomly selected across the whole diffusive chain, following a given distribu- tion J (see Section 3.2). The inherent perturbation, 2j. enables the localized ht-lngth sub-chain, Doc effectively capturing the visual scale information represented around that step (see Section 3.2 for choosing h).
Surrogate estimator via parameter perturbation. For the remaining times steps, C 21, 7} b,...,,...... j f1,, we inject noise directly into the model's parameters to construct ZO estimation ensure unbiasedness. This approach is computationally cheap without caching intermediate latent variables.
4.2 OPTIMIZING h AND j: VARIANCE-MEMORY TRADEOFFS
The remaining task is to optimize the two variables in the RLR estimator, h and to solve the optimization problem (4). Notably, the choice of j does not directly affect this surrogate objective, but instead influences the ability capture multi-scale information across different steps, treat its decision as a separate problem of interest.
Optimizing h. To reduce the number of problem parameters that need to be estimated, we use an upper bound on the variance of the RLR estimator (see the Appendix (.4) as a surrogate objective:
T
min
Var(g:) + Σ
/Vaargg)Var(gr>
kgNo: G(R)EGRLR
t=1
Byh B2
h) B.
where and (T (T-1-hi are the number steps for HO and ZO; B₁ and B, are coefficients indicating the magnitude of the memory cost of HO and ZO per step. In practice, the available budget satisfies B.T < B < B,T, meaning that using pure ZO underutilizes the budget, while using pure HO exceeds it. Let V2 and V denote the per-step variance of the HO and 70, respectively. Since HO
5

---

Published as a conference paper at ICLR 2026
and FO have much lower variance than ZO, common Vi, for both HO and FO, and assume V1. < V2 and > These conditions ensure the optimization problem admits the solution:
B
B (T-1)
TV. 2(V - V..)
h- = min
1J3 > 0.
(8)
B B=
We set B₁ = 8GB and B = 0.24GB, which is supported by empirical evidence in Table i the Appendix. the memory budget B is between 30GB and 40GB. It recommended set / As the formula exhibits diminishing improvement with increasing h. In other aspects, the memory consumption grows linearly with h, and the computational time grows even more rapidly. The above claims are corroborated by our ablation results in Table [ Therefore, even with larger memory budget, blindly increasing h is not advisable. Moreover, since V4, < V., the second term in (8) simplifies 24, which typically larger than the first term. As result, have h" = [B - (T 1)) B.J in practice, and there is no need to estimate V/, and V.,
7 indicates, the variance decreases as / increases. However, the performance
to approximately 2
Determining j. We use the gradient norm to represent the importance of different steps. Then sample j from the categorical distribution j J(1, h) = (SAT(So (Softmax((1991)).. 11
EXPERIMENTS
We verify the superiority and upplicability of the RLR optimizer against various baselines on two generation tasks: Text2lmage and Tex12Video. We compare the RLR with the RL-based method (DDPO), and the truncated-BP-ue -asbeed methods (Aligmprop and VADER). Moreover, other baselines, e.g. closed-source models, are also included. Finally, we propose a novel prompt technique that natural for the RLR optimizer, demonstrating the enhanced capability of the proposed RLR optimizer to capture multi-scale information for visual generation. Ablations are included to verify the validity of the proposed RLR optimizer. Please refer to Appendix B for detailed settings.
Table 2: Text to Image reward score. We evaluate methods under different DM under different reward
models. The higher the score, the better the performance.
Pick-a-PP Pic AES
HPD v2
Model
Methods
PickScore
HPSv2
ImageReward
PickScore
HPSv2
AES
ImageReward
Base
16.24
21.03
4.48
32.74
16.19
22.08
4.42
32.90
DDPO
17.56
23.15
5.47
49.33
17.53
22.79
5.52
52,06
SDL.4
Alignprop
18.08
26.64
5,91
65.07
19.17
27.02
6.02
67.18
RIR
20.14
28.57
6.53
75.65
21.38
29.22
6.65
76.55
Base
16.37
22.14
4.53
35.40
16.25
23.32
4.57
36.03
DDPO
17.70
24.55
5.58
52.48
17.43
24.56
5.62
52.85
SD2.1
Alignprop 19.23
27.20
6.07
68.09
21.60
27.40
6.11
72.62
RLR
22.58
30.11
77.26
23.22
30.98
6.74
83.07
Table 3: Text2Video Generation Evaluation the Vbench. The weighted average is calculated by assigning a weight of to all metrics, except for the Dynamic Degree metric, which is assigned weight of 0.5.
Subject Consistency
Background Consistency
Motion Smoothness
Dynamic Degree
Aesthetic Quality
Imaging Quality
Weighted Average
Methods
VideoCrafter
95.44
96.52
96.88
53.46
57.52
66.77
79.97
Pika
96.76
98.95
99.51
37.22
63.15
62.33
79.87
Gen-2
97.61
97.61
99.58
18.89
66.96
67.42
79.75
T2V-Turbo
96.28
97.02
97.34
49.17
63.04
72.49
81.96
DOODL
95.47
96.57
96.84
55.46
58.27
66.79
80.30
DDPO
95.53
96.63
96.92
58.29
59.23
66.84
80.78
VADER
95.79
96.71
97.06
66.94
66.04
69.93
83.45
RLR
97.64
97.19
98.05
70.69
66.15
71.08
84.63

---

Published as a conference paper at ICLR 2026
DETO
DIPO Alaatror HLLI
Xlgnfrr HLLtt
Arsibett soxe
11P5v2 Scots
(a) Reward curves for aesthetic score.
(b) Reward curves for HPS v2 score.
Figure 5: Sample efficiency analysis, reward curves under the same training steps for SD 1.4.
5.1 TEXT2IMAGE GENERATION
We evaluate our methods on two DMs: Stable Diffusion 1.4 and 2.1 (Rombach et al. 2022). As shown in Table the RLR methods achieve higher reward scores on the unseen prompts from the test set. The RL-based method have limited improvement with respect to the base model, due to the sample inefficiency nature. Aligmprop has considerable improvement over the base model. However, the biased estimator limits its further improvement. Training details and hyperparameters can be found in the appendix.
Sample efficiency analysis. The compare the sample efficiency and the variance of different methods, we show the reward curves of SD 1.4 when training on the AES and HPS v2 models in Figure The DDPO optimizes the reward sample efficiency. In the earlier phase, the AlignProp has comparable performance as the RLR. In the later phase, while the RLR can continue to improve the reward, the AlignProp suffers from severe model collapse.
very slow pace, indicating high variance and low
5.2 TEXT2VIDEO GENERATION
We compare our RLR not only with RL and truncated BP but also with a series of open-source or API-based Text2 models. In the metric of DD and AQ. the RLR surpasses other methods by a large margin. In other metrics, RLR achieves considerable improvement over the base model, VideoCrafter. Some API-based models have better performance small. In terms of the weighted average score, our RLR has the best performance over all baselines.
metrics, but the gaps
5.3 DIFFUSIVE CHAIN-OF-
Original Prompi
Multi-orale prompis
premp!
prampt
prompt
R(x,)
Shifling window
longth
Figure 6: The framework of Diffusive Chain-of Thought. Thoight. The DM generates images in multi-scale manner: earlier steps for low-resolution features and later steps for high-resolution features. If specific scale has deficiencies, we utilize the HO estimator to enhance the corresponding steps.
Furthermore, we propose the Diffusive Chain-of- Thought (DCoT), prompt technique that is natural for our RLR optimizer to demonstrate the applicability of our method. The core idea is that DMs generate content in a multi-scale (coarse-to-fine) manner, and deficiencies at particular scale can be addressed by focusing gradient updates on the corresponding steps of the diffusion process by the HO sub-chain. We propose dividing all the diffusion process steps into three groups: coarse-level, mid-level, and fine-level. The coarse-level chain includes steps adjacent to the initial noise, focusing

---

Published as a conference paper at ICLR 2026
generating rough outline. The fine-level chain includes steps adjacent to the final output, focusing on the fine- geee-grrindd details. The mid-level chain in between focuses on the geometric structure the content.
The idea of DCoT is shown Figure 9 which converts the original prompt into multi-scale prompts reflecting the coarse-to-fine nature. Dif- ferent generation steps should be conditioned different prompts instead of being conditioned on the same prompt. The HO estimator term in the RLR enables a h-step local computational chain for low-van oow-variance, unbiased gradient esti- mation. By integrating DCoTT, we can target the HO sub-chain precisely at the time steps (i.e., scales) where generation is deficient, as revealed by the multi-scale prompt decomposition. The HO estimator term uniformly picks a starting point j ~ (1, T h) from the entire T T'-step chain to start the local h-step BP chain. When applying the DCoT to the fine-tuning process, we should constrain the sample (L b), experiment for the hand task, we set a = 30 and b = 40.
Table 4: Experiment results for Diffusive Chain- of of-Thought
Methods
PickScore
HPSv2
AES
ImageReward
SD1.4
15.33
20.50
4.76
33.45
SDI,4-DCoT
16,38
21.68
4.83
39.24
RLR w/o
18.05
23.58
5.27
43.09
RLR-DCoT 19.45
25.80
5.83
49.88
range of in the steps where deficiencies exist,
<<<
h. In our h, b
We write 5 prompts for hand generation and then prompt ChatGPT to generate the multi-scale prompts for the three levels. We report the performance in Table 4 As shown in the table, either simply applying DCoT the base model combining it with the RLR improve the performance significantly. The RLR with the DCoT has the best performance. Qualitative results are shown in the Appendix
5.4 ABLATION STUDY
We conduct the ablation study, using SD 1.4 and HPD v2, verify the con- tribution of different parts in the RLR optimizer. Table we evaluate the RLR and its variants (V1: the RLR without HO and ZO; V2: the RLR
Table 5: Ablation of the RLR.
Methods
PickScore
HPSv2
AES
ImageReward
RLR w/o HO & ZO
18.43
23.66
5.78
60.07
RLR w/o ZO
20.11
27.07
6.23
68.35
RLR w/o HO RLR
19.28
26.70
5.92
63.85
without 70; V3: the RLR without HO). The VI performs the worst since it actually reduces to the truncated BP with only one time-step. The V2 and V3 perform better than the V1. It is worth noting that the V2 is better than the V3. The V3 without HO is actually an unbiased estimator since it takes all time steps into account when estimating the gradient. Even though the V2 rearranges the computational graph by the HO, it is still a biased estimator. This phenomenon indicates the importance of unbiasedness when conducting the fine-tuning task.
21.38
29.22
6.65
76.55
THEORETICAL PROPERTIES OF GRADIENT ESTIMATORS: BIAS, VARIANCE, AND CONVERGENCE
In this part, we analyze the bias of truncated BP and compare the variance of different estimators, backing the claim Table [ Thanks the unbiasedness of the RLR estimator, the convergence of the optimization is also guaranteed.
To alleviate the memory burden of full-step BP, the truncated variant is often employed, backpropa- gating the gradient with only T' steps; T" < T. However, the truncation introduces a structural bias into the gradient estimator. We have the following proposition to justify this structural bias.
Proposition 6.1 (Biasedness of Truncated-BP ). Assume R and & are differentiable almost everywhere, R and o, have bounded gradients, then the FO estimator unbiased. However, the truncated BP estimator V R( truncated has a structural bias, which can be specified as below:
VOE|R(zo)] E (Vol((o)rrunnateel EETr
Σ
de 21:0)
Dzz-1
T ddR2zo)
00
0x,
dao
8

---

Published as a conference paper at ICLR 2026
Bias in the estimator lead to suboptimal updates may not follow a true descent direction. This can cause two major issues: model collapse and loss of multi-scale information. In contrast, ZO (Spall 1992) and HO (Jiang et al. 2024) are unbiased.
training failure, the truncated gradient
The stochastic nature the DM results the variance of the estimator. As expected, the variance of the FO estimator is lower than that of the HO and ZO estimators because the differentiation leverages the structural information the neural network. However, BP introduces significant computational and storage overhead. The following proposition demonstrates that this additional cost is, some extent, justified, as BP leverages accurate internal structures to reduce estimation variance.
Proposition 6.2 (Variance Comparison). Under Assumptions (A.1-3) in the Appendix, the variance of FO estimators is less than or equal to 70 estimators, i.e.
Var V V&R(zo)) S Vl
Based on the above proposition, is straightforward to conclude that the variance of the HO estimator also less than or equal to that of the ZO estimator, as it is essentially an FO estimator with perturbation at the start of the sub-chain. The Table presents all the gradient computation methods discussed above.
Overall, the RLR reorganizes the recursive computation chain by perturbation- based estimation, seamlessly integrating ZO, HO, and FO optimization techniques. RLR strikes a balance between computational cost and gradient accuracy, achieving both unbiasedness and low variance. The following Theorem 6.3 establishes its unbiasedness.
Theorem 6.3 6.3 (Unbiasedness of RLR). The RLR estimator is an unbiased estimator:
v (2T; 0)) = E - Dj 01 (21, dl(z))
dx
(10)
- R((x)) DJ (osth(
0 V) - > R( In
REC
The variance the RLR estimator, denoted by HRLR- is discussed in the appendix. Under limited computational resources where full BP is infeasible, RLR achieves substantially lower variance compared to other unbiased gradient estimators. Finally, the convergence rate of RLR is provided in the following Theorem 6.4
Theorem 6.4 (Convergence Rate). Suppose that the reward function R(-) is L-smooth. By appropri- ately selecting the step size, the convergence rate of the RLR is given by
1 K+I
EEIVRON) K=0
SLAOO
RLR
2LAb
K+1
K
where K is the number of iterations, 0% is the trainable parameter in the k-th iteration, and Do [R(0g) - R² difference between initialization performance and optimal performance.
CONCLUSION
We propose the RLR optimizer, half-order gradient estimation framework designed for efficient fine- tuning of diffusion models. Theoretically, we analyze the bias, variance, and convergence the RLR estimator and formulate a constrained optimization problem to guide its design. Empirically, RLR consistently outperforms both reinforcement learning and truncated backpropagation methods on Texi2Image and Text2 Video tasks across multiple human preference reward models and benchmarks. Furthermore, we introduce a novel prompt technique, Diffusive Chain-of-Thhoule (DCoT), which complements the RLR and further boosts performance. Although determining the appropriate sub- chain length h can be nontrivial in practice, we provide both theoretical justification and empirical ablations guide practitioners in making informed choices.

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported in part by the National Natural Science Foundation of China (NSFC) under Grants 72325007, and 72250065, and the Science and Technology Innovation Program Hunan Province under Grant 2024RC7003.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint (1XXh:2333308774 2023.
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1728-1738, 2021.
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305. 13301, 2023.
Aochuan Chen, Yimeng Zhang, Jinghan Jia, James Diffenderfer, Jiancheng Liu, Konstantinos Parasyris, Yihua Zhang, Zheng Zhang, Bhavya Kailkhura, and Sijia Liu. Deepzero: Scaling up zeroth-order optimization for deep model training. arXiv preprint arXiv:2310.020 0.00222 2023.
Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7310-7320, 2024a.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint 164.v:160.066174 2016.
Yiming Chen, Yuan Zhang, Liyuan Cao, Kun Yuan, and Zaiwen Wen. Enhancing zeroth-order fine-tuning for language models with low-rank structures. arXiv preprint arXiv:2410.07698, 2024b.
Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. Directly fine-tuning diffusion models on differentiable rewards. arXiv preprint arXiv:2309. 2023.
Zhenyu Cui, Michael C Fu, Jian-Qiang Hu, Yanchu Liu, Yijie Peng, and Lingjiong Zhu. On the variance of single-run unbiased stochastic derivative estimators. INFORMS Journal Computing, 32(2))390-407, 2020.
Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Reinforcement learning for fine- tuning text-to-image diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
Weiguo Gao and Ming Li. Toward theoretical insights into diffusion trajectory distillation via operator merging. arXiv preprint arXiv:2505. 16024, 2025.
Paul Glasserman. Gradient estimation via perturbation analysis, volume 1 116. Springer Science & Business Media, 1990.
Audrunas Gruslys, Rémi Munos, Ivo Danihelka, Mare Lanetot, and Alex Graves. Memory-efficiee backpropagation through time. Advances in neural information processing systems, 29, 2016.
Xuan He, Dongfu Jiang, Ge Zhang, Max Ku, Achint Soni, Sherman Siu, Haonan Chen, Abhranil Chandra, Ziyan Jiang, Aaran Arulraj, et al. Videoscore: Building automatic metrics to simulate fine-grained human feedback for video generation. arXiv preprint arXiv:2406. 15252, 2024.
Geoffrey Hinton. The forward-forwae algorithm: Some preliminary investigations. arXiv preprint arXiv:2212. 13345, 2022.
10

---

Published as a conference paper at ICLR 2026
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 15:6840-6851 2020.
Ziqi Huang. Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, al. Vbench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 21 2187-21888, 2024.
Jinyang Jiang, Zeliang Zhang, Chenliang Xu, Zhaofei Yu, and Yijie Peng. One forward is enough for neural network training via likelihood ratio method. In The Twelfih International Conference on Learning Representations, 2024.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint ar arXiv:2001.08361, 2020.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in neural information processing systems, 04226665-267777 2022.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-pie: An open dataset of user preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36:36652-36663, 2023.
Nathan Lambert, Louis Castricato, Leandro von Werra, and Alex Havrilla. Illustrating reinforcement learning from human feedback (rlhf). Hugging Face Blog, 2022. https//tugginngaacc.o/bl//g////
Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint 2r0iv 12192, 2023.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in neural information processing systems, 35:5775-5787, 2022.
Haoming Lu, Hazarapet Tunanyan, Kai Wang, Shant Navasardyan, Zhangyang Wang, and Humphrey Shi. Specialist diffusion: Plug-and-play sample-eflicient fine-tuning of text-to-image diffusion models to learn any unseen style. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14267-14276, 2023.
Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning language models with just forward passes. Advances in Neural Information Processing Systems, 36:53038-555 -3075,,53355--10 2023.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195 4205,,2055550 2023.
Yijie Peng, Li Xiao, Bernd Heidergott, Jeff Hong, and Henry Lam. new likelihood ratio method for training artificial neural networks. INFORMS Journal on Computing, 34(1):638-655 2022.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.07952 2023.
Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning text-to-image diffusion models with reward backpropagation. arXiv preprint arXiv:2310.037.9 2023,
Mihir Prabhudesai, Russell Mendonca, Zheyang Qin, Katerina Fragkiadaki, and Deepak Pathak. Video diffusion alignment via reward gradients. arXiv preprint arXiv:2407.08737, 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.

---

Published as a conference paper at ICLR 2026
Tao Ren, Zishi Zhang, Jinyang Jiang, Guanghao Li, Zeliang Zhang, Mingqian Feng, and Yijie Peng. Flops: Forward learning with optimal sampling. In The Thirteenth International Conference on Learning Representations, 2025.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
W. Rudin. Real and Complex Analysis. Mathematics series. McGraw-Hill, 1987. ISBN 9780071002769, Utthttss: :/tookss////os...o httpbooo..oogge..ooooooooooog..p com. hwtp.hh...ooooodoooo.. Non.okkkSs...NNN////AAAAAA
David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning representations by
back-propagating errors. nature, 323(6689) 1986.
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as scalable alternative to reinforcement learning. arXiv preprint arXiv:1703.0386/ 2017.
Christoph Schuhmann. Improved aesthetic predictor, 2022.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-25299 2022.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 17X1v:1707.06347, 2017.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. International conference on machine learning, pp. 2265-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020a.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13455 2020b.
James Spall. Multivariate stochastic approximation using a simultaneous perturbation gradient approximation. IEEE transactions on automatic control, 37(3)3332-34, 1992.
A Vaswani. Attention all you need. Advances Neural Information Processing Systems, 2017.
Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shatiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8228 8238, 2024.
Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang. Xiang Wang, and Shiwei Zhang. Mod- elscope text-to-video technical report. arXiv preprint arXiv:2308. 06571, 2023a.
Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, al. Internvid: A large-scale video-text dataset for multimodal understanding and generation. arXiv preprint 1rXXi:2307.06943 2023b.
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306.09334 00340,66094, 2023.
Jiazheng Xu, Yu Huang, Jiale Cheng. Yuanming Yang, Jiajun Xu, Yuan Wang, Wenbo Duan, Shen Yang. Qunlin Jin, Shurun Li, et al. Visionreward: Fine- tree-graied multi dimens-aimesiionall human preference learning for image and video generation. arXiv preprint arXiv:24/2.2 2024a.
12

---

Published as a conference paper at ICLR 2026
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024b.
Kai Yang, Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human feedback to fine-tune diffusion models without any reward model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8941-89511 2024.
Hangjie Yuan, Shiwei Zhang, Xiang Wang, Yujie Wei, Tao Feng, Yining Pan, Yingya Zhang, Ziwei Liu, Samuel Albanie, and Dong Ni. Instructvideo: instructing video diffusion models with human feedback. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 6463-6474, 2024a.
Huizhuo Yuan, Zixiang Chen, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning of diffusion models for text-to-image generation. Advances in Neural Information Processing Systems, 37: 73366-73398, 2024b.
Yihua Zhang, Pingzhi Li, Junyuan Hong, Jiaxiang Li, Yimeng Zhang, Wenqing Zheng, Pin- Yu Chen, Jason D Lee, Wotao Yin, Mingyi Hong, et al. Revisiting zeroth-order optimization for memory-efficient Ilm fine-tuning: A benchmark. arXiv preprint arXiv:2402. 1192, 2024a.
Zhenyi Zhang, Tiejun Li, and Peijie Zhou. Learning stochastic dynamics from snapshots through regularized unbalanced optimal transport. arXiv preprint arXiv:2410.00844, 2024b.
Yanjun Zhao, Sizhe Dang, Haishan Ye, Guang Dai, Yi Qian, and Ivor W Tsang. Second-order fine-tuning without pain for Ilms: A hessian informed zeroth-order optimizer. arXiv preprint arXiv:2402.15173, 2024.
Daniel Ziegler, Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint 1rXiv:1909.00: 2019.
13

---

Published as a conference paper at ICLR 2026
APPENDIX CONTENTS
Extended Related Works
15
Experiments settings
15
B.
Overall Setting
15
B.2
Prompts
16
B
3
Reward Models and Avaluation Metrics
16
4
Baselines
16
B.5
Orthogonal tricks
17
Hypeparameters
17
D
Qualitative Results of Text2lmage
18
Qualitative Results of Text2Vider
19
Details for Diffusive Chain-of-Thene Thought experiment
20
Memory Profile, Time Complexity, and Different Diffusion Solvers
21
Ablations on the Sub-chain Length h
22
More Results on Vhench
22
Proofs
23
J.1 Proof of Proposition 6.
23
J.2
Proof of Proposition 6.2 and Additional Assumptions
................................
24
Proof of Theerrm 6.3
24
4
Variance of the RLR Estimator
26
0.3
Proof of Theorem 6.4
27
J.6
Solution of Optimization Problem (7)
27
14

---

Published as a conference paper at ICLR 2026
A EXTENDED RELATED WORKS
Difflusion Probabilistic Models. Denoising Diffusion Model (Ho et al. 2020 Lu ct al. 2022) is one
of the strongest models for generation tasks, especially for visual generation (Rombach al. 2022
Peebles & Xie 2023 Chen et al. 2024a). Extensive research has been conducted from theoretical and empirical perspectives (Song et al. 2020al Karras al., [022 [Song et al. 2020b). It has achieved phenomenal success in multi-modality generation, including image, video, audio, and 3D shapes. The
DM trained on enormous images and videos from the internet (Bain et al. 2021 Wang al.) 2023b
Schuhmann et al.) 2022). Empowered by modern architecture (Vaswani
2017 it has a powerful
learning capability for Pixel Space Distribution.
Alignment and Post-training. After pre-training to learn the distribution of the targeted modality (Achiam et al. 2023 Kaplan et al. (2020), post-training is conducted to align the model toward specific preferences or tune the model to optimize a particular objective. RI. has been utilized to align
the foundation models toward various objectives (Ziegler et al., 2019 Lambert et al. [2022] Black
et al. 2023). DPOK (Fan et al. 2024) studies KI. regularization when training a separate DM for each prompt. Supervised learning can also be applied to the post-training phase (Rafailov al. 2024).
either optimizing an equivalent objective (Wallace et al. (0224) or directly differentiating the reward
model (Clark et al., 2023 Prabhudesai et al. [023]] 2024). D3PO Yang et al. (2024) utilizes the DPO loss to train the DM. Specialist Diffusion Lu et al. (2023) focuses on sample-efficiee
few-shot fine-tuning of large pre-trained diffusion models enable the generation of new visual styles from as few as 5-10 examples. SPIN (Yuan et al.] 2024b) introduces a self-play learning paradigm for diffusion models. For DM, most methods use neural reward model to align the pre-trained model,
and there has been continual effort to design better reward models (He et al. 20024 Xu et al. b).
Forward Learning Methods.
After extensive exploration of model training using forward infer-
ences only (see, e.g., |eng et al. 2022 Hinton 2022), forward-learning methods (Salimans et al.)
2017: Malladi al., 2023 Chen et al.
2023 Jiang al. 2024) based stochastic gradient estima- tion have recently emerged as promising alternative to classical BP for large-scale machine learning
problems (Zhao et 2024 Zhang et al. 2024a). Subsequent research (Ren 2025 Chen
et al. 2024b) has further optimized computational and storage overhead from various perspectives, achieving greater efficiency.
B EXPERIMENTS SETTINGS
B.1 OVERALL SETTING
Prompts dataset. We use Pick-a-Pic (Kirstain et al. 2023) and HPD v2 (Wu et al. 2023) for the Texr2lmage experiments. We report the performance of the trained model on unseen prompts from the training phase. For the Text2Video task, we prompt ChatGPT to generate series of prompts that describe motions and train models under the prompts. After the training, we evaluate the model's performance on the unseen prompts from the VBench (Huang et al. 2024).
Reward model and benchmark. We adopt multiple human preference reward models to train and test our methods, including PickScore (Kirstain et al. 2023), HPS v2 (Wu et al. 2023), and ImageReward (Xu al. 2024b). All the models are trained large-scale preference datasets. We also included the traditional aesthetic models, e.g., AES (Schuhmann 2022).
For the video generation task, we use the VBench (Huang et al. 2024 to the methods according to various perspectives of the generated videos. We report 6 aspects of the generated videos in the main text: Subject Consistency (SC), Background Consistency (BC), Motion Smoothness (MS), Dynamic Degree (DD), Aesthetic Quality (AQ), and Imaging Quality (IQ). For more results of 16 metrics on VBench, please refer to Table [0 the appendix.
Baselines. We compared our methods with RL-based methods and BP-based methods. DDPO (Black et al. 2023) is the state-of-the-ar RL method for DMs. For the Text2lmage experiment, include the AlignProp (Prabhudesai et al. 2023 randomized truncated BP method. For
15

---

Published as a conference paper at ICLR 2026
the Video experiment, we included VADER (Prabhudesai et al. especially catering to video generation.
2024 BP-based method
B.2 PROMPTS
HPD v2. Human Preference Dataset v2 (HPD v2) is a large-scale collection designed to evaluate human preferences for images generated from text prompts, comprising 798,090 numan-annotated preference choices from 433,760 image pairs. It includes images from diverse text-to-image models and utilizes cleaned prompts, processed with ChatGPT to remove biases and style-related words.
Pick-a-Pic. The Pick-a-Pic dataset is a publicly available collection of over half million human preferences for images generated from 35,000 text prompts. Users generate images using state- of-the-art text-to-a to--iigiigg models and select their preferred image from pairs, with each example including a prompt, two images, and a preference label. Collected via a web application, this dataset preferences and is used to train the PickScore scoring function, which
better reflects real-world enhances model evaluation and improvement.
ChatGPT Created Prompts. We ask ChatGPT to generate imaginative and detailed text descriptions for various scenarios, including people engaging in sports, animals wearing clothes, and animals playing musical instruments. We the ChaiGPT-gennratee prompts train the model.
Vbench Prompt Suite. The Prompt Suite in VBench is a carefully curated set of text prompts designed evaluate video generation models across 16 distinct evaluation dimensions. Each dimension is represented by approximately 100 prompts tailored to test specific aspects of video quality and consistency. The prompts are organized reflect different categories, such animals, architecture, human actions, and scenery, ensuring comprehensive coverage across diverse content types. These prompts are used to assess models' abilities, such as subject consistency, object class generation, motion smoothness, and more, providing insights into the models' strengths and weaknesses across various scenarios.
B.3 REWARD MODELS AND EVALUATION METRICS
PickScore. The PickScore Reward Model is a scoring function trained on the Pick-a-Pic dataset, which includes human preferences for rext-to-image generated images (Kirstain et al. (023). It uses a CLIP-based architecture to compute scores by comparing text and image representations. Trained to predict user preferences, minimizes Kl--divergence between true preferences (preferred image or tie) and predicted scores.
HPSv2. Human Preference Score v2 (HPSv2) (Wu et al., 2023) is model designed evaluate human preferences for images generated by text-to-image models. Trained on the Human Preference Dataset v2, which includes 798,000 human preference annotations on 433,760 image pairs from various generative models, HPSv2 predicts which images are preferred based on text-image alignment and aesthetic quality, offering a more human-alige alligndd evaluation compared to traditional metrics like Inception Score or Fréchet Inception Distance.
AES. The Aesthetic Score (AES) (Schuhmann) 2022) is obtained from a model that builds on CLIP embeddings and incorporates additional multilayer perceptron (MLP) layers to capture the visual attractiveness of images. This metric serves as tool for assessing the aesthetic quality of generated images, offering insights into how closely they match human aesthetic preferences.
ImageReward ImageReward (Xu et al. 2024b) is a reward model designed to evaluate human preferences in text-to-image image generation. It is trained using a systematic annotation pipeline that rates and ranks images based on alignment with text, fidelity, and harmlessness. Built on the BLIP model, ImageReward accurately predicts human preferences, outperforming models like CLIP, Aesthetic, and BLIP. It serves as a promising automatic evaluation metric for text-to- eet-to-image synthesis, aligning well with human rankings.
large dataset of 137k expert comparisons,
B.4 BASELINES
DOODL. DOODL (Direct Optimization of Diffusion Latents) optimizes diffusion latents to improve image generation by directly adjusting latents based model-based loss. Unlike traditional
16

---

Published as a conference paper at ICLR 2026
classifier guidance methods, DOODL avoids the need for retraining models using approximations, providing more accurate and efficient guidance. It enhances text-conditioned generation, expands pre-trained model vocabularies, enables personalized image generation, and improves aesthetic quality, offering better control and higher-quality outputs in generative image models.
DDPO. DDPO (Denoising Diffusion Policy Optimization) is an RL-based method for optimizing diffusion models towards specific goals like image quality or compressibility. By treating denoising as multi-step decision-making task, DDPO uses policy gradients to maximize reward function, unlike traditional likelihood-based methods. DDPO also shows strong generalization across diverse prompts, making it highly effective for fine-tuning generative models.
Alig AlignProp fine-tunes text-to-image diffusion models by backpropagating gradients through the entire denoising process using randomized truncated backpropagation. This method reduces memory and computational costs by employing low-rank adapter modules and gradient checkpointing. The randomized TBTT approach, which randomly selects the number of backpropa- gation steps, prevents overfitting and mode collapse, improving both sample efficiency and reward optimization. AlignProp outperforms other methods terms of generalization, image-text alignment, and aesthetic quality, making it a highly efficient and effective tool for optimizing diffusion models to specific downstream objectives.
VADER. VADER (Video Diffusion via Reward Gradients) fine-tunes video diffusion models by backpropagating gradients from pre-trained reward models. It enhances sample and computational efficiency, using reward models assess aesthetics, text-video alignment, and other video-specific tasks. VADER maintains temporal consistency and generalizes well to unseen prompts, making it an effective tool for adapting video models to complex objectives.
B.5 ORTHIOGONAL TRICKS
LoRA applies low-rank adaptation to the original parameters, 8, by fine-tuning only the low-rank components rather than the full parameters. Specifically, each linear layer in the backbone (U-Net or Transformer) of the diffusion model is modified as h Wx Axx, where W E Rmxm A E m and R with à < m. The LoRA weights are initialized to zero, ensuring no initial impact on the pre-trained model's performance. This method reduces the number of parameters to be trained while achieving performance comparable to -allmpareetter fine-tuning. We apply LoRA with & = 16 to all experiments.
Gradient checkpointing is a well-known technique for reducing memory usage during neural network training (Gruslys al.. |2166 Chen et al. (016)) Instead of storing all intermediate activations for backpropagation, it selectively saves only those needed for gradient computation and transfers the rest the CPU's main memory. However, this comes with the cost of additional data transfer and computation overhead, which can increase training time. In the case of truncated backpropagation. checkpointing unavoidable. For our RLR optimizer, though, gradient checkpointing is not necessary technique.
C HYPEPARAMETERS
All the experiments are conducted on a machine with 8 NVIDIA V100 GPUs. Each GPU has 32GB of memory.
For the Text2lmage and the DCoT experiment, we use Adam optimizer with the learning rate of 5 x 10-4, The batch size and the gradient accumulation steps are 2. The DDIM steps are 50 and the classifier guidance weight is 7.5. The local sub-chain has a length of 2. We use Gaussian noise with standard deviation of 10-3 for perturbing the parameters.
For the Video experiment, we use Adam optimizer with the learning rate of x 10-4, The batch size is 1 and the gradient accumulation steps are 8. The DDIM steps are 25 and the classifier guidance weight is 7.5. The local sub-chain has length of 2. We use Gaussian noise with standard deviation x 10 44 for perturbing the parameters.
17

---

Published as a conference paper at ICLR 2026
D QUALITATIVE RESULTS OF TEXT2IMAGE
Aiuuisiic rover noving out of the staikn us cxplore the sufacce ot's distart moon.
A high tech mee oar speads sacrina desolate desert.
noalt aiccuft aips through a suevillaa sky.
mosivo, faturitic drose silently sprawiing
angoy robots in ligh sech operating room.
A heperkoop mun. Bluminned with noon lggts.
Stable Dillasion 1.4
flne-fu--mmii 4umin
lucary yocht docks murina filed with clagent saiboots
janpe
meching thrugh the war fiedd
maccing drough filld.
A boy playing ohess in mama with sod light.
blok pulls
Stable Dilfesiom 1-4
After fime fine-tniigg
Semeoase ndlig murionyclle the
girl riding hurs: fiedd belind
man working project his gangs, wearing eafety popple
A gil walling with a 1oy boar bas land.
Suull of people downtown
cooking
bitchen
Stable Diflisiom Ld
fine-tuning
Figure 7: Qualitative results for Text2lmage generation.
18

---

Published as a conference paper at ICLR 2026
E
QUALITATIVE RESULTS OF TIXT2VIDEO
widdn dalo there
ViluCuate
A Nor.kl hoccka
orirtd tirad
VidooDater
Avem bleae chom md
ALg Muligan Du Daec a
Wndd besinggnnn be becledsirngggiiooo
ViluCaofer
Figure 8: Qualitative results for Tideo generation.
19

---

Published as a conference paper at ICLR 2026
F DETAILS FOR DIFFUSIVE CHAIN-OF-THOUGHT EXPERIMENT
Original prompts for the hand task: (1) A realistic open palm facing upward. (2) A picture of a hand facing downward. (3) hand in relaxed position. (4) hand. (5) A photo of a hand. The generated
DCoT prompts are in Figure TO We show the instructions to generate the multi-scale DCoT prompts Figure
Stable Difitssion 1.4
After fine tuning RLR with DCoT
Figure 9: Qualitative results for the hand task in DCoT.
A realistic open palm facing upward
Coarse-level
A single human hand positioned with the palm facing upward
An open palm with fingers naturally spread and relaxed, oriented upward
Mid-Ievel
Fine-level
Detailed skin texture, visible creases on the palm, and natural lighting on fingernails and knuckles
A picture of a hand facing downward
Coarse-level
human hand with the palm facing downward
Fingers extended naturally, back of the hand visible, wrist slightly relaxed
Mid-level
Visible veins, knuckle contours, skin texture, and soft shadows on the fingers
Fine-level
A hand in relaxed position
Coarse-level
A human hand resting in neutral, relaxed position
Fingers slightly curved, palm partially visible, wrist aligned naturally
Mid-level
Subtle skin folds the joints, soft lighting on the knuckles, natural skin tone and texture
Fine-level
Figure 10: Qualitative examples for DCoT prompts
20

---

Published as a conference paper at ICLR 2026
Instruction to generate the DCoT prompts
Given an original prompt for visual generation, need to break it down into three prompts that controls different scales of the visual content: coarse-scale, mid-scale, fine-scale. The generated prompts should be concise and clear.
Here is the original prompt: <PROMPT>
Your answer format should be:
Coarse-level:
<COMPLETE>
Mid-level:
<COMPLETE>
Fine-level:
<COMPLETE>
Figure 11: Instruction generate DCoT prompts.
G MEMORY PROFILE, TIME COMPLEXITY, AND DIFFERENT DIFFUSION SOLVERS
We give the memory cost for the experiments with Tex12Image on SD 1.4 in Table We offload the memory to the CPU RAM avoid the out-of-memory error. The AlignProp based on truncated BP has the largest memory consumption. The DDPO, based on RL, has the smallest consumption, while sample efficiency terrible, as shown in Figure Our RLR has significantly lower memory consumption than the AlignProp.
Table 6: Memory cost of Text2lmage experiments on SD 1.4
Methods
VRAM
System RAM
Total
DDPO
12.4 GB
0 GB
12.4 GB
AlignProp
25.4 GB
78.5 GB
103,9 GB
RLR
22.4 GB
GB
22.4 GB
We report the per-step time cost for the truncated BP method, AlignProp, and compare the three methods T7ble(7] As shown in the following table, AlignProp has the highest per-step cost, while RL the fastest. When considering the total time reach the same performance level (AES 5.4), the advantage of RLR becomes more evident. RL requires significantly more steps to converge due high-variance gradients. whereas RLR achieves the target fewer steps thanks to its low-variance property, resulting in faster convergence than both RL and AlignProp.
Table 7: Time complexity of Text2lmage experiments SD 1.4
Methods
RLR
RL
Alignprop
Time(min/step)
1.61
0.82
2.85 285 (100 steps)
Time(to same score) 121 (75 steps)
492 (600 steps)
We use RLR to train SD1.4 with the DeMtsoveeMttssseppchhhueeerrrrrr to verify its effectiveness on different samplers. We set the inference steps to 20 and the solver order to 2. The results in the Table S show that RLR maintains robust performance across both samplers. Diffusion-basse
generation methods such DDIM are deterministic when 7 O, and only become stochastic when 17 > 0. In our experiments, when we require the HO estimator at a given step. we explicitly set 7) = 0.1 for the step perform HO. For all other steps, retain the deterministic ODE solver structure by setting
21

---

Published as a conference paper at ICLR 2026
7 O. This approach allows to flexibly combine deterministic and stochastic transitions within the same framework, ensuring the correctness of the HO estimator where needed. DPM-Solver supports both deterministic and stochastic sampling modes. Stochasticity can be selectively enabled at the specific steps where the HO estimator is used, with all other steps remaining deterministic.
Table 8: Ablation of different diffusion solver in Text2lmage experiments on SD 1.4
Methods
PickScore
HPSv2
AES
ImageReward
DDIM DPMSolver
20.14 20.21
28.57
6.53
75.65
28.55
6.56
75.70
H ABLATIONS ON THE SUB-CHAIN LENGTH h
Table 9: Comparison of methods on HPSv2 and ImageReward with memory(GB) and time
cost(minute per step).
Method
HPSv2
ImageReward
Memory
Wall clock time
ZO
22.31
40.82
11.8
0.79
DDPO (RL)
22.79
52.06
12.4
0.82
RLR(1 0)
26.70
63.85
12.7
0,90
RLR(1 1)
28.02
69.85
18.8
1.15
RLR(h =2)
29.22
76.55
22.4
1.61
RLR(A =3)
29.36
76.62
32.6
5.65
RLR(h 4)
29.55
76.70
45.8
9.23
We provide ablation over different sub-chain lengths h and different estimators in the following two tables. We report the reward score from two models, memory, and wall clock time. As the increases, the performance increases due the reduced variance. Increasing h would also increase the computation overhead, including memory and time. We find that when h is larger than 2, the performance gain is marginal compared the increasing computation burden (high memory requirement and long wall clock time). When the h is smaller than 2, high variance would lead performance degradation. Therefore, choose h the best hyperparameter in the main experiment.
I MORE RESULTS ON VBENCH
VBench Evaluation Dimensions. VBench evaluates video generation models across 16 disentan- gled dimensions, categorized into Quality and Semantic groups
The Quality category consists of metrics: Subject Consistency and Background Consistency measure identity and scene consistency across frames using feature similarity: Motion Smoothness evaluates physically plausible motion via motion prior model; Dynamic Degree quantifies motion magnitude to penalize overly static videos; Temporal Flickering measures high-frequene, temporal instability; Aesthetic Quality reflects perceptual appeal using an aesthetic predictor; and Imaging Quality assesses distortions like blur or noise using an image quality model.
The Semantic category includes 9 metries: Object Class and Multiple Objects test object presence and compositionality: Human Action verifies accurate motion execution; Color. Spatial Relation- ship, and Scene check fidelity to prompt-specified attributes and layouts; Appearance Style and Temporal Style assess stylistic alignment in space and time; and Overall Consistency captures general text-video correspondence. Each dimension has tailored prompts and automatic evaluation pipelines, ensuring fine-grained, human-alignen assessment.
According the VBench protocol, the Total Score(TS) is computed Quality Score(QS) and Semantic Score(SS), following the formula: TS 0.8 QS + 0.2 SS. In evaluation, the proposed RLR method achieves the highest scores across all three levels: QS, SS,
weighted sum of the
22

---

Published as a conference paper at ICLR 2026
and the final TS. Notably, most metrics contributing heavily QS reported in the main text due to their strong correlation with perceptual quality and their larger weights in the TS calculation. For completeness, present all 16 VBench metrics in Table I0 where RLR consistently outperforms existing baselines.
Table 10: Automatic evaluation on VBench.
(a) Quality dimensions and total score.
Subject Consistency
Background Consistency
Motion Smoothness
Dynamic Degree
Aesthetic Quality
Imaging Quality
Temporal Flickering
Quality Score
Total
Method
Score
ModeIScopeI2V
89.87
95.79
66.39
52,06
75.75
Open-Soun
92.09
97.39
95.61
48.61
57.76
61.51
98.41
78.82
75.91
Pika
98.
37.22
63.15
62.33
99.77
Gen-2
97.61
97.61
99.58
18.89
66.96
67.42
99.56
82.47
80.58
T2V-Turbo
97.02
63.04
97.48
DDPO
95.53
96.63
96.92
$8.29
59.23
66.84
97.63
81,43
79.84
VADER
96.71
97.62
84
RLR
97.64
97.19
98.05
70.69
66.15
71.08
97.70
85,20
83.14
(b) Semantic dimensions.
Object Class
Multiple Objects
Human Action
Spatial Relationship
Appearance Style
Temporal Style
Overall Consistency
Semantic Score
Method
Color
Scene
MrdelSaope
82.25
38.98
92.40
81.72
33,68
39.26
23.39
25.37
25.67
66.54
Open-Sora
74.98
33.64
85.00
78.15
43.95
37.33
21.58
25.46
26.18
64.28
Pika
87.45
Gen-2
90.92
53.47
89.20
89.49
66.91
48.91
19,34
24.12
26.17
73.03
T2V-Turbo
93.96
38.67
25.51
28.16
74.76
DDPO
91.36
45.67
94.81
90.22
37.54
55.37
25.27
25.04
28.03
73.47
VADER
91.89
48.67
95.12
91.56
39.78
$4.39
25.14
25.23
28.19
74.21
92.52
50.22
95.20
92.31
40.33
54.18
25.74
25.35
28.27
74.87
J PROOFS
J.1 PROOF OF PROPOSITION 6.1
Proof. We assume: R Rd + R and Φ Rd Rm x θ - Rd are continuously differentiable, and their gradients with respect the targeted arguments are uniformly bounded. That is, there exist finite
constants MR > 0 and M, > 0 such that sup,, V I MR. sup | (x..9) .).9) (Ir,, Rt, 0) <
Mg. Under these assumptions, for the composite function ((CITT(ET 21.7, 0)), the partial derivative with respect to is also uniformly bounded by the chain rule (as product of bounded terms). Therefore, the integrability condition required by the Dominated Convergence Theorem (Rudin 1987) satisfied, which justifies interchanging the gradient and the expectation (Glasserman 1990).
By chain rule and vp 01.T (Dr,( 21:7; 0)) (2 21; 21; 0), have
@R(xo)
axo
dR(xv))
[bhh 10 =1; =1;0)
axo Ox
dR(ro) dx
+
00
00
dra
08
Dx 00
Obi (21, 11;0)
Oxn Or
002(12,
z2; 0)
Or
dR(zo) Or 00 dr
Oxz
00
00
Tbp (I1, 21; 21;0)
( 250
0x,-1
dR(Fo)
00
00
0x,
dxa
j=1
Therefore, we can reach the conclusion that
[o Ezyr
(o
Ab(11,
T
i-1 0x 1 0eh (Ni, 2;
aR((ao)
E
00
00
0x;
dæ
j=1
23

---

Published as a conference paper at ICLR 2026
which means the unbiasedness of the FO estimator. Furthermore, the truncated BP estimator is
doi (01 (11,
(2, 2;
2-1
dR(roo)
V ) truncated
(1I)
00
00
Ozi
dro
The structural bias the truncated BP estimator can be specified by combining the above results:
ao(a;; 0)
dØ(1;; 0)
dR(wo) drg
[oR[R(20)) E R(30 R
Σ
08
0z,
1=
which completes the proof.
J.2 PROOF OF PROPOSITION 6.2 AND ADDITIONAL ASSUMPTIONS
Assumption J.1. Define R(;; 0) as R(01.T(TT) 21:7: 0)). Assume that R(2; 0) is differentiable with respect to 0 almost surely, PP ((R(0)) = r) = 0 for every E R, and Lipschitz condition holds for every 0 and 02:
[R(2; 01) - R(2;00) v mm (z) - 02/,
where mi (=) is integrable. Assumption J.2. For any 2t, whose randomness comes from 2, the density ((x;; () is differentiable with respect to 0, and uniform integrability holds:
② f (mx; 0) I m2(2),
sup R(2; 0) 0g
where m2(=) is integrable.
Assumption J.3. R(2;0) 0) is twice continuously differentiable. The following functions are integrable: (-)2, (2))) supg [R(::0) m-(-), sup, [R(: 0) x sups (:" ())
Since R(2;0) is the reward function and the random variables 21:7 are Gaussian distributions in our case, is easy to check that the above assumptions are satisfied. By applying Theorem Cui et al. (2020), we can reach the conclusion.
Proof of Proposition 6.2
J.3 PROOF OF THEOREM 6.3
Proof. The RLR estimator contains three parts: FO estimator terms, HO estimator terms, and ZO estimator terms. For simplicity, we can consider the terms with an FO estimator term, an HO estimator term, and ZO estimator term. Substituting the specific form of the iteration process, have
To 60x1, 21;
#1 ( #2), T2 ( 0) 23+
where #1 = (p(2;; + 22) is an ZO estimator term and x2 = p( 0) + 23 an HO estimator term. We define 2g f2o(2) and 23 fuo(2) indicate the noise distribution associated with ZO and HO estimators. The overall gradient can be written as:
ToRz,a(R(a))
Σ Vo, E.,a ((((((((((x)))))) 23: O2 + 22); 21; (1;;1))]]
(12)
10, 00=00=m 00=
For the FO term, (xx1, f), the gradient is derived by applying the chain rule:
Vo, E E [R(((((((((Ts) 0) 23; 0 22), 21: 01
-
dR(zz)
-V, E. (
En.
E
DJ
o(xx)
21;
dæg
0,=0
24

---

Published as a conference paper at ICLR 2026
To derive the gradient of the ZO and HO terms, the key idea is push the parameter of interest into the density function of the corresponding noise via a change of variables, so that the gradient operator only acts the log-density function. We have the gradient of the ZO term:
002 22 ((((p(((((3)) 0) + z3; 02 + 22), 21; 0))
02=0
Vo, Ea R( 82 22), 21: 0)]
0==0
Vo, E0, (R 22). 21; (1; [z)
23]]
02=0
Vo, 2s, 1 21; 23
0==0
L (og f2o 02)121 23|1
10=0
E2 [ E2g~i1o
In f20 23]l
0==8
= R(xo) (=2)]
where the first equality follows from the tower property of expectations, the second from change of variables, i.e.,2% A + 2g, the third uses the likelihood-ratio trick, and the fourth substitutes the variable back and collapses the expectation. Likewise, with the change of variables 2t = <p(333 0g) + 23, the derivation for the HO term follows analogously, yielding:
V E. 22. [((0(p(p((x300) + z3: 0 +2), z1i 0))
0g
Va, Peabe//sssSsss//mmmoooooo/oo/eee 0 21) =2]]
8==0
Tashnnaasannnnanooddooaaaaahaada c 0z)) 21 22
m2
P==0
Hea (Ho (00::00333000000000000000
0gg
Bes-/nts) (n _2]
10==0
Es
D3 ((13) p(13; Vn (n
0=0
Finally, we have the unbiased RLR gradient estimator:
100(2 0):
dR(zo)
0p(xs:0) 0) T V. fuo(2a)
VgE [[(oo] E E2ys
R(zo) V In fzo(22) dxp
09
09
0=0
Since the sum of unbiased estimators still unbiased, it is easy to generalize the result for any number of terms along the lines of the above proof. Also. the unbiasedness of the estimators does change no matter how we pick the combination of the HO estimator and the ZO estimator. Therefore. we can reach the conclusion that:
E2i [R(wn)] DJ 6(11, =1; dl(zz) dro
Σ R(1o) f(z)). IEC
-
which completes the proof.
25

---

Published as a conference paper at ICLR 2026
J.4 VARIANCE OF THE RLR ESTIMATOR
In this section, we discuss the variance of the RLR estimator, which consists of 3 terms, which are denoted as A, B, and C, respectively;
0 (21, 21:
RLR
R(xo)Do
- V(oo))
00
A
(13)
It is easy to verify that the variance of RLR can be bounded by the variances of terms A, B and C:
Var Var(A + B C) VVa(A) + Var(B) + Var(C) + 2 Cov(A, 2 Cov(A, C)) Cov(B,C)
S Var(A) + Var(B) + Var(C) + Var(A)) + Var + Var(
(14)
which gives an upper bound for the variance of the RLR estimator. Next we derive the specific expression of the terms Var(A), Var(B) and Var(C"..
First, the term represents the original exact BP without any injected noise, so its variance Next, for the terms B and C, since both are LR estimators, present them in the following unified form for simplicity:
n = R( Vo In (z, 0)).
where Vg In f(2,0) = D3 () V In (z; in the term B. The variance is given by
Var(n) E R V In f(2 0)) -()))
where u(0) is the estimator mean.
To better characterize the variance, we now derive an alternative form of the gradient:
ʃ(z; + h) [(2; R(z)dz
In [(2 = lim h-+0
h
/
f(z; 0 + h) f(2;0) R(2)dz
lim
(15)
lim 1 h- To [(z:0)
[f(z;0+h) +h) R( R(2)dz
[2;
lim (E[ h) R()( -E
h- h
where the importance weight h) f(a:++h) f(z;0)
With this alternative form Equation 15 have the variance of the LR estimator
Var(m)
lim [F[(w(0, h) R() v())3;
(16)
h-+
By the Hanmersty-Chhppma--oobhe bound, we can derive a lower bound for the variance:
(ui0 + h) -)))))))))))) Elw 1]2
Var(n) sup h
which is generalization of the more widely-knows Cramer-Rat Rao bound and describes the minimal variance achievable by the estimator.
Under limited computational resources where full BP is infeasible, the only unbiased gradient estimation methods available are RLR, zeroth- ooder optimization, and RL. Compared to the latter two, RLR incorporates h-length half-order optimization and single-step precise BP. According to Proposition 6.2 and the subsequent discussion, it is evident that RLR achieves the lowest variance among these methods.
26

---

Published as a conference paper at ICLR 2026
J.5 PROOF OF THEOREM 6.4
Proof. Since R(.) is L-smooth, we have
Bx41 + L E [110+] 0x [Fa]
E[R(Ok+1) R(O,)
522 E [[ + [[
R(O+)) - 7 (V VR(O0)++ + + 2
(17)
S R(0s)) - (1 5 2
VI(O0)11 LORO
2
S R(Ok)) 2 | 1(x))1 + 2 T
where the last inequality holds if 7 ~ 1/1. By taking expectations over the filtration FR. we have which is equivalent to
E[R(OA+1)) - E[R(O.)) 3B(VR(0))) LDRORLR
2
2 (B(R(O))) - +(((O+++)))++ +
[V(()) <
Taking the average over i: 0, 1, ...... K, have
1 E EVR
2(R(zo) - R²) 7(K 1)
+ LLR
K
Defining Ao = R(vo)) - It", if we set the step size
24 (K +1++ IK M.R
L
then we have
8LAOo RLR
2LAo
K B(II)(())))))) -
K K+1
K+1
which completes the proof.
J.6 SOLUTION OF OPTIMIZATION PROBLEM
Quadratic Upper Bound on Variance. Using the variance decomposition bound 14 the objective of the problem 7 can upper bounded by quadratic function of l:
Q(h) a(h + 1)² + b(h + 1) + c,
(18)
where the coefficients a, b, e are given by:
V/2 + V22 2V V > 0, b -TVV2 + V2.
Then, the unconstrained optimal solution h1 is given by:
T(V2 V/.v2) 1. h 2(V2 + V22 - 22v
(19)
Where h and (7-1-1 h) are the number of steps for HO and 70; V2 and V2 are coefficients indicating the magnitude of the variance of HO and 70 per step, and Vn & V, since HO has lower variance. Notice that T' > 2. Therefore, hine > 0. B₁ and B, are coefficients indicating the magnitude of the memory cost of HO and ZO per step and B3 > B.. B,T < B < B,T. The
constraint of the problem 7 implies that h < B, -B.
BT- and BTB,(T-1) BN-B.
Then, the solution to the optimization problem is given by:
B - (T -
T(V? V.V-))
ming
2(2 + V2 2V, V.) 1}}.
Br. B2
27


Reviews:
[{"review_id": "M7caMEE5FJ", "reviewer": "Reviewer_joeE", "summary": "The paper proposes a Half-order Fine-tuning method for efficiently adapting large-scale diffusion models (e.g., Stable Diffusion) to downstream datasets.  The authors also present a theoretical proof that RLR is unbiased, has lower variance, and enjoys convergence guarantees. The experiments demonstrate RLR’s efficiency.", "questions": "NA.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper presents a novel fine-tuning scheme and devises gradient estimators for the diffusion model’s chain-of-thought, which appears genuinely innovative.\n- The theoretical analysis is careful and offers a credible justification for the proposed approach.", "weaknesses": "- Missing a comparison with related diffusion model fine-tuning baselines, such as D3PO[1].\n- The experiments are limited to SD 1.4 and SD 2.0, which are now dated. Moreover, the method’s generalization to the Flux architecture remains unclear.\n\n$\\text{[1] Yang, Kai, et al. \"Using human feedback to fine-tune diffusion models without any reward model.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.}$", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZsZmRR1b1A", "reviewer": "Reviewer_KciP", "summary": "This paper addresses the crucial task of efficiently aligning diffusion models (DMs) to meet downstream application requirements after pre-training.   Contemporary fine-tuning methods, such as Reinforcement Learning (RL) and truncated Backpropagation (Truncated BP), suffer from high variance and biased gradient estimation, respectively.  The authors propose the Recursive Likelihood Ratio (RLR) optimizer, a novel \"Half-Order\" (HO) fine-tuning paradigm. RLR constructs a new gradient estimator by cleverly combining First-Order (FO), Half-Order (HO), and Zeroth-Order (ZO) estimators within the recursive chain. The paper theoretically proves that the RLR estimator is unbiased (overcoming the defect of truncated BP) and has a lower variance than RL/ZO methods. Extensive experiments on text-to-image and text-to-video tasks validate the superiority of RLR. It not only outperforms baseline methods (like DDPO and Alignprop) on multiple reward benchmarks, but critically, it also avoids the \"model collapse\" problem caused by truncated BP9. Furthermore, the paper proposes a novel prompt technique called \"Diffusive Chain-of-Thought\" (DCoT), which synergizes naturally with RLR's HO estimator, allowing the model to optimize for specific generation scales (e.g., \"fine-grained\" details).", "questions": "- As in weakness, why different descriptions occur, and how to choose the sampling strategy?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper addresses the crucial task of efficiently aligning foundation diffusion models. This represents a highly important and practically valuable problem. \n- This paper proposes the Recursive Likelihood Ratio (RLR) optimizer, a novel \"Half-Order\" (HO) fine-tuning paradigm that successfully overcomes these challenges. T\n- he paper also introduces a novel prompt technique that synergizes naturally with the RLR optimizer, further enhancing the originality of the contribution. \n- The paper rigorously demonstrates its method's advantages in terms of bias, variance, and convergence from both theoretical and experimental standpoints, while also proving its practical effectiveness.", "weaknesses": "- The paper exhibits significant inconsistencies in its core methodology description, particularly regarding the sampling strategy for the Half-Order (HO) sub-chain starting point, $j$. In Section 4.2 (Methodology), the paper describes $j$ as being sampled from a categorical distribution based on gradient norms. However, in Section 5.3 (DCoT Experiment), $j$ is described as being selected from a uniform distribution ($j \\sim \\mathcal{U}(1, T-h)$). This contradictory description makes it impossible to determine which sampling strategy the standard implementation of RLR is supposed to use.\n- Furthermore, the implementation of DCoT (Diffusive Chain-of-Thought) introduces a critical external dependency. As shown in Section 5.3 and Appendix F, DCoT relies on an external Large Language Model (LLM) to generate the 'coarse-mid-fine' grained prompts. This raises doubts about its robustness in practical deployment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EeMI5marpQ", "reviewer": "Reviewer_4kwL", "summary": "This paper proposes an innovative \"half-order\" fine-tuning paradigm, making a substantial contribution to diffusion model optimization. By intelligently combining three gradient estimation strategies, it achieves a high level of theoretical and practical application. Rigorous mathematical proofs and extensive experimental validation demonstrate the RLR method's significant advantages over traditional approaches.", "questions": "I wonder when I use ZO, when the noise is large, do I still need to go through the entire denoising trajectory to get the reward? If this is the case, then ZO also seems to have a high cost problem, which might be solved by process reward model such as SPO.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The innovative concept of \"half-order\" fine-tuning paradigm is proposed, which fills the gap between traditional first-order and zero-order methods.\n2. FO, HO and ZO complement each other's strengths and find the optimal balance between variance and computational cost by optimizing the h and j parameters, taking into account the actual computational budget constraints.", "weaknesses": "1. The problem with FO is its high cost, and the problem with ZO is its high variance, but the author does not provide a clear analysis to explain this. For example, for a specific scenario, how many NFEs are needed for FO, ZO, and HO respectively, how is this calculated, what are the variances of these three, and why is there such a large variance problem. I think this needs a clearer analysis.\n2. The visualization results look average, and the improvement is not significant enough. Lacks comparison with recent works such as FlowGRPO and ReFL.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)