TreeGRPO: Tree-Advantage GRPO for Online RL Post-Training of Diffusion Models
Authors: Zheng Ding
Weirui Ye
Venue: ICLR 2026 Poster (2026)
Keywords: Generative Models
Flow Matching
Reinforcement Learning
GRPO
Tree Search
Primary area: generative models

Abstract: Reinforcement learning (RL) post-training is crucial for aligning generative models with human preferences, but its prohibitive computational cost remains a major barrier to widespread adoption. We introduce **TreeGRPO**, a novel RL framework that dramatically improves training efficiency by recasting the denoising process as a search tree. From shared initial noise samples, TreeGRPO strategically branches to generate multiple candidate trajectories while efficiently reusing their common prefixes. This tree-structured approach delivers three key advantages: (1) *High sample efficiency*, achieving better performance under same training samples (2) *Fine-grained credit assignment* via reward backpropagation that computes step-specific advantages, overcoming the uniform credit assignment limitation of trajectory-based methods, and (3) *Amortized computation* where multi-child branching enables multiple policy updates per forward pass. Extensive experiments on both diffusion and flow-based models demonstrate that TreeGRPO achieves **2.4$\times$** faster training} while establishing a superior Pareto frontier in the efficiency-reward trade-off space. Our method consistently outperforms GRPO baselines across multiple benchmarks and reward models, providing a scalable and effective pathway for RL-based visual generative model alignment. The project website is available at https://treegrpo.github.io.

Published as a conference paper at ICLR 2026
TREEGRPO: TREE-ADVANTAGE GRPO FOR ONLINE RL
POST-TRAINING OF DIFFUSION MODELS
Zheng Ding UC San Diego zhdi agding@ucsd.eer
Weirui Ye MIT
ywr@csail mi edu
ABSTRACT
Reinforcement learning (RL) post-training is crucial for aligning generative mod- els with human preferences, but its prohibitive computational cost remains a major barrier to widespread adoption. We introduce TreeGRPO, a novel RL frame- work that dramatically improves training efficiency by recasting the denoising process as a search tree. From shared initial noise samples, TreeGRPO strategi- cally branches to generate multiple candidate trajectories while efficiently reusing their common prefixes. This tree-structured approach delivers three key advan- tages: (1) High sample efficiency, achieving better performance under same train- ing samples (2) Fine-gra -raie-g credit assignment via reward backpropagation that computes step-specific advantages, overcoming the uniform credit assignment methods, and (3) Amortized computation where multi-child branching enables multiple policy updates per forward pass. Extensive experiments on both diffusion and flow-based models demonstrate that TreeGRPO achieves 2.4x faster training while establishing superior Pareto frontier in the efficiency- trade-off space. Our method consistently outperforms GRPO baselines across multiple benchmarks and reward models, providing scalable and effective pathway for RI.-based visual generative model alignment. The project website is available at https:/treegrroogihubb..
limitation of trajectory-
INTRODUCTION
Recent advances in visual generative models, particularly diffusion models Ho et al. (2020) Rombach al. (2022): Podell al. (2023) and rectified flows Lipman al. (2022): Liu et al. (2022) ) Esser
et al. 2024). have achieved state-of-the-art fidelity in image and video generation. Although large- scale pre-training establishes strong data priors, incorporating human feedback during post-trainis crucial to align model outputs with human preferences and aesthetic criteria Gong et al. (2025)
Inspired by the success of reinforcement learning (RL) in aligning large language models (LLMs), researchers have begun adapting RL to visual generative models. Early methods like DDPO B lack cct al. (2023) and DPOK Fan et al. (2023) demonstrated feasibility but faced challenges in scalability and stability. The introduction of GRPO (Shao et al. (2024) and its adaptations, such DanceGRPO Xue et al. (2025) and FlowGRPO Liu al.25) provides a PPO-style update framework based on group-relative advantages. However, these GRPO-based methods suffer from two critical limitations: (1) poor sample efficiency, since each policy update requires sampling complete, computationally expensive denoising trajectories, and (2) coarse eredit assignment, where a single terminal reward is uniformly attributed to all denoising steps, obscuring the contribution of individual actions. While MixGRPO Li et al. (2025) attempts reduce costs via hybrid sampling and sliding windows, it often sacrifices final performance for efficiency.
In this work, we propose TreeGRPO, a novel RL framework that introduces rre-sstuctured advan- tages to overcome these limitations. Drawing inspiration from the exceptional sample efficiency of tree search sequential decision-making domains like game playing Silver et al. (2016) 2017); Ye et al. 2021), ) we recognize that the fixed-horizon, stepwise nature of denoising makes diffusion/flow
'Equal Contribution

---

Published as a conference paper at ICLR 2026
Comparative Anslysis: Parelo Frontier vs Training Dynamics
Anelysis Averaged Nermaliad Score in Medch
Evalantion durkng Traintng (258 spechs)
RE00 Twicano
140
X0
Tratiaio
260%
Noctaked Eto
100%
0.32
110
340
20%
031
030
G2U Theae (Fkgle A196
Trikiag This (BCA106)
WEECOME
Figure 1: The proposed TreeGRPO achieves the best pareto performance across the rewards and training efficiency, where the Mingl-GPUUI runtime is the normalized wall-clock time. In (a), following the normalized metrics in RL domains (Mnih et al. 2013), the nromalized reward scores here is calculated by (r -- Tsddss) ClipScore reward models are (1.0, 2.0, 10.0, 1.0) respectively.
rsd3.s), where the mmax in the HPS, ImageReward, Asethetic,
generation particularly amenable to tree-based exploration. Our key insight is to recast the denoising process search tree where we efficiently explore multiple trajectories from shared prefixes.
As illustrated in Figure 2 the proposed TreeGRPO framework initiates from shared noise samples and branches strategically intermediate steps, reusing common prefixes while exploring diverse completions. Specifically, at denoising step 1, we expand N candidate paths for n subsequent steps before producing final images. These candidates are evaluated by reward models, and we backprop- agate rewards through the tree to compute dense advantages for each edge-providing more accurate credit assignment than uniform trajectory rewards. This design provides three principal benefits: (1) High Sample Efficiency: Achieving higher performance under the same training samples; (2) Precise Step-wise Credit Assignment: Reward backpropagation through the tree structure com- putes step-specific advantages, addressing coarse credit assignment; (3) Amortized Compute per Forward Pass: Multi-child branching generates multiple advantages per node, enabling multiple policy updates per forward pass.
In terms of experiments, following prior works Xue et al. (005); y Li et al. 2025): : Liu et al. 2025 we employ 1 (Wu et al. 2023), ImageReward (Xu al. 2023), Aesthetics (Wu al.
2023)
and ClipScore (Radford et al. 2021) as reward models. We report both single-reward (HPS-v2.1 only) and multi-reward (HPS- and CLIPScore) settings, and evaluate on all four rewards. Our results demonstrate that TreeGRPO achieves 2-3x faster training convergence while outperforming baselines, establishing superior Pareto trade-off between efficiency and final reward I Our main contributions are:
We introduce TreeGRPO, tree- structuue RL framework for fine fine-tuning visual generative models that enables exploration through branching and prefix reuse.
We develop precise credit assignment mechanism that backpropagates rewards through the tree to compute step-specific advantages.
We demonstrate significant efficiency and performance gains, including 2.4x improve- ment in training efficiency and consistent improvements across multiple reward models.
2

---

Published as a conference paper at ICLR 2026
Xy
GRPO Update for Edges
SDE (trainable path)
at n
L = A, (()))
A
X
ODE (frozen path)
X
Back Propagate for Advantages
Σ wiAlnaf
X
X A1
X
Aedge
X
6d
X 62
X 12
TT(a)) Wi Σ E (a))))
XB
iT)
Tmean
Ateaf
12
K3
f4
F6
Tstd
Leaf Node Advantages
Figure 2: Introduction of TreeGRPO: Our framework optimizes the denoising process of diffu- sion/flow models by constructing search trees. Starting from shared initial noise, explores multiple trajectories by branching at intermediate steps, leveraging prefix reuse for step-wise advantages.
RELATED WORK
2.1 RL POST-TRAINING GENERATIVE MODELS
Modern visual generative models are dominated by diffusion and flow-based approaches. Diffusion models learn to denoise Gaussian-corrupted data, supporting both stochastic (SDE) and deterministic (ODE) sampling Ho et al. (020);; Song et al. (2020). Flow matching methods learn velocity fields for continuous normalizing flows, with recent advances enabling efficient ODE-style sampling Lipman et al. (2022); Karras al. (2022). Theoretical work has unified these approaches through stochastic interpolants and pedimal-contro perspectives Allergo Vanden Vanden-Eijnden (2022); Domingo-Enrich
al. (2024). Alignment with human preferences remains key challenge. Current methods include direct reward optimization [ee et al. (2033)
:: Xu et al. (203),, 3, off-policy techniques like advantage- preference-based learning (DPO, RAFT) that avoid explicit value functions Rafailov et al. 2023); Dong et al. (2023). Policy gradient methods (c.g.. PPO) provide general RL frameworks for exploration-sensiiiee Ceenarios/Schulle
weighted regression Peng al. (2019),
et al. (2017). Gao et al. 2024) reduce policy optimization to regressing the relative reward for generative models. The success RL post-training enhancing language models Jaech et al. (2024); Guo et al. (2025) has inspired similar approaches for visual generation. However, visual domains pose unique challenges for step-wise credit assignment along denoising trajectories.
2.2 TREE-BARE BASED REINFORCEMENT LEARNING
Tree search methods combined with learned policies offer exceptional sample efficiency and precise credit assignment. The AlphaGo series demonstrated superhuman performance through neural- network-guided search and pruning Silver al. (2016) 2017), with later works confirming remarkable efficiency Ye et al.) (2021). In language domains, strustrde reasoning organizes inference as path search Yao al. (2023). while recent RL methods leverage structured exploration to amplify training signals Jaech et al. (2024); Guo et al. (2025). Most related, Yang et al.) (2025) applies tree-based optimization to language models, searching over token sequences. Our work adapts tree- structured search to denoising processes. TreeGRPO leverages shared noise prefixes across branches for efficiency while enabling step-level credit assignment via reward backpropagation.
3 BACKGROUND
We briefly review flow matching and its formulation as Markov decision process (MDP) for RL fine-tuning. We then introduce ODE-SDE conversion that enables stochastie, probability-aware

---

Published as a conference paper at ICLR 2026
sampling while preserving marginal distributions, which is essential for policy-gradien RL. Finally, we contextualize our approach within existing RL methods.
3.1 FLOw MATCHING AND RECTIFIED FLOws
Flow matching models define probability path between data x Pdata and noise x1 through linear interpolation:
Pnoise
x, (1 - (1-1)xx + Ix1, E [0. 1].
A velocity field () is trained to predict the direction x1 Xo using the objective:
( E,,,,.x [Ilvo(x,,1) (x1 xx)ol
(2)
Inference follows the probability-flow ODE dx, va(x, r)dr or its stochastic variant.
3.2 DENOISING AS MARKOV DECISION PROCESS
We formulate generation as a finite-horizon MDP (S,A,,, R) where state s; = (c, includes conditioning information (e.g., text prompts), timestep 1, and current latent Xy. Actions parame- terize transitions X, - X1+1, and a terminal reward R(x,, c) is provided by preference metrics. The RL objective maximizes:
(( = Br-ro c). (so. ao. 5T),
(3)
enabling optimization of black-box rewards inaccessible supervised training.
3.3 ODE TO SDE CONVERSION FOR POLICY GRADIENTS
Deterministic ODE solvers lack the transition probabilities required by policy-gradien RL, Following
Song et al. 2020 Albergo et al. (2023). we convert the probability-flow ODE
dx, fe
(4)
to an equivalent SDE that admits tractable likelihoods while preserving marginals:
1 dx, = fo(x,,)) + (r) Pa(x; c.t) dt + ott)dW,
(5)
Here ((1) controls the noise scale, with (r) = 0 recovering the deterministic ODE. This stochastic formulation enables proper credit assignment while maintaining sample quality.
3.4 COMPARISON RL FINE-TUNING METHODS
DDPO/DPOK samples trajectories independently and normalizes advantages at the batch level. DanceGRPO introduces group advantages but requires full regenerations. Flow-GRPO adapts GRPO to flows with stochastic sampling, similar to DanceGRPO as they published at the same time. MixGRPO improves efficiency ODE-SDE hybrid sampling but lacks fine-grained credit assignment.
Our TreeGRPO approach formulates denoising as a sparse tree rooted shared noise. By branching strategically, it simultaneously achieves all three desiderata: prefix reuse enables superior training efficiency; reward backpropagation through the tree structure provides fine-grained step-wise credit assignment; and multi-child branching facilitates group advantage comparisons. This unified ap- proach overcomes the fundamental limitations of trajectory- based methods that assign uniform credit and require independent sampling.
METHOD
We present TreeGRPO, tree-structured reinforcement learning (RL) post-training framework for diffusion and flow-based generators. TreeGRPO (i) reuses shared denoising prefixes to markedly improve sample efficiency, (ii) assigns step-wise credit by propagating leaf rewards back through the

---

Published as a conference paper at ICLR 2026
tree produce per-edge advantages. and (iii) optimizes GRPO-style objective these per-edge advantages. At a high level, TreeGRPO builds a sparse search tree over a fixed denoising horizon by branching only within scheduled SDE windows while using ODE steps elsewhere. Leaf rewards are group-normalized per prompt and then backed up to internal edges to yield dense advantages that weight the aalicy-gradient update.
4.1 PROBLEM SETUP
We consider a conditional generator given conditioning с (e.g., text), a denoising/flow horizon I . and latent states x,- Sampling is viewed MDP with state 8, (c,), and policy (a, sy) that induces X,+1- A terminal, non-differentiable reward R(xr, is provided by a preference model. The post-training objective
max Br | [
= (so. 40. .T))
(6)
This formulation permits direct optimization of black-box alignment signals while preserving the fixed-length trajectory structure of diffusion/flow sampling.
4.2 OVERVIEW OF TREE-ADVAN GRPO
TreeGRPO addresses the sample inefficiency of standard reinforcement learning for diffusion models by leveraging tree-structured sampling and temporal credit assignment. The key insight is that denoising trajectories share common prefixes, allowing us to efficiently explore multiple branching paths from shared intermediate states.
The framework operates in three phases: First, construct sparse search tree where deterministic ODE steps preserve shared prefixes and stochastic SDE windows create strategic branching points. Second, compute final rewards for all leaf nodes and propagate these rewards backward through the tree using a log-prod boobability-weighhee action. Third, optimize GRPO objective that uses these per-edge advantages update the policy, with clipping for stability.
average to assign step-wise advantages to each denoising
This approach provides candidate diversity with linear computational cost in the number of SDE windows, while the advantage propagation enables fine-nrained credit assignment that distinguishes the contribution of each denoising step the final outcome.
4.3 PREE-STRUCTURED SAMPLER
For a given prompt e and a predefined window W S (0....,--1), we sample an initial noise X (0, I) and run fixed denoising schedule ...... . with two kinds of steps:
1. ODE steps (no branching). If & W, we apply a deterministic update to every frontier node. This advances all paths without creating new branches and reuses a common prefix across descendants.
2. SDE windows (branching). / W, each frontier node spawns & children by adding a small stochastic perturbation to the ODE mean update. For each child edge e, we compute and store its sampling log- prrbaabbbiit log Te. (e) under the frozen sampler.
Repeating this until = T yields tree whose leaves share deterministic prefixes and differ only at the SDE windows. We then decode each leaf to an image and use the stored edge pog-pobbabilitiss for advantage propagation and GRPO updates.
4.4 RANDOM WINDOW
We select a single contiguous SDE window of fixed length w along a T-step denoising schedule with timesteps indexed ,, T-1. For a start index the window is W, ii ..., i+w-1),
E 10, T - w 1}. At the beginning of each training epoch, draw the start i from truncated geometric distribution over (O...., r E (0, 1):
with parameter
(1 (1-r)r'
Pr[
0. I, w
1-y7-w T-w
5

---

Published as a conference paper at ICLR 2026
Algorithm TREEGRPO: SDE-window branching, per-step advantages, GRPO update
Require: Policy лœ; sampler uses Ta steps T; branch b; leaf cap N; SDE window W(I); rewards {Ry] with stats (rK)
1: for all prompt do
2: Sample shared seed ~ N I); init root (Xo, r=0)
3: for 0 to 7 7 do
Build tree
4:
r E (!) then for all frontier node do for j to do p SDE,sTEP(U, It Reaa c, record edge log aaa
> SDE branching
5:
6:
7:
8:
else
ODE continuation
9:
for all frontier node do
10:
V ー ODE.ODEESPPPP.. (u, Aau C, f); record edge log taa
Decode leaves {x7 (y("))) ); (== Σx (y (c)-p set μ. or over (r(1)
11:
12:
for all leaf edge e (p- do
(()) -
13:
Aodge e)
14:
for / = max (W(I)- down to min (I) do
post-order backup
15:
for all internal node at time 1+1 with child edges Su)) do π (e)) - softmax/ (log T (e): (E)()))
16:
Anode (u) S(e (e) (e)
18:
Aedge (p 11) - (n)
LGRPO Σrew() Eece, log No (a,()) (x()) Aedgole); 0-0 17V eLRpP
This distribution places more mass on earlier timesteps when r is small and becomes closer to uniform I. In practice this early-time bias is desirable because post-training primarily targets corrections in the initial denoising stages.
4.5 LEAF ADVANTAGES CALCULATION
For each prompt e with leaf set L(c), we first aggregate raw reward scores from one or more evaluators (Ry) using nonnegative weights (wx) (typically uniform):
5(0) Σ wE Re(
l L(c), WE 2 O, E
(8)
Let Me and oh be the mean and standard deviation of (S(1) within the prompt group
DeE(()) The leaf advantages are computed
((1) 510)-Ho Me E L(c),
Alea(()
Oe
These promitecon leaf advantages serve as boundary conditions for the subsequent tree backup to obtain per-edge advantages.
4.6 LEAF-TO-ROT ADVANTAGE PROPAGATION
We convert leaf-level advantages into per-step (edge) advantages by bottom-up pass over the tree. For an internal node u, let S(u) be the set of outgoing child edges and let e' (p - u) denote the incoming edge of u. Each child edge S(u) stores (i) its advantage Aedge(e) and (ii) its sampling log-probability log Tew (e) from the frozen sampler.
Define logprob-based mixture weights by normalizing the stored probabilities; equivalently, take softmax over the stored log- bog-probabilities:
expl log Tama (e))
π T (e)
(e(e) Eeresiw exp( log π To. (e'))
E S(u).
(10)
Eeres(w) T (e')

---

Published as a conference paper at ICLR 2026
The advantage assigned to the incoming edge of u is the weighted average of its children:
Aedge (e')
Σ
Wu(e) Aedge
(11)
eES(u)
When [S(4) Eq. equation I1 reduces identity and the parent's edge advantage equals that of its unique child. Applying equation [1 in reverse topological order yields distinct per-timestep advantages for all internal edges up to the root.
4.7 GRPO UPDATE WITH PER-EDGE ADVANTAGES
For consistency with our setting, we describe the update as GRPO: it is the standard PPO clipped surrogate applied to group-relative, per-edge advantages. For each SDE-window edge & with stored behavior log reg-pobbablity Tog. (x, c,r), define
Y (e; 0) cxpl log ne (a)
log π Bai (a((e))x((e))...))
The GRPO (clipped) objective over all SDE-window edges is
LORPO Σ Σ min rr(e; 0) Aedgo clip(r)(e;)), 1- 1 Aodge
(12)
tEW EEE,
with clip parameter (no explicit KL term). We optimize equation 12 and periodically refresh the behavior policy by setting Pold t 0. In short, GRPO here is PPO with prompt-relative, per-edge advantages computed by our tree backup.
THEORATICAL ANALYSIS OF TREEGRPO
In this section, we provide theoretical justification for the efficacy of TreeGRPO. We highlight that the tree-structured advantage estimation acts as principled method for variance reduction and robustness regularization through weighted averaging based on action probabilities.
5.1 VARIANCE REDUCTION VIA WEIGHTED AGGREGATION
Standard RL fine-tuning methods such as vanilla GRPO estimate the gradient using Monte Carlo samples of single trajectories. In contrast, TreeGRPO aggregates information from multiple branches (l..... K} originating from a shared state 8,. Crucially, this aggregation is a probability- weighted average rather than simple arithmetic mean.
Let wk be the normalized weight for the k-th branch, derived from the policy's 's bog--robbailiitis::
exp(log x Oola (a, (sy))
(13)
ΣK exp(log ane (a,"1ss)) (s,))
K The advantage estimator for the parent node is computed Arre Ek=1 A "eal"
Proposition 5.1 (Variance Reduction with Weighted Estimator), Let 2 be the variance of the reward realization due to future diffusion noise. The variance of the TreeGRPO weighted estimator is strictly less than equal the variance of single-sample estimator, provided the effective sample size is greater than 1.
Proof. The variance of a single-sample estimator (standard GRPO) is Var( AAngli)) = o For the TreeGRPO estimator Ause = -x= (k), assuming conditional independence of branches given St, the variance is:
K
K Σ
(Auc))
Σ
W Var (kk)
2
(14)
km
Since Σ WA = and Wa (0, 1), it implies that Σ w are 0). The quantity 1/(Σ w?) represents the effective sample size (ESS). Thus, the variance is
< (unless one weight and others
of reduced by factor equal the ESS: Var(Auce) ESS This variance reduction leads more stable gradient estimates and larger trust-region updates. Our experiment also shows that increasing the branch number will result in better performances in general, which proves this empirically.

---

Published as a conference paper at ICLR 2026
5.2 REGULARIZATION ROBUSTNESS
The weighted average structure also provides conceptual link to regularization against "noise overfitting." " In diffusion models, a high reward might be obtained purely by chance due to a specific noise seed (a "sharp peak" in the reward landscape), even if the action probability is low.
Proposition 5.2 (Weighted Averaging as Smoothness Regularization). By calculating advantages as an expectation over the local policy distribution (approximated by the weighted tree), TreeGRPO optimizes a smoothed objective that penalizes solutions with high local curvature (sharp peaks).
Proof. The standard update maximizes J(0) = Á(s,, a,) V (ag) Å comes from single lucky path, the policy may collapse to deterministic action that is brittle to noise. TreeGRPO assigns the parent value based on (s,) Σn wxQ(s,, This approximates the true value function V (*()s, Ea-= [Q(s,.
Consider the Taylor expansion of the expected reward around the mean outcome. Maximizing the weighted average effectively maximizes:
1 V
B (2 a)) = (a) +
(15)
By explicitly using multiple samples weighted by æ, the optimization signal favors regions where the expected return is high (high Q) AND where the surrounding region is robust (the second-order term V is largely negative). This acts as an implicit regularizer, discouraging the policy from converging to sharp, narrow optima where slight deviations in sampling (noise) would lead to collapse in reward. This theoretical property aligns with the improved Pareto frontier observed empirically.
In summary, the log- probability weighted aggregation TreeGRPO mathematically equivalent to performing Kaa-BBaackwlllizztte on the advantage estimator (Prop. 5.1 and implicitly optimizing a smoothness-rregulaiized objective (Prop. 5.2
EXPERIMENT
We evaluate TreeGRPO against previous methods under identical sampling budgets (NFE=10) and report both efficiency (per-iteration wall clock) and alignment metrics across multiple reward models. We use HPDv2 dataset for both training and evaluation across all the methods.
6.1 EXPERIMENTAL SETUP
Foundation Models and Datasets We use SD3.5-medium our base model, following recent works on diffusion model alignment. For training and evaluation, we use the HPDv2 dataset (Wu al.) 2023), which contains 103,700 text prompts focused human preference alignment. The evaluation is performed on held-out set of 3,200 prompts to ensure fair comparison.
Reward Models We employ four different reward models to evaluate comprehensive aligmment with human preferences: HPSv2. (Wu et al. 2023), ImageReward (Xu et al. (023).. Aesthetic Score (Wu et al. 2023). and ClipScore (Radford al. [202I). These models capture different aspects of human judgment- HPSv2.1 1 and RewaeReward focus on overall preference, Aesthetic Score evaluates visual appeal, and ClipScore measures text-image alignment. We conduct experiments under both single-reward (HPSv2.1 only) and multi-reward training settings.
Training Configuration All methods are trained with a fixed NEE (Number of Function Evalu- ations) budget of 10 steps. We use batch size of 32 and train for 250 epochs with the AdamW optimizer (learning rate le-5, weight decay 0.01). Training is conducted on 8xA100 GPUs with mixed precision. The same random seed is used across all experiments for reproducibility.
8

---

Published as a conference paper at ICLR 2026
Table 1: Train HPS-v2.1 reward model and Eval on four reward models. Here the comparison results for overhead and performance.
Human Preference Alignment
Method
Iter. Time (s)
HPS-v2.11
ImageReward
Aesthetic?
ClipScore]
SD3.5-M
0.2725
0.8870
5,9519
0.3996
DDPO
166.1
0.2758
1.0067
5.9458
0.3900
DanceGRPO
173.5
0.3556
1.3668
6.3080
0.3769
MixGRPO
145.4
0.3649
1.2263
6.4295
0,3612
TreeGRPO (Ours)
72.0
0.3735
1.3294
6.5094
0.3703
Table Train on HPSSv2.1 and ClipScore reward model with ratio 4:1 and Eval on four reward models. Here are the comparison results for overhead and performance.
Human Preference Alignment
Method
Iter. Time (s)
HPS-v2.1T
ImageReward
Aesthetic
ClipScoret
SD3.5-M
0.2725
0.8870
5.9519
0.3996
DDPO
178.2
0.2748
1.0061
5.8500
0.3884
DanceGRPO
184.0
0.3485
1.3930
6.3224
0.3862
MixGRPO
152.0
0.3521
1.2056
6.0488
0.3812
TreeGRPO (Ours)
79.2
0.364
1.3426
6.4237
0.3830
Baselines We compare against three strong baselines: (1) DDPO Black al. 2023): Uses
PPO for diffusion denoising with batch advantage estimation; (2) DanceGRPO (Xue et al. 2025
:
Applies GRPO with group-based advantage calculation for same prompts; (3) MixGRPO (Li al., Combines ODE and SDE sampling during inference with GRPO updates. All baselines are re-implemented and trained under identical conditions for fair comparison.
2025):
6.2
MAIN RESULTS
Single-Reward Training Table shows results when training with only HPSv2.1 reward. Tree- GRPO achieves the best HPSv2. score (0.3735) and aesthetic score (6. (6.5094) while being significantly faster
than all baselines. DanceGRPO achieves the highest ImageReward score but 2.4x slower than our method.
Multi-Reward Training For multi-reward setting, we use ddvaantag--wiighee summation instead of direct reward addition. Specifically, we set the ratio of HPSv2. reward and ClipScore reward 0.8:0.2 (wo = 0.8, wi, 0.2). We calculate leaf-advantages A A2 for each reward model, and obtain the weighted advantage A = Σi= W,A; as the final advantage. This advantage is then backpropagated through the tree structure using our proposed method. Table demonstrates that TreeGRPO maintains strong performance across all metrics while being 2.4x faster than DanceGRPO, showing particular strength ImageReward (1.3426) and aesthetic scores (6.4237).
Efficcency Analysis
The significant speed advantage of TreeGRPO (72.0-79.2s vs 145.4-184.0
for baselines) comes from our tree-based parallel sampling strategy, which maximizes trajectory diversity within the same NFE budget while minimizing computational overhead through efficient advantage backpropagation.
6.3 ABLATION STUDIES
Tree Structure Analysis Table investigates how different tree configurations affect performance. As for Optimal Branching, & 3, d provides the best trade-off between performance and efficiency. Larger branching (k = 4) improves HPSv2.1 score to 0.3822 but increases computation time by 75%. In terms of Depth Impact, Deeper trees (d provide more training steps but

---

Published as a conference paper at ICLR 2026
Table Ablation on sample tree structure. We set NEF 10 as the default, and train the models on different search tree structure.
Method
Tree#
EffGrp
EffSteps
Time (8)
HPSv21
ImgRwd]
Aesth.
CLIPT
k=3,d=3
27
13
70.0
0.3735
1.3294
6.5094
0.3703
k=3,d=3
54
26
120.2
0.3771
1.3229
6.4598
0,3659
k=2,d=3
8
7
39.4
0.3271
1.0650
6.2736
0.3738
k=2,d=3
2
16
14
60.0
0.3381
1.1002
6.3472
0.3584
k=2,d=4
16
15
59.6
0.3537
1.1725
6.2955
0.3575
(=4,d=3
64
21
126.3
0.3822
1.3857
6.4201
0,3664
Notes. is the branching factor, d is the depth, and "Tree Num" is the number of trees for each prompt. Iteration time is per-step wall clock (lower is better). EffGrp is effective group size which is the number of generated images for the same prompt. EffSteps is the effective training steps per prompt.
Table 4: Ablation of inference strategies during sampling.
Sampling Strategy
HPSv2
ImageReward
Aesthetict
ClipScoref
Random, r 0.5
0.3735
1.3294
6.5094
0.3703
Random, r 0.3
0,3632
1.2815
6.6067
0.3556
Random, r = 0.7
0.3576
1.2384
6.2161
0.3611
Shifting
0.3652
1.3207
6.2736
0.3738
Notes. the ratio parameter in randome window. The smaller will choose the frontier noise step to expand search tree larger probability.
show diminishing returns. k 2, = 4 offers good efficiency but lower overall performance. For Multiple Trees, Using 2 trees with & = 3, improves HPSv2.1 score marginally (0.3771 vs 0.3735) but doubles computation time, suggesting limited practical benefit.
Sampling Strategy Analysis Table 4 compares different inference strategies during tree sampling: (1) Random Window Sampling: The default = 0.5 provides balanced performance. Smaller 0.3 prioritizes aesthetic quality (6.6067) at the cost of text alignment, while 0.7 shows the opposite trade-off. (2) Shifting Strategy: Achieves best ClipScore (0.3738) but compromises on other metries, making suitable for text-heavy applications.
Adaptive Sampling: Our experiments show that dynamic adjustment of during training based on reward progress can provide additional 2-3% improvement, though we use fixed 0.5 for simplicity in main results.
Advantage Weighting Analysis We ablate the multi-reward weighting strategy by comparing equal weighting (0.5:0.5) against our chosen ratio (0.8:0.2). The 0.8:0.2 ratio provides better balance across all evaluation metrics, while equal weighting tends to over-optimize for ClipScore at the expense of other rewards.
7 DISCUSSION
Our work introduces TreeGRPO, novel RL framework that overcomes the prohibitive computa- tional cost of aligning visual generative models by recasting the denoising process as a tree search, achieving exponential sample efficiency through strategic branching and prefix reuse, and enabling fine-grained credit assignment via reward backpropagation. While this approach establishes su- perior Pareto frontier in efficiency and performance, its current limitations include the introduction of new hyperparameters governing the tree structure and an increased memory footprint during training. Future work will focus on developing adaptive scheduling for these parameters, integrating computationally
learned value functions for early tree pruning, and extending the framework intensive domains like video and 3D generation to further enhance its scalability and impact. Apart from GRPO, such tree-based advantages also be applied to other methods for post-training.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Michael S Albergo and Eric Vanden-Eijnden Building normalizing flows with stochastic interpolants. arXiv preprint arXiv:2209. 00.22220995557.... /5571, 2022.
Michael Albergo, Nicholas M Boffi, and Eric Vanden-Eijndenn Stochastic interpolants: unifying framework for flows and diffusions. arXiv preprint arXiv:2303.088797 2023.
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXivv2305. 2023.
Carles Domingo-Enrich, Michal Drozdzal, Brian Karrer, and Ricky TQ Chen. Adjoint matching: Fine-tuning flow and diffusion generative models with memoryless stochastic optimal control. arXiv preprint XrX:::2499. 08890.60,,, 2024.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06761 2023.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high righ-resolution image synthesis. In Forty-first international conference machine learning, 2024.
Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models. Advances in Neural Information Processing Systems, 36:79858-79885 -79885, 2023.
Zhaolin Gao, Jonathan Chang, Wenhao Zhan, Owen Oertell, Gokul Swamy, Kianté Brantley, Thorsten Joachims, Drew Bagnell, Jason D Lee, and Wen Sun. Rebel: Reinforcement learn- ing via regressing relative rewards. Advances in Neural Information Processing Systems, 37:
52354 52400, 2024.
Lixue Gong, Xiaoxia Hou, Fanshi Li, Liang Li, Xiaochen I Lian, Fei Liu, Liyang Liu, Wei Liu, Wei Lu, Yichun Shi, et al. Seedream 2.0: A native chinese-english bilingual image generation foundation model. arXiv preprint arXiv:2503.07703 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Pciyi Wang. Xiao Bi, et al. Deepseck-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12945 2025.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed Kl-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412. 2024.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in neural information processing systems, 35:26565-26577 265777, 2022.
Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint 2rXiv:2302.12192 2023.
Junzhe Li, Yutao Cui, Tao Huang. Yinping Ma, Chun Fan, Miles Yang, and Zhao Zhong, Mixgrpo: Unlocking flow-based grpo efficiency with mixed ode-sde. arXiv preprint arXiv:2507.21802. 2025.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint 1rXiv:2210.02744 2022.

---

Published as a conference paper at ICLR 2026
Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching models via online rl. arXiv preprint arXiv:2505.05470, 2025.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003 2022.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv: 2013.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weightel regression: Simple and scalable off off-policy reinforcement learning. arXiv preprint 1rX0v//110.0017,, 2019.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint wrxiv:2307.01952 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pp. PmLR, 2021.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 11:53728-53744 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 08684-10695, 2022.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: (107:0667.344 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2442.003300 2024.
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Pannecrshelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484-455) 2016.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with general reinforcement learning algorithm. arXiv preprint arXiv: 1712001 2017.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011. 133466. 2020.
Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Better aligning text-to-image models with human preference. arXiv preprint arXiv:2303. /4420, 1(3), 2023.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36: 5903-119355 2023.
Zeyue Xue, Jie Wu, Yu Gao, Fangyuan Kong, Lingting Zhu, Mengzhao Chen, Zhiheng Liu, Wei Liu, Qiushan Guo. Weilin Huang, et al. Dancegrpo: Unleashing grpo on visual generation. arXiv preprint arxiv:2505.088/1 2025.
Zhicheng Yang, Zhijiang Guo, Yinya Huang, Xiaodan Liang, Yiwei Wang, and Jing Tang. Treerpo: Tree relative policy optimization. arXiv preprint 088 2025.
12

---

Published as a conference paper at ICLR 2026
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree thoughts: Deliberate problem solving with large language models. Ad- 11822, 2023.
vances in neural information processing systems,
Weirui Ye, Shaohuai Liu, Thanard Kurutach, Pieter Abbeel, and Yang Gao. Mastering atari games with limited data, Advances in neural information processing systems, 34:25476-25488, 2021.
13


Reviews:
[{"review_id": "Dfgt01yVcS", "reviewer": "Reviewer_5wvK", "summary": "The authors propose a tree-structured technique for advantage estimation and apply it to training diffusion models via GRPO.", "questions": "1. Can you add in a comparison to https://arxiv.org/abs/2404.16767 if you have the compute?\n\n2. Beyond just being empirically promising, I think this paper would be much stronger if there were a more compelling conceptual / theoretical reason why tree-structured advantage estimation is the \"right\" answer. I spent some time thinking about this but I couldn't come up with a reason why. Could you provide some thoughts on this, perhaps working things out on simple toy problems (e.g., flows between two Gaussians)?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "(+) I haven't seen the idea of tree-structured advantage estimation before.\n\n(+) I appreciate the ablations performed.", "weaknesses": "(-) From what I can tell, the results are not significantly stronger than similar baselines performance-wise. That said, requiring less compute is definitely a plus.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YC6D5Fzhyk", "reviewer": "Reviewer_tWQc", "summary": "The paper proposes TreeGRPO, a tree-structured RL framework that reformulates diffusion-model denoising as a search tree, where shared prefixes and selective branching improve sample efficiency and step-wise credit assignment. The algorithm constructs trees by alternating deterministic ODE steps and stochastic SDE windows, then backpropagates rewards through branches to compute per-edge advantages for PPO style updates. Experiments on SD-3.5-Medium show TreeGRPO achieves comparable or higher alignment rewards than GRPO based baselines while cutting training time.", "questions": "1. The tree structure seems designed for training; does it also accelerate inference or sample diversity at deployment? If not, could the branching be leveraged for generating diverse images at inference time?\n\n2. The method combines tree search with GRPO; how much of the observed gain comes from tree exploration versus the per-edge GRPO update? Would a tree based extension of standard PPO yield similar benefits?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.\tRecasting diffusion denoising as a search tree with shared prefixes is a creative idea that directly addresses sample efficiency and credit assignment issues. The use of log probability weighted backup for per-edge advantages is theoretically sound.\n2.\tIn terms of efficiency gains, TreeGRPO reduces per‑iteration training time by ~$2\\times$ - $3\\times$ while matching or surpassing baseline alignment scores. The method shows especially strong improvements in aesthetic scores.\n3.\tThe paper provides a thorough empirical evaluation by comparing TreeGRPO against strong baselines across multiple reward models and both single reward and multi reward settings. Ablation studies systematically vary tree width, depth and sampling strategies, illustrating tradeoffs.", "weaknesses": "1.\tWhile the method amortizes computation, branching multiple trajectories simultaneously increases memory usage, especially for large diffusion models. The paper does not quantify the computational overhead relative to baselines or provide strategies for memory management beyond acknowledging the issue.\n2.\tThe performance improvement is somewhat marginal. Although TreeGRPO outperforms baselines on HPS and aesthetics, DanceGRPO achieves the highest ImageReward score in the single reward setting. The difference in ClipScore between TreeGRPO and baselines is small. Additional analysis could clarify why some metrics benefit less. Also, the Aesthetic metric is much easier to improve compared to others.\n3.\tBranching creates many edges from shared prefixes. That amortizes compute but correlates samples. So the effective on‑policy batch may be narrower than it looks. Importance weighting corrects policy drift but not correlation. Tracking effective sample size or using prefix‑level decorrelation, for example, multiple independent seeds per batch, would clarify the true statistical efficiency.\n4.\tThe framework introduces extra hyperparameters, branching factor, tree depth, SDE window ratio, and number of trees, which significantly affect performance. Adaptive schedules are mentioned as future work but not explored.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hl6WXFJrav", "reviewer": "Reviewer_orTx", "summary": "This paper introduces an approach to fine-tuning diffusion models through a search tree.", "questions": "See Weakness section.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper tackles the problem of making RL-based fine-tuning of vision-based generative models more efficient, which is a well-motivated and common problem in the existing literature. Additionally, TreeGRPO presents a significant improvement in runtime against the baselines considered, while matching or improving performance.", "weaknesses": "My main concern is the following, with some additional questions/concerns listed below. In Section 3.3., the paper claims that “Deterministic ODE solvers lack the transition probabilities required by policy-gradient RL...” and “we convert the probability-flow ODE... to an equivalent SDE that admits tractable likelihoods...”. Likelihoods necessary for RL are not tractable in SDE’s (e.g. due to the Brownian motion)? How is the paper computing log likelihoods for the advantage-based update in Equation 13? \n\nAdditionally, the paper claims an improvement in sample efficiency, but the results are in terms of runtime. In Figure 1, how does the training on 8 GPUs correspond to the runtime on a single GPU? Are these different training setups? There is limited explanation of the implementation details provided. No appendix was provided in the PDF.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vQoxV4VgyC", "reviewer": "Reviewer_8di4", "summary": "This paper presents TreeGRPO, an online RL algorithm for finetuning diffusion and flow matching models. Unlike existing RL methods that treat the multi step denoising process as independent full trajectory sampling, resulting in low sample efficiency and coarse credit assignment, TreeGRPO reformulates the process as a multi step decision problem optimized through tree structured RL. By modeling multi-step denoising as a search tree, it reuses shared prefixes to enhance sample efficiency and propagates node-level advantages for fine grained credit assignment. Experiments across multiple reward models show that TreeGRPO achieves markedly higher training efficiency and establishes a new Pareto frontier for RL based visual generative model alignment.", "questions": "1. Could the authors elaborate on why TreeGRPO performs less favorably on certain reward metrics such as ImageReward and CLIPScore? Does this limitation stem from the local advantage propagation design or other factors like the tree structure or sampling schedule?\n\n2. There is a concurrent work named BranchGRPO [1] that appears closely related to TreeGRPO in terms of motivation and overall design. Could the authors briefly compare TreeGRPO with BranchGRPO?\n\n[1] Li, Yuming, et al. \"Branchgrpo: Stable and efficient grpo with structured branching in diffusion models.\" arXiv preprint arXiv:2509.06040 (2025).", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper is the first to apply tree search–based GRPO optimization to image diffusion and flow matching models, achieving substantial improvements in training efficiency.\n\n2. The paper provides a comprehensive review of existing work and clearly articulates the similarities and differences between TreeGRPO and prior approaches.\n\n3. The proposed method is concise and well designed, with comprehensive ablation studies conducted to thoroughly evaluate the effects of the introduced hyperparameters.", "weaknesses": "1. While TreeGRPO demonstrates impressive efficiency and gains on HPS-v2.1 and Aesthetic rewards, its performance on ImageReward and CLIPScore is weaker than DanceGRPO. This variability suggests that the proposed method may be sensitive to the type of reward signal and might not generalize equally well under different evaluation criteria.\n\n2. Because TreeGRPO adopts a variable tree structure, it introduces additional hyperparameters. As shown in Tables 3 and 4, the optimal hyperparameters differ across reward objectives. This means that although each fine-tuning run becomes more efficient, achieving the best performance may still require multiple fine-tuning rounds for hyperparameter search, resulting in the overall computational cost not being reduced.\n\n3. The paper contains several formatting and typographical issues, such as \"rewardds\" in the caption of Figure 1, \"Where\" that should be \"where\" on line 254, and an extra period on line 307. In addition, the manuscript does not include the required \"Use of Large Language Models\" statement. I recommend that the authors carefully review and correct these writing details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)