Dichotomous Diffusion Policy Optimization
Authors: Ruiming Liang
Yinan Zheng
Kexin ZHENG
Tianyi Tan
Jianxiong Li
Liyuan Mao
Zhihao Wang
Guang Chen
Hangjun Ye
Jingjing Liu
Jinqiao Wang
Xianyuan Zhan
Venue: ICLR 2026 Poster (2026)
Keywords: reinforcement learning
diffusion model
autonomous driving
robotics
Primary area: reinforcement learning

Abstract: Diffusion-based policies have gained growing popularity in solving a wide range of decision-making tasks due to their superior expressiveness and controllable generation during inference. However, effectively training large diffusion policies using reinforcement learning (RL) remains challenging. Existing methods either suffer from unstable training due to directly maximizing value objectives, or face computational issues due to relying on crude Gaussian likelihood approximation, which requires a large amount of sufficiently small denoising steps. In this work, we propose DIPOLE (Dichotomous diffusion Policy improvement), a novel RL algorithm designed for stable and controllable diffusion policy optimization. We begin by revisiting the KL-regularized objective in RL, which offers a desirable weighted regression objective for diffusion policy extraction, but often struggles to balance greediness and stability. We then formulate a greedified policy regularization scheme, which naturally enables decomposing the optimal policy into a pair of stably learned dichotomous policies: one aims at reward maximization, and the other focuses on reward minimization. Under such a design, optimized actions can be generated by linearly combining the scores of dichotomous policies during inference, thereby enabling flexible control over the level of greediness.Evaluations in offline and offline-to-online RL settings on ExORL and OGBench demonstrate the effectiveness of our approach. We also use DIPOLE to train a large vision-language-action (VLA) model for end-to-end autonomous driving (AD) and evaluate it on the large-scale real-world AD benchmark NAVSIM, highlighting its potential for complex real-world applications.

Published as a conference paper at ICLR 2026
DICHOTOMOUS DIFFUSION POLICY OPTIMIZATION
Ruiming Liang!.2+1, Yinan Zheng 3x 1, Kexin Zheng Tianyi Tan Jianxiong Li*,
Liyuan Mao Zhihao W Guang Chen', Hangjun Ye", Jingjing Liu', Jinqiao Wang1.2 1, Xianyuan Zhan
1 Fundation Model Research Center, Institute of Automation, Chinese Academy of Sciences
2 School of Artificial Intelligence, University of Chinese Academy of Sciences
3 Institute for AI Industry Research (AIR), Tsinghua University
The Chinese University of Hong Kong 5 Shanghai Jiao Tong University
6 Peking University Xiaomi EV
1 aggruiimngg wtw.eng22222111 i2a@ia.... ac. en, zhengyn23@male wow.glsstsssiuuueeeeee xhanxianyuan tsinghua....nn.. edu..cd cn
ABSTRACT
Diffusion-based policies have gained growing popularity solving a wide range of decision-making tasks due to their superior expressiveness and controllable generation during inference. However, effectively training large diffusion poli- cies using reinforcement learning (RL) remains challenging. Existing methods either suffer from unstable training due to directly maximizing value objectives, face computational issues due to relying crude Gaussian likelihood approx- imations, which require a large amount of sufficiently small denoising steps. In this work, propose DIPOLE (Dichotomous diffusion Policy improvement), a novel RL algorithm designed for stable and controllable diffusion policy opti- mization. We begin by revisiting the KL-regularized objective in RL, which of- fers desirable weighted regression objective for diffusion policy extraction, but often struggles balance greediness and stability. We then formulate a greedi- fied policy regularization scheme, which naturally enables decomposing the op- timal policy into a pair of stably learned dichotomous policies: one aims at re- ward maximization, and the other focuses reward minimization. Under such a design, optimized actions can be generated by linearly combining the scores of dichotomous policies during inference, thereby enabling flexible control over the level of greediness. Evaluations in offline and fflline-to-oline RL settings ExORL and OGBench demonstrate the effectiveness of our approach. We also use DIPOLE to train large vision-language-actioo (VLA) model for end- to-end autonomous driving (AD) and evaluate it on the large-scale real-world AD benchmark NAVSIM, highlighting its potential for complex real-world applica- tions. Project Page: https://rmbbbj http://mmbb..gituu..... gitthub.io/DIPOOO
INTRODUCTION
Due to the strong capability of diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020) in modeling multi-modal action distributions and controllable generation during inference (Dhariwal & Nichol, 2021; Ho & Salimans, 2022), modeling policies using diffusion models has become pop- ular choice in solving complex Docision-making tasks such as embodied roboties (Chi et al., 2023; Zheng et al., 2026) and autonomous driving (Zheng et al., 2025b; Tan et al., 2025; Li et al., 2026). Although proven to be effective in imitation learning-basee settings, training large diffusion/flow matching policies that surpass data-level performance with reinforcement learning (RL) (Sutton et al., 1998) has remained an important yet challenging direction.
"Equal contribution.
Corresponding authors.
Work done during internships at Institute of Al Industry Research (AIR), Tsinghua University.
Prroject lead.

---

Published as a conference paper at ICLR 2026
Training diffusion policies with RL faces numerous challenges, most notably, learning stability and computation efficiency. A naive approach to train diffusion policies with RL is to directly optimize the reward or value objective via gradient backpropagation through the multi-step denoising pro- cess (Xu et al., 2023b; Clark et al., 2023), which often suffers from noisy and unstable gradient updates, while also being extremely costly. To avoid this, some studies adopt compromise by freezing the diffusion model and instead searching for optimized noises (Wagenmaker et al., 2025; Hansen-Estruch et al., 2023), a strategy often referred as inference-time scaling (Ma et al., 2025b). However, these approaches rely heavily on rell-pretrained diffusion policies and are fundamentally limited by their performance upper bound. Another explored direction is to adopt policy gradient methods (such as PPO (Schulman et al., 2017)) for diffusion policy optimization, which models the denoising process as a multi-step Markov decision process (MDP) and uses Gaussian approx- imations to compute the log-likelihood of intermediate denoising steps (Black et al., 2024b; Ren et al., 2025). However, the crude Gaussian-basee approximation only provides reasonable likeli- hood information when adopting sufficiently small denoising steps, which inevitably results in large exploration spaces and prolonged training, making such methods difficult to scale and prone to ap- proximation error accumulation in practice. Therefore, a critical research question arises: Can we build more effective and stable RL method for diffusion policy optimization?
To answer this question, we turn our attention to the KL-regularized RL objective, which offers nice, closed-form weighted regression objective for optimal policy extraction (Peng et al., 2019). We can thus optimize a diffusion policy by incorporating an exponential reward- or value-based weighting term, scaled by temperature parameter, into the standard diffusion regression loss (Lee et al., 2023; Kang et al., 2023; Zheng et al., 2024). Although promising, this approach also suf- fers from several limitations. A fundamental issue is that weighted regression can only achieve greedy reward maximization when the temperature parameter is set to a large value, which easily leads to exploding loss and training instability. Moreover, the learning loss becomes dominated by a small number of high-reward samples, which severely undermines training effectiveness and scal- ability even with increased data (Park et al., 2024). To address the previous challenges, we propose DIPOLE (Dichotomous diffusion Policy improvement), a novel RL framework designed for highly stable and controllable diffusion policy optimization. Specifically, we introduce greedified KL- regularized RL objective, which regularizes policy learning towards a value-reweighted reference policy. Interestingly, we show that the original unstable exponential weighting term in the optimal policy can be decomposed into two bounded smooth dichotomous terms. This naturally allows us to decompose the optimal policy into a pair of stably learned dichotomous policies: one aims at reward maximization and the other focuses on reward minimization. Moreover, the optimized policy can be recovered through a linear combination of the scores from both dichotomous policies, which closely aligns with the widely used flassffier-rree guidance mechanism in diffusion models (Ho & Salimans, 2022), enabling perfect controllability over the greediness of action generation.
Extensive experimental results demonstrate the effectiveness of DIPOLE across a wide range of lo- comotion and manipulation tasks in EXORL (Yarats et al., 2022) and OGBench (Park et al., 2025a) benchmarks, evaluated under both offline and offline-to-online RL settings. Furthermore, we scale our learning approach to a large lanag---aguao-ttttion (VLA) model and evaluate it on the large- scale real-world autonomous driving benchmark NAVSIM (Dauner et al., 2024), showcasing signif- icant performance improvements over the pre-trained baseline. These results highlight the strong applicability of DIPOLE for complex, real-world decision-making scenarios.
2 PRELIMINARY
Reinforcement learning. We consider the RL. problem presented as a Markov Decision Process (MDP), which is specified by a tuple M == P,r,) S and A represent the state and ac- tion space: P : S x A + A(S) is transition dynamics; r : S x A + IR is the reward function; and Y E (0. is the discount factor. We aim to find a policy " : S + A(A) that maximizes the expected return: E., [E 2 r(skk ax)]. We define the discounted visitation distribution as: d (s((s) = (1 -1) Ee 7'p(sk = T), which measures how likely to encounter when interacting with the environment using policy æ. We also consider replay buffer D {s,, ai, & which can be a static dataset in the offline setting or dynamically updated with new samples in the offline-
to-online setting. The state-value function and action-value functions are defined as: V*(() =
E, [Et= ** -r(8, ax) | so s] and (s) FF, [E 7* r(8k, 88 = 8, ag a).
2

---

Published as a conference paper at ICLR 2026
and the advantage function is defined A" a) = (s.s.aa (s). Their optimal counter- parts under the optimal policy x' are denoted as V*, Q"" and A*.
Diffusion//low matching policies. Diffusion and flow matching models have attracted significant attention due to their strong expressiveness in capturing multi-moda data distributions, making them popular policy classes for complex decision-making tasks such robotics (Chi et al., 2023; Black et al., 2024a) and autonomous driving (Zheng et al., 2025b; Tan et al., 2025). The action generation state-conditional generation problem in which a probability path transforms source distribution (lypically a standard Gaussian) into a target action distribution. A neural network Eg is trained to predict the noise along the path using the objective over given dataset D:
can be formulated
Ley [ [Ile - 6o (ae, 8, 0)]]]
where ar = a,a+ + T&E (we use the subscript to distinguish diffusion steps from MDP steps k), with O₂ and 0; being predefined noise schedules commonly used in core-based diffusion models (Song et al., 2021) or flow matching models (Lipman et al., 2022). The multi-step diffusion process endows diffusion models with strong distribution-fiu fitting capabilities. However, it also poses challenges for RL fine-tuning: gradient propagation through the entire diffusion process is costly and unstable; the exact likelihood computation with diffusion models is intractable, causing a series of problems when optimizing with existing policy gradient RL methods due to approximation error.
3 METHODS
In this section, revisit the KL-regularized objective for diffusion policy optimization, revealing its strengths and limitations. We then introduce DIPOLE, a novel RL framework that decomposes the optimization problem into dichotomous policy learning objectives, thereby enabling stable training and greedy diffusion policy extraction.
3.1 KL-REGULARIZED OBJECTIVE IN RL
Reinforcement learning with KL regularization is a highly flexible framework that has been widely used in various RL settings, which constrains policy optimization to remain close reference policy and has the following general form:
max (s) E (a-r(als) [G(s, [)) - / DKI. (-1s) ll (-1s))
1
(2)
where B > 0 is the temperature parameter, and DKL (pll4) Euwp [log (p(x)/q(z)) G(-) is the evaluated return to be maximized, which can either be the reward function r(s, a) as in single-step problems such LLM RL fine-tuning (Korbak al., 2022; Shao et al., 2024), the action-value function Q* a) or advantage function A (s, a) as in standard multi-step settings. The specific choice of reference policy μ gives rise different RL task settings. For example, setting μ be the uniform distribution, we recover maximum entropy RL as in SAC (Haarnoja et al., 2018); setting y be the behavior policy in offline datasets D, we obtain many offline RL algorithms (Wu al., 2019; Xu et al., 2023a; Garg et al., 2023; Mao et al., 2024); lastly, setting j to be pre- pre-trained policy mo or the recently updated policy Th- 1, corresponds to offline-to-onlinn fine-tuning scenar- ios (Nakamoto et al., 2023; Li et al., 2023) or trust-region style online policy optimization (Schulman et al., 2015).
The flexibility of the KL-regularized RL framework makes it ideal choice for diffusion policy optimization. The best part is, it is known that the optimization objective in Eq. (2) also provides a closed-form solution for optimal policy xt follows (Nair et al., 2020):
(a s) x p(a s) exp(3G(ssaa).
Intuitively, the optimal policy is reweighted version of the reference policy μt, in which actions with higher values are assigned greater probability density. As shown in several existing studies (Kang et al., 2023; Zheng et al., 2024), if given a pre-trained diffusion policy eø trained with Eq. (1) as the reference policy μt, we can further optimize it with the weighted diffusion loss in Lemma I to extract the optimal diffusion policy
3

---

Published as a conference paper at ICLR 2026
Exp-Weighted
Dichotomous Policy Extraction
Controllable Inference
exp o)
o(c(s, 1- (G)(
du
Clip
Valuue landascape
G(s, 0)
2.0
Illustration of the policy weighting scheme in DIPOLE. Based greedified policy optimization objective, the regression weight of the optimal policy be decomposed into pair of dichotomous terms, and the greediness reward/value maximization can be flexibly controlled by w.
Figure 1:
Lemma 1.
We can generate optimal x² (a|s) in Eq. (3) by optimizing the weighted diffusion loss Eg. (4) and solving the diffusion reverse process with obtained e* (Zheng al., 2024).
Lep
(sx()3((saa) 16c-ep (ar, 8, 1))
Compared diffusion-based RL methods that rely on unstable reward/value maximization (Xu et al., 2023b; Clark et al., 2023) or biased likelihood approximation (Black et al., 2024b; Ren et al., 2025), Eq. (4) offers a simple and scalable training scheme for policy optimization, requiring only the addition of a weighted term to the base diffusion learning objective in Eq. (1). Despite its sim- plicity, we do not observe the adoption of this scheme in many recent diffusion- RL methods. Why is that? Actually, there exist several limitations for this exp-weighted regression scheme:
sttibiltty trade-off. As the exponential function exp(-) grows rapidly, a high-quality action with a large G[s, a) value can lead to extremely large weight term when B is large, caus- ing the explosion of learning loss and destabilizing the training process (illustrated in Figure I). practice, many methods mitigate this issue by either using small / or clipping the weighting term (Garg et al., 2023; Xu et al., 2023a; Mansen-Estruch et al., 2023). However, these treatments compromise the optimality of the extracted policy.
Inefficient learning. The training loss becomes dominated by small number of high high-return sam- ples, which is inefficient for policy optimization (Park et al., 2024). Additionally, poor-quality samples still retain positive weight, which can adversely affect policy learning. The constrained optimization objective also makes the learning process highly dependent the quality of the reference policy 14, thereby limiting the potential greedy policy optimization.
3.2 DICHOTOMOUS DIFFUSION POLICY IMPROVEMENT
To address the drawbacks of the previous weighted regression scheme while preserving its simplicity and scalability, instead consider greedified KL-regularized RL objective.
Greedified policy optimization. We begin by formulating greedier learning objective compared Eq. (2), presented in Eq. (5). At first glance, it appears be complex; however, will show in the later derivation, its resulting closed-form optimal solution can lead to remarkably elegant form for effective diffusion policy optimization.
1 wB
DKL. T (-18) Il (-1s) o (BG(s, a)) ]
max (e) Basx(als) [G(s, a))
Z(s)
In this revised objective, instead regularize policy with greedified, value-aware refer- ence policy weighted by a ((BG(s,a)) (s), where (s)) denotes the normalization factor and (xx) 1/(1 + exp(-x:) the sigmoid function. This design shares similar spirit with some
4

---

Published as a conference paper at ICLR 2026
offline RL methods that enhance policy performance by regularizing towards greedier behavior policy or reward-wert -eighted datasets (Singh et al., 2022; Hong et al., 2023; Xu et al., 2025). It is a bounded and smooth sigmoid function the weighting function, which
worth noting that greedily assigns high weights to high-return samples while avoiding numerical instability. Moreover, introduce a hyperparameter w, termed the greediness factor, which provides an additional interface for adjusting the greediness of policy extraction. We will reveal its role in the later deriva- tion. Based the optimization objective in Eq. (5), we can get its closed-form solution follows:
Theorem 1. The optimal solution for Eq. (5) satisfies:
x² x=(a 8) x p(a) 8) (3G(s, a)) exp(w- (G(s,)))(s)))))
(6)
Proof of this theorem can be found in Appendix B. The optimal solution corresponds to a value- aware reference policy with special weighting scheme, where both 3 and the greediness factor control the level greediness in the resulting policy. Next, we will show how this solution enables natural decomposition into pair of dichotomous policies.
Dichotomous policy extraction. Leveraging the property of the sigmoid function, it's easy to show:
π (a 8) x ja(a
8) a (BG(s,a) explw (G((s a))
8) a (BG(s,a)) ( a (BG(s,a) n' (a x p(a
1- (BG(s,a)) (1-()C((,0))
<a 8) x [u(a|s) (3G(s, /[((a s). (1-o0 (BG(s,
Eq. (7) suggests that the optimal policy can actually be expressed the ratio of two weighted reference policies with distinct exponents and weighting functions. Specifically, we can define a positive policy t and negative policy as:
ta 8) & p(a 8) a)) (T x p(a (1 (1-0(
(8)
where the positive policy π aims to maximize the return and the negative policy π minimizes it. We call t and dichotomous policies, they share similar form but with opposite focuses. With this definition, the optimal policy can be simply expressed as x² [ +1+++))) [[r Careful readers will notice that both π and π are weighted by strictly bounded sigmoid weight functions, instead of the unstable and unbounded exponential weight term exp(BG(s, a)) in the optimal solu- tion of the original KL-regularized objective Eq. (3). This means that the decomposed dichotomous policies can be stably trained, precluding loss explosion as the positive policy π+ prioritizes learning from high-return samples, while the negative policy prioritizes learning from low-return samples, we can thus simultaneously utilize both good and bad data for policy optimization, completely resolving the issue of being dominated by high-return samples as in exp- exp-weeghted regression, and enabling more efficient learning.
discussed in Section 3.1. Moreover,
Following Lemma 1, we can train the positive and negative policies π and using two diffusion models with their bounded sigmoid weight functions, parameterized as ct, and cg.-
Cej E...jo..) (o (BG(s, ||c- Ct, (at, (11
(9)
(1 a (BG(s, a))) ||c- Co, (at, s, [1
L
Controllable generation. To sample from the optimal policy x*, note that based on Eqs. (7-8),
logr"(a (a 8) = w) log (a s) w log π (a + log C (1 () log ग + (a
V lo Ta
w V log (a s).
(10)
where C' is a constant. This shows that the score function of the optimal policy n² can be expressed as a linear combination of scores of the dichotomous policies, weighted by w. Due to the inherent connection between the score function and the noise predictor in diffusion model (Ho et al., 2020),
we can use à (a,, s, f) = (1 + w)e (a,, We,, (a,, s, t) in the reverse process of diffusion or flow matching for action sampling.
5

---

Published as a conference paper at ICLR 2026
Interestingly, the formulation in Eq. (10) is remarkably similar to classifier- guidance (CFG) (Ho & Salimans, 2022), a popular method for enhanced conditional diffusion generation, which has the form of E(Zz; c) (1 + (2), t) wed (2,,),, t). where Ez c, is conditioned version of <0( () with conditioning signal c. This reveals the inherent connection between our greedified Kl.rregllarized RI. objective and the CFG mechanism. Intuitively, our method further strengthens the positive distribution by pushing the negative distribution in the opposite direction, thus enabling flexible control of the optimality level of generated actions with the greediness factor w (see il- lustration in Figure 1). Our final formulation also has some similarity with CFGRL (Frans et al., 2025), which can be perceived as setting . x 4 IA20 and π = ş (A is the advantage function). However, their method lacks theoretical backing, and using identical weights for both positive and negative samples limits the greediness of policy optimization, leading to suboptimal performance.
3.3 PRACTICAL. IMPLEMENTATIONS
Offline and offline-to-online RL. For standard multi-step RI. settings, we can set G(s, a) as the advantage function A(s, a). In the offline RI. setting, the reference policy μ in Eq. (5) corresponds to the behavior policy T3 of the offline datasets. In the offline-to-online setting, the reference policy set as the policy updated in the previous step πk-1 (To the offline pre-trained policy). The algorithm pseudocode and additional implementation details are provided in Appendix C and D.
End-to-end autonomous driving. We also implement DIPOLE to train large end-to-end au- tonomous driving model to demonstrate its scalability to solve real-world complex tasks. Specif- ically, we employ a non-reactive pseudo-closed-logg simulation based on real-world datasets for policy training. In this setup, the return G(s,, a) is defined by reward function that evaluates trajec- tory quality based on safety, progress, and comfort. We employ vision-language model (Florence- 2 (Xiao et al., 2024)) as the encoder and a diffusion action head as the decoder (Zheng et al., 2025b). The model processes images from the left-front, front, and right-front cameras, along with lan- guage instructions such "turn left", "turn right". and "go straight". This architecture results in a 1-billion parameter model, which we name DP-VLA, and is pre-trained using imitation learning. Subsequently, two separate LoRA modules are applied to the decoder to construct the positive and negative policies, allowing us to leverage Eq. (9) for training. We follow the offline-to-online RI. setting to fine-tune the VLA model. Further implementation details are provided in Appendix E.
4 EXPERIMENTS
4.1 EXPERIMENTS ON RL BENCHMARKS
Experimental setup. We evaluate our approach on two commonly-used benchmarks, OG- Bench (Park al., 2025a) task suite and EXORL (Yarats et al., 2022) benchmark. OGBench provides challenging robotic locomotion and manipulation tasks, including complex whole-body humanoid control, maze navigation, and object manipulation. We use the default dataset collected by RND (Burda et al., 2019), including tasks in complex high-dimensional state-based domains: Walker, Quadruped, Jaco, and Cheetah. Our evaluation encompasses 30 tasks across 6 domains on OGBench and 9 tasks across 4 domains on EXORL for offline learning, totaling 39 tasks. Finally, we select default tasks across domains on OGBench for offline-to-online validation. Further details are provided in Appendix D.
Baselines. We use representative baselines across policy types for comprehensive comparison:
Gaussian policy. Standard RI. uses Gaussian policies by default. In comparison with standard methods, we select 1) IQI. (Kostrikov al., 2022): typical weighted regression offline RI. method. 2) ReBRAC (Tarasov et al., 2023): an effective behavior-regularized actor-critic approach incorporates several specific designs tailored for offline learning.
Difffusion/Flow policy. We also include offline RL baselines built on diffusion or flow poli- cies according to the following learning strategies: 1) IDQL (Hansen-Estruch et al., 2023) and IFQL (Park et al., 2025b): both approaches employ expectile regression for value learning and utilize imitation pre-trained diffusion or flow models with rejection sampling during inference. 2) FQL (Park et al., 2025b): a behhaiir-regularized actor-critic variant that uses flow policy distilla- tion and shows strong performance on OGBench. 3) CFGRI. (Frans et al., 2025): recently pro-

---

Published as a conference paper at ICLR 2026
Table 1: EXORL Results. We report the average performance. (w/o Is: without rejection sampling)
8 random seeds. DIPOLE achieves the best
Gaussian Policy
Diffusion/Flow Policy
Domain
Task
IQL
ReBRAC
CFGRL
IFQL
FQL
DIPOLE w/o
DIPOLE
stand
603
461:23
782
873:1
801::4
793
953 L:4
Walker
walk
444-4
208
608-332
844::11
755::12
679::16
910::5
run
247:
98 1:2
2
406
294:111
256::11
442 ::9
Quadruped
15
344:1:7
762.:22
883:112
739.:25
813:221
928 1:55
485
344::3
571:::25
595:::18
503:::5
560::11
657 10
Cheetah
run
168::7
97
216::15
269::11
222:1:14
194=9
274 12
uun-backward
146::
262
310::244
231 :312
227=7
350::15
reach top-right
33:12
13
72:1:
193.
224
84:55
117:118
reach-top-left
30:18
39:15
46:26
181::11
222:1-42
63:18
110: 12
Table 2:
OGBench Results. We report the aggregate score on all single tasks for each category, averaging over 8 random seeds. DIPOLE achieves best near-best performance against other baselines across challenging task categories. See appendix D.I for results.
Gaussian Policy
Difiusion/Flow Policy
Task Category
IQL
ReBRAC
IDQL
IFQL
FQL
DIPOLE
humanoidmaze-mediun navigate (5 tasks)
33:12
2:8
1:0
60
58:55
1
humanoidmaze largennnezzze-ennaaaa large-navigate (5 tasks)
2:11
2:11
1::0
11 1:222
4
6:12
niviinn tasks)
8::2
0.:0
12:14
33:16
60:12
57::7
cube-single-play (5 tasks)
83:13
91:12
95
79-12
96:11
97-12
cube-double-play (5 tasks)
7+11
12:
15:16
14:13
29:12
44
scene-play (5 tasks)
41:53
46:13
30:13
56:12
posed policy improvement framework relies on classifier-free guidance, which uses high-quality actions for conditional policy training and unconditional behavior cloning.
For the offline RL setting, we compare our approach with IQL, ReBRAC, CFGRL, IFQL, and FQL on the EXORL benchmark. We also include variant, DIPOLE w/o rs, which does not rejection sampling during inference for clear comparison. For IFQL, we utilizes the default hyperparameters, and for FQL, we select the hyperparameter a reported in previous work (Park al., 2025b) with optimal performance in EXORL. Additionally, we compare with IQL. ReBRAC, IDQL, IFQL, and FQL on OGBench demonstrate our method's effectiveness against state-of-the-arr approaches in challenging benchmark.
Evaluation results. We evaluate the performance of each method after fixed number of offline updates for the offline RL setting. Specifically, we report the average return on EXORL and the success rate on OGBench, following the standard evaluation assessment methods (Yarats et al., 2022; Park et al., 2025a). For the
evaluation, we assess the final performance on a fixed number of online updates following the formal offline pretraining stage. All evaluations are averaging over 8 random seeds, with + indicating standard deviations in tables. We present our results by answering the following questions:
Can DIPOLE outperform prior SOTA algorithms offline setting? Table reports per-task comparison results on EXORL. DIPOLE outperforms other baselines in most domains, indicating its capability to fully utilize valuable data dataset. Specifically, DIPOLE fully surpasses IQL, indicating its strong improvement over the Gaussian policy-based weighted regression method. Furthermore, DIPOLE w/o rs demonstrates better performance compared to CFGRL, highlight- ing the importance of our design for achieving more greedy policy optimization. Finally, Table summarizes the aggregate benchmarking results on OGBench. In most task categories, DIPOLE achieves better performance compared to other baselines, demonstrating its strong capability in

---

Published as a conference paper at ICLR 2026
Table 3: OGBench Offinne-to-Olliee Results. We report the score the default task for each category,
averaging over 8 random seeds. (humanoidmaze-m humanoidmaze-mediinn
Gaussian Policy
Diffusion/Flow Policy
Task Category
IQr.
ReBRAC
IFQI.
FQr.
DIPOLE
humanoidmaze-n
21::13 16
16::20
56::35 + 82::20
12:27
22:112
61::10 + 97:22
antsoccer-aie
2 0::0
0::0 + 0::00
26:1:5 39:110
28
86:25
43:14 + 1:
cube-double
0=11
+ 0::0
6 - 28 28
12::9 - 40:55
40-:11
+ 92:33
41::6 + 89 39-=10
scene
14::11 + 10:19
55:110 + 100
0::1
+ 60
82::1 + 100
97:144 100::0
Table 4: NAVSIM Closed-Loop Results. We scale up DIPOLE large VLA model, demonstrating its
potential for real-world applications. (navtrain/navtest represent different data splits used trajectory rollout)
Method
Input
NC1
DACT
TTC1
Comf.1 1
EPT
PDMST
Constant Velocity
68.0
57.8
50.0
100
19.4
20.6
Ego Status MLP
93.0
77.3
83.6
100
62.8
65.6
UniAD
Cam
97.8
91.9
92.9
100.0
78.8
83.4
PARA-Drive
Cam
97.9
92.4
93.0
99.8
79.3
84.0
LFT
Cam
97.4
92.8
92.4
100
79.0
83.8
Transfuser
Cam & Lidar
97.7
92.8
92.8
100.0
79.2
84.0
Hlyra-MDD
Cam & Lidar
98.3
96.0
94.6
100.0
78.7
86.5
DP-VLA (ours)
Cam
98.0
97.0
94.3
100.0
82.5
88.3
DP-VLA w/ DIPOLE navtrain (ours)
Cam
98.2
98.0
95.2
100.0
83.6
89.7
DP-VLA w/ DPPO navtest
Cam
97.9
97.6
94.1
100.0
83.5
89.0
DP-VLA DIPOLE naviest (ours)
Cam
99.2
98.7
95.6
99.8
94.8
solving challenging long-horizon tasks. These results confirm that weighted regression can effec- tively achieve greedy policy extraction across robotic locomotion and manipulation RL tasks.
How does DIPOLE perform with online finetuning? Table 3 reports the exact performance vari- ation after 1M of online updates. We demonstrate that method can be successfully applied to online fine-tuning settings. Compared to IFQL, achieves a higher performance upper bound. When compared to the direct value maximization approach in FQL, our method shows competitive performance, demonstrating the effectiveness of our design for achieving both greedy and stable policy optimization. Moreover, provide pixel-based online fine-tuning results in end-to-end autonomous driving later, further demonstrating the effectiveness of our approach.
Moreover, refer to Appendix D.4 for ablation studies.
4.2 EXPERIMENTS ON AUTONOMOUS DRIVING BENCHMARK
Experimental setup. Our method is evaluated the large-scale real-world autonomous driving benchmark NAVSIM (Dauner et al., 2024) using closed-loop assessment. Following the official eval- uation protocol, we report the PDM score (higher indicates better performance). which aggregates five key metrics: NC (no-collision rate), DAC (drivable area compliance). TTC (time-to-collision safety), Comfort (accelcration/jerk constraints), and EP (ego progress). All methods are tested un- der the official closed-loop simulator, and results are averaged over the public test split. We also consider R application scenario where RI. can be applied in human take-over situations or com- plex environments lacking ground-truth supervision. To address this, we provide variant of our model trained on the test split without using any ground-truth.
Baselines. We select several baselines: 1) UniAD (Hu et al., 2023): integrates multiple auxiliary tasks such tracking, mapping, prediction, and occupancy prediction using transformer blocks, and employs latent representations for planning. 2) PARA-Drive (Weng et al., 2024): adopts a parallel architecture design compared to UniAD. 3) Transfuser (Chitta et al., 2023): fuses image and LiDAR information through a dual-bran branch architecture and incorporates detection and BEV semantic maps for auxiliary supervision. Its latent variant, LFT, replaces LiDAR inputs with learaale embeddings. 4) Hydra-MDD (Li et al., 2024): winner the CVPR2024 Challenge, which uses trajectory anchors and learned reward model for anchor selection. Moreover, we also consider baselines where the
8

---

Published as a conference paper at ICLR 2026
Figure 2: NAVSIM Results: DP-VLA w/ DIPOLE fine-tuned model trajectory; ground truth ego trajectory; DP-VLA imitation pretrained model trajectory.
agent either maintains its current state or uses a simple MLP for trajectory regression. For our imitation pre-trained VLA model, which directly generates trajectories without post-processing, denote it as DP-VLA. When fine-tuned with DPPO(Ren et al., 2025), we refer to them as DP-VLA w/ DP-VLA w/ DIPOLE. As mentioned
DPPO. When fine-tuned with our RL algorithm, refer above, we also provide two variants trained on the navtrain and navtest splits.
Evaluation results. We present the experimental results Table 4. Notably, our imitation-based VLA model significantly outperforms other baselines, providing strong foundation for RL. fine- tuning. Building on this, fine-tuning with DIPOLE on the navtrain dataset improves the PDMS score by 1.4 points (from 88.3 to 89.7), with gains observed in both safety and progress metrics. Further- more, DIPOLE fine-tuning on navtest scenarios yields a substantial 6.5-point PDMS improvement (from 88.3 to 94.8), demonstrating its potential for real-world autonomous driving applications. These results confirm that even for large-scale policies exceeding billion parameters, DIPOLE consistently delivers significant performance improvements through stable and greedy policy op- timization. To further illustrate the efficacy of the DIPOLE fine-tuned model, we present several cases in Figure where the pretrained model fails but succeeds after DIPOLE fine-tuning, Notably, DIPOLE enables DP-VLA to mitigate compounding errors and low-level controller tracking errors, effectively correcting trajectories to prevent collisions and erratic driving behavior.
RELATED WORK
Reinforcement fine-tuning of diffusion models remains challenging due their multi-step diffusion process, primarily in terms of learning stability and computational efficiency. A brute-force solu- tion involves directly optimizing the reward via gradient backpropagation. ReFL (Xu al., 2023b) optimizes human preference scores for image generation by backpropagating gradients at specific single steps during the reverse process. DRaFT (Clark et al., 2023) extends this approach by ap- plying gradient optimization across multiple steps at the end of the reverse process. Such methods are widely used in motion generation (Karunratanakul al., 2024), image generation (Prabhudesai et al., 2023), and decision-making tasks (Wang et al., 2022), but suffer from instability due to noisy gradient backpropagation during the demoising process. Some methods avoid gradient computation and instead search for the optimal noise to maximize reward, strategy referred to inference-ting scaling (Hansen-Estruch Estruch et al., 2023; Ma et al., 2025b; Singhal al., 2025). Recent approaches also utilize RL to directly search for the best noise (Wagenmaker et al., 2025). In both cases, perfor-

---

Published as a conference paper at ICLR 2026
mance remains constrained by the capabilitiies of the pre-trained model. Moreover, DDPO (Black et al., 2024b) treats each noise step as a Gaussian distribution, enabling likelihood estimation and optimization via the REINFORCE (Mohamed et al., 2020) algorithm. DPPO (Ren et al., 2025) optimizes this approach and extends it to multi-step MDPs using PPO (Schulman et al., 2017) for policy improvement These methods rely Gaussian approximations that require sufficiently small sampling steps, resulting in inefficient training. Some methods (Lee et al., 2023; Kang et al., 2023; Zheng et al., 2024; Ma et al., 2025a; Zheng et al., 2025a) use KL-regularized RL (Kostrikov al., 2022; Peng et al., 2019), whose solution leads to simple weighted regression loss. However, these approaches often face a trade-off between greediness and stability.
CONCLUSION
We propose DIPOLE, an RL method that enables stable and controllable diffusion policy optimiza- tion. We revisit KI. Kl-regularized RI., which suffers from a trade-off between greediness and stability, and introduce greedified policy regularization scheme. This scheme decomposes the optimal pol- icy into dichotomous policies with stable training losses. During inference, actions are generated by linearly combining the scores of these policies, enabling controllable greediness. We evaluate DIPOLE widely used RL benchmarks to demonstrate its effectiveness and also train large VLA model for end-to-end autonomous driving, highlighting its potential for real-world applications. Due to space limit, more discussion limitations and future direction can be found in Appendix F.
ACKNOWLEDGEMENT
This work is supported by Xiaomi EV and National Natural Science Foundation of China under Grant 62.6227,,60, funding from Wuxi Research Institute of Applied Technologies, Tsinghua Uni- versity under Grant 20242001 120 and the Xiongan Al Institute. Furthermore, Jianwei Cui and Kun Ma in Xiaomi EV for their resource support. We would like to express our gratitude to Bin Huang, Enguang Liu and Jianlin Zhang in Xiaomi EV for their valuable discussion.
would like thank
REFERENCES
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubezak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky. mo: A vision- language-action flow model for general robot control, 2024a. URL https: /ttss///axxvvoooo/ee abs/2410.244
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. In International Conference on Learming Representations, 2024b.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In The Seventh International Conference on Learning Representations, 2019.
Cheng Chi, Siyuan Feng. Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Proceedings of Science and Systems (RSS), 2023.
Robotics:
Kashyap Chitta, Aditya Prakash, Berhhard Jaeger, Zehao Yu, Katrin Renz, and Andreas Geiger. Transfuser: Imitation with transformer- based sensor fusion for autonomous driving. Pattern Anal- ysis and Machine Intelligence (PAMI), 2023.
Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. Directly fine-tuning diffusion models on differentiable rewards. In The Twelfth International Conference on Learning Representations, 2023.
Daniel Dauner, Marcel Hallgarten, Tianyu Li, Xinshuo Weng, Zhiyu Huang, Zetong Yang, Hongyang Li, Igor Gilitschenski, Boris Ivanovic, Marco Pavone, Andreas Geiger, and Kashyap
10

---

Published as a conference paper at ICLR 2026
Data-driven non-reactive autonomous vehicle simulation and benchmarking. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
Chitta. Navsim:
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-98 099888097774 2021.
Kevin Frans, Seohong Park, Pieter Abbeel, and Sergey Levine. Diffusion guidance is controllable policy improvement operator. arXiv preprint arXiv:2505.23456 2025.
Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learning: Maxent rl
without entropy. In The Eleventh International Conference on Learning Representations, 2023.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning, pp. 1861-1870. Pmir, 2018.
Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine Idql: Implicit q-learning as actor-critic method with diffusion policies. arXiv preprint arXiv:2304. 10573, 2023.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance, 2022. URL BelLhttps://aaa org/ htg//bb/// 2207. 12598.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Zhang-Wei Hong, Aviral Kumar, Sathwik Karnik, Abhishek Bhandwaldar, Akash Srivastava, Joni Pajarinen, Romain Laroche, Abhishek Gupta, and Pulkit Agrawal. Beyond uniform sampling: Offline reinforcement learning with imbalanced datasets. Advances in Neural Information Pro- cessing Systems, 36:4985-5000 2023.
Yihan Hu, Jiazhi Yang. Li Chen, Keyu Li, Chonghao Sima. Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al. -oininng-rriented autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17853-17862 2023.
Bingyi Kang, Xiao Ma, Chao Du, Tianyu Pang, and Shuicheng Yan. Efficient diffusion policies for offline reinforcement learning. Advances Neural Information Processing Systems, 36:67195- 67212, 2023.
Korrawe Karunratanakul, Konpat Preechakul, Emre Aksan, Thabo Beeler, Supasorn Suwajanakomn, and Siyu Tang. Optimizing diffusion noise can serve universal motion priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1334-1345, 2024.
Tomasz Korbak, Ethan Perez, and Christopher Buckley. RI with kl penalties is better viewed as bayesian inference. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 1083-1091, 2022.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. In International Conference Learning Representations, 2022.
Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, and Pieter Abbeel. URLB: unsupervised reinforcement learning benchmark. In Proceed- ings of the Neural Information Processing Systems Track on Datasets and Benchmarks NeurIPS Datasets and Benchmarks 2021, December 2021, virtual, 2021.
Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh. and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint arXiv:2302. 12192, 2023.
Jianxiong Li, Xiao Hu, Haoran Xu, Jingjing Liu, Xianyuan Zhan, and Ya-Qin Zhang. Proto: Iterative policy regularized offline-to-onling oo-one reinforcement learning. arXiv preprint arXiv:2305. 2023.
11

---

Published as a conference paper at ICLR 2026
Pengxiang Li, Yinan Zheng, Yue Wang. Huimin Wang, Hang Zhao, Jingjing Liu, Xianyuan Zhan, Kun Zhan, and Xianpeng Lang. Discrete diffusion for reflective vision- saiio-language-actee models in autonomous driving. In International Conference on Learning Representations, 2026.
Zhenxin Li, Kailin Li, Shihao Wang, Shiyi Lan, Zhiding Yu, Yishen Ji, Zhiqi Li, Ziyue Zhu, Jan Kautz, Zuxuan Wu, et al. Hydra-mdp: End-to-end multimodal planning with multi-targer hydra- distillation. arXiv preprint urXiv:2406.069975 2024.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow match- ing for generative modeling. In The Eleventh International Conference on Learning Representa- tions, 2022.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022.
Haitong Ma, Tianyi Chen, Kai Wang, Na Li, and Bo Dai. Efficient online reinforcement learning for diffusion policy. In Forty-second International Conference Machine Learnnng 2025a.
Nanye Ma, Shangyuan Tong, Haolin Jia, Hexiang Hu, Yu-Chuan Su, Mingda Zhang, Xuan Yang, Yandong Li, Tommi Jaakkola, Xuhui Jia, et al. Inference-time scaling for diffusion models beyond scaling denoising steps. arXiv preprint arXiv:2501 097322, 2025b.
Liyuan Mao, Haoran Xu, Weinan Zhang, and Xianyuan Zhan. Odice: Revealing the mystery of distribution correction estimation via aoroogoonalggddiiieg update. In The Twelfth International Conference on Learning Representations, 2024.
Shakir Mohamed, Mihaela Rosca, Michael Figurnov, and Andriy Mnih. Monte carlo gradient esti-
mation in machine learning. Journal of Machine Learning Research, 21(132):1-62, 2020.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online rein- forcement learning with offline datasets. arXiv preprint arXiv:2006.09:35 2020.
Mitsuhiko Nakamoto, Simon Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline rl pre-training for efficient online fine- 2023.
tuning. Advances in Neural Information Processing Systems,
Seohong Park, Kevin Frans, Sergey Levine, and Aviral Kumar. Is value learning really the main bottleneck in offline ri? Advances in Neural Information Processing Systems, 37:79029-79056. 2024.
Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Ogbench: Benchmarking offline goal-condittoned rl. In International Conference Learning Representations, 2025a.
Seohong Park, Qiyang Li, and Sergey Levine. Flow q-leaniigg In International Conference on Machine Learning (ICML), 2025b.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weigghted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:19100001177, 2019.
Mihir Prabhudesai, Anirudh Goyal, Deepak Pathak, and Katerina Fragkiadaki. Aligning text-to- image diffusion models with reward backpropagation. arXiv preprint arXiv:2310.03739, 2023.
Allen Z Ren, Justin Lidard, Lars Lien Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Ma- jumdar, Benjamin Burchfiel, Hongkai Dai, and Max Simchowitz. Diffusion policy policy opti- mization. In International Conference on Learning Representations, 2025.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning. pp. 1889-1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347, 2017.
12

---

Published as a conference paper at ICLR 2026
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi. Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Anikait Singh, Aviral Kumar, Quan Vuong, Yevgen Chebotar, and Sergey Levine. Offline with realistic datasets: Heteroskedasticity and support constraints. arXiv preprint arXiv:2211.01052
2022.
Raghav Singhal, Zachary Horvitz, Ryan Techan, Mengye Ren, Zhou Yu, Kathleen McKeown, and Rajesh Ranganath. A general framework for inference-time scaling and steering of diffusion models. In Forty-second International Conference on Machine Learning. 2025.
Jascha Sock-Dicks,ein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics, 2015.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations, 2021.
Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
Tianyi Tan, Yinan Zheng, Ruiming Liang, Zexu Wang, Kexin Zheng, Jinliang Zheng, Jianxiong Li, Xianyuan Zhan, and Jingjing Liu. Flow matching- autonomous driving planning with advanced interactive behavior modeling. In The Thirty-ninth Annual Conference on Neural Infor- mation Processing Systems, 2025.
Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the min- imalist approach to offline reinforcement learning. Advances in Neural Information Processing Systems, 2023.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap, and Martin Ried- miller. Deepmind control suite, 2018. URL httttp:://axxi..orr aab/ http//abb//110..///////11//1//1/ 110111111111 00690.
Andrew Wagenmaker, Mitsuhiko Nakamoto, Yunchu Zhang, Seohong Park, Waleed Yagoub, Anusha Nagabandi, Abhishek Gupta, and Sergey Levine. Steering your diffusion policy with latent space reinforcement learning. arXiv preprint arXiv:2506. 15799, 2025.
Zhendong Wang, Jonathan Hunt, and Mingyuan Zhou, Diffusson policies as an expressive policy class for offline reinforcement learning. In The Eleventh International Conference on Learning Representations, 2022.
Xinshuo Weng, Boris Ivanovic, Yan Wang, Yue Wang, and Marco Pavone. Para-drive: Parallelized architecture for real-time autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2024.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning.
arXiv preprint 1911v:1911.1136/ 2019.
Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2. Advancing a unified representation for variety of vision tasks. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4818- 4829, 2024.
Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang. Victor Wai Kin Chan, and Xi- anyuan Zhan. Offline rl with ood actions: In-sample learning via implicit value regularization. In The Eleventh International Conference on Learning Representations, 2023a.
Haoran Xu, Shuozhe Li, Harshit Sikchi, Scott Niekum, and Amy Zhang. An optimal discriminator weighted imitation perspective for reinforcement learning. arXiv preprint vrXiv:250411336 2025.
13

---

Published as a conference paper at ICLR 2026
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li. Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36:15903-15953 2023b.
Denis Yarats, David Brandfonbrener, Hao Liu, Michael Laskin, Pieter Abbeel, Alessandro Lazaric, and Lerrel Pinto. Don't t change the algorithm, change the data: Exploratory data for offline reinforcement learning. arXiv preprint 2Xiv:2201.133222 2022.
Candi Zheng and Yuan Lan. Characteristic guidance: Non-linear correction for diffusion model at large guidance scale. In International Conference on Machine Learning, pp. 61386-61412. PMLR, 2024.
Jinliang Zheng, Jianxiong Li, Zhihao Wang, Dongxiu Liu, Xirui Kang, Yuchun Feng, Yinan Zheng, Jiayin Zou, Yilun Chen, Jia Zeng, al. X-vla: Soft-prompted transformer scalable cross- embodiment vision-language-action model. In International Conference on Learning Represen- tations, 2026.
Kexin Zheng, Lauriane Teyssier, Yinan Zheng, Yu Luo, and Xianyuan Zhan. Towards robust zero- shot reinforcement learning. In The Thirty-ninti Annual Conference on Neural Information Pro- cessing Systems, 2025a.
Yinan Zheng, Jianxiong Li, Dongjie Yu, Yujie Yang. Shengbo Eben Li, Xianyuan Zhan, and Jingjing Liu. Safe offline reinforcement learning with feasibility-guide diffusion model. In The Twelfth International Conference on Learning Representations, 2024.
Yinan Zheng, Ruiming Liang, Kexin ZHENG, Jinliang Zheng, Liyuan Mao, Jianxiong Li, Wei- hao Gu, Rui Ai, Shengbo Eben Li, Xianyuan Zhan, and Jingjing Liu. Diffusion-based planning for autonomous driving with flexible guidance. In The Thirteenth International Conference on Learning Representations, 2025b.
14

---

Published as a conference paper at ICLR 2026
A LLM USAGE
In this paper, we employed Large Language Models (LLMs) solely for polishing the writing. No parts of the technical content, experimental results, conclusions were generated by LLMs.
B THEORETICAL INTERPRETATIONS
We define problem under the reinforcement learning problem presented as Markov Decision Process (MDP) (Sutton et al., 1998) given by M = AS r. r)),, which comprises a state space S, action space A. state transition S, reward function and discount factor 7. In this setting, a policy is probability distribution actions conditioned on a state. In addition, we assume that all policies induce irreducible Markov Chain, with any states reachable from each other by sequence of transitions that have positive probability. Our goal to find policy ı that maximizes predefined action evaluation criteria G. constrained on reference policy.
Theorem 1. The optimal solution for Eq. (5) satisfies:
x xa 8) x p(a) 8) a (3G(s, a)) exp(w (G(s.)))
Proof. Consider the optimization problem Eq. (5) with constraints the probability distribution: max E Eend' (s) [unn((a a)] -
1 w3 DKa(T(- ((+11 ((G((a)))))
2(x)
s.t.
T(a 8) da = 1, vs
(11)
x(a 8) 2 0, Vs,a
The Lagrangian is given by:
[ (s) a(a s)G(s,
L(M, Clsy Yssa)
[ (s) wB [ x(a s) ( T(a s)
do ds (12)
p(a ()o
1 as (ʃ. M(a s)da s)da -1) ds + / Ta s)dads
Take the derivative over (aal s) and set to zero:
OL
1
log llog T(a s) + 1- log m(aa ()o(3G((.a)) ) + 1saa (13)
@ι(a = G(s,a)
wB
Z(s)
Solve the equation and one obtain the optimal policy as:
(a s) = pla| ()G(GG(s a)) exp(w a)) exp wB O +/.0 - log 2(s)
(14)
d=(s)
Note that since we assume all policies induce irreducible Markov chain, thus d= (s) > 0, Vs. Con- sider the support of μ with positive probability and the final resulted optimal policy satisfies:
x (a 8) x p(a s) o(BG(s, exp(w (G(s,
(15)
As shown in Eq. (10), the score function the target optimal policy and the dichotomous policies satisfy the linear combination = 0:
(a(a)s) = (1 + w (a|s) weD (als)
(16)
With the marginal condition satisfied, obtain the exact intermediate score by (Zheng & Lan, 2024):
(arls) (1 w) (at warlls) we, (at (1 + w: Asy
(17)
15

---

Published as a conference paper at ICLR 2026
where Aa, is non-linear correction term satisfying the following equation:
Aa = e (a+ + (1 w) - e (a, + w(a,ls))
(18)
However, solving this non-linear equation requires iteration the learned neural networks, which is expensive. In practice, one can directly sample using the direct combination positive and negative score functions, as approximation when is relatively small (Ho & Salimans, 2022):
e (a(b) R (1 wee (als) We, (als)
(19)
C ALGORITHM PSEUDOCODE
Algorithm Training
while not converged do
Collect data,
offline data D.
~ (0 I), ~ U[
0, diffusion/flow forward process
(3G) ||e +t (ae, 8, 8, 11 0g 02 AV
01 01 XVe;
(1- o (BG)) -|e- c02 (ac, s, (oc, 11]
end while
Algorithm Sampling
a1 ~N t
for m E [l...., NJ. do
é W) (as, s,t) - we (ac, 8,
- t- (n/N)
0: < diffuusioon/fo reverse process, given
end for return do
D DETAILS ON RL BENCHMARKS
D.1 EXPERIMENTAL DETAILS
In this section, we provide the experimental details, including benchmarks, datasets, and tasks. Our experiments span two primary benchmarks: ExORL (Yarats et al., 2022) and OGBench (Park et al., 2025a)
EXORL. EXORL consists of datasets collected by multiple unsupervised RL agents (Laskin et al., 2021) on the DeepMind Control Suite (Tassa et al., 2018). We utilize datasets collected by unsuper- vised RL algorithms RND (Burda et al., 2019) across four domains (Walker, Jaco, Quadruped, and Cheetah). For each environment, we use the full dataset with all transitions from each dataset.
Walker (locomotion): A bipedal robot with 24-dimensional states (joint positiions/velocites)) and 6-dimensional actions. Test tasks include run, stand, and walk. Rewards combine dense objec- tives: maintaining torso height (stand) and achieving target velocities (Run/Walk).
Quadruped (locomotion): A four-legged robot with 78 78-dimensional states and 12-dimensional actions. Tasks include run and walk, with rewards for torso stability and velocity tracking.
Jaco rgoalnneahhing): A 6-DoF robotic arm with -5-dimensional states and 6-dimensional ac- tions. Tasks involve reaching four target positions (Top Left/Right) using sparse rewards based on proximity to goals.
Cheetah (locomotion): A rumning planar biped with 17-dimensional states consisting of positions and velocities robot joints, and 6-dimensional actions. The reward is linearly proportional to the forward velocity. We consider tasks run and backward for evaluation.
16

---

Published as a conference paper at ICLR 2026
(a) Walker
(b) Cheetah
(c) Quadruped
(d) Jaco
Figure 3: EXORL. environments. We experiment on high-dimensional complex domains: Walker, Cheetah,
Quadruped, and Jaco Arm.
(a) antsoccer-arena (b) humanoidmaze
(c) scene
(d) cube-single
(c) cube-double
Figure 4: OGBench environments. experiment complex domains: antsoccer- humanoidmaze,
scene, single, and cube-double.
OGBench. OGBench is designed for offline goal-conditioned RL, containing multiple challenging tasks across robotic manipulation, navigation, and locomotion. We use 30 state-based manipula- tion and navigation tasks from 6 domains huumaniddaaz--medee navigate, cube-single-playy cube-double-plat scene-play, and antsoccer-arens. naaviiigatenn--aiaatt) Each do- main contains different tasks, and is the default task. To be compatible with standard offline RI. settings, we leverage its single-task variant. We evaluate offline performance on all single tasks and online fine-tuning performance on the default tasks of the selected 4 challenging domains.
navigate, humanoidmaze-larte
humanoidmaze Slavigation):.Cnntrolli 21-DoF Humanoid agent reach goal position in given maze.
cube-play (manipulation):
Controlling a robot arm to pick and place cube-shaped blocks order assemble designated target configurations.
scene-play (manipulation): Long-horizon control of multiple objects, including cube block, window, drawer, and two button locks.
antsoccer-anee (navigation): Controlling an Ant agent to dribble soccer ball. The agent must also carefully control the ball while navigating the environment.
D.2 IMPLEMENTATION DETAILS
We implement DIPOLE JAX on top of FQL (Park et al., 2025b) and CFGRL (Frans al., 2025).
Architectures. We train 5 neural networks in parallel: two policy networks (positive policy and negative valicy)value networks (two Q-value estimators and one V-value estimator). We use three- layer multi-layer perceptron (MLP) with 512 hidden dimensions for both the policy networks and the value networks. We select the flow policy for the evaluation of our approach due to efficient training process. Our flow policy is based linear paths and uniform time sampling.
Value Learning. all experiments, we learn an optimal V-value by IQL-style expectile regression. For Q-value learning method, we compute the target by optimized V-value in ExORL, and the the traditional temporal difference (TD) target Q in OGBench separately. Full details of hyperparameter settings are provided in Table 5.
Policy extraction and action reweighting. In RL setting, one the critical selections G(s.a) in Eqs. (7) is the advantageous function, i.e. A(s, (s,a) a) (s a) V(s). To induce more flexible
17

---

Published as a conference paper at ICLR 2026
and controllable learning process, additionally introduce tunable hyperparameter to shift the distribution G(s, a). Specifically, the weighting function becomes ( a) + k) for positive policy and - (BBG (s,a) k) for negative policy. The full details of per-task hyperparameters provided in Table 6.
We implement rejection sampling for inference time policy output. Specifically, sample N ac- tions for a single state input and select the action that has the highest Q-value:
a' A arg
max (a pou p.... (+))))))))) Qs,
(20)
Evaluation. We report the average return on ExORL and the success rate on OGBench, following the standard evaluation assessment methods (Yarats al., 2022; Park al., 2025a). For offline RL performance evaluation, we fix the gradient to be 1M and report the final score. For the offline-to- online RL performance evaluation, we report both the 1M offline score and the 1M online score.
Computation resource. We train our model on NVIDIA A6000 GPUs. Training single task on one GPU takes approximately 0.5 hours on ExORL and 1.5 hours on OGBench.
D.3 HYPERPARAMETERS
In this section, we provide the detailed hyperparameter setup in Table 5 and Table 6, In our experi- ments, the model architecture and basic algorithm hyperparameters remain unchanged, as detailed in Table 5. To encourage better trade-off between greediness and stability, we adopt Nomain-specint hyperparameters, including expectile parameters т, beta B, shift factor k and discount factor 7 as detailed in Table 6.
Table General hyperparameters used for DIPOLE
Hyperparameter
Value
Optimizer
Adam
Policy learning rate
3e-4
Value learning rate
3e-4
Offline learning steps
1,000,000
DIPOLE Hyperparameters
Online fintuning steps
(offline).
1,000,000 loffiie-ttoonlingg
Mini-batch
512 (ExORL), 256 (OGBench) 0.005
Soft update factor X Diffusion/Flow steps Clip Q
32 (ExORL), (OGBench) false (cube-single-plee scene-play), true (others)
Policy MLP hidden dimension Value MLP hidden dimension Activation function
[512, 512, 512] [512, 512, 512] tanh
Architecture
D.4 ABLATION STUDY
Hyperparameter. Both hyperparameters beta B, shift factor k,, expectile factor 7, and rejection sampling action number N are important for DIPOLE's performance. In Figure 5, we present the performance changes when fixing single action sampling, and present performance changes for the default action sampling number when tuning the expectile factor. We further ablate the influence of beta 33 and CFG scale w on DIPOLE. conducted experiments on ExORL benchmark, average over 3 random seeds. Both the impact of 8 and are within similar pattern, which shows casy-tuning property of DIPOLE.
D.5 ADDITIONAL RESULTS
OGBench full results. In this section, provide the full experimental results for all single tasks on the OGBench, as shown in Figure 7. All results are averaged over 8 random seeds. We report the mean and standard deviation of the final score, after 1M gradient steps.
18

---

Published as a conference paper at ICLR 2026
Table 6: Task-specific hyperparameters for DIPOLE.
Task Category
betu 3
shift factor
discount y
expectile r
sample actions N
OGBench--uue henennnn-uummniiddnzzeeeee-e medium- navigate
0.9
siifench-hummnnidmaaa large-navigate
0.995
0.9
UGBench-antsoeee urena-navigute
1
0
0.995
0.9
GeBeenchcube-single-Pee
0.5
0.99
0.9
0GBench--chhe douhlee-py
0.5
0.5
0,99
0.9
OGiBench-scene-peee
ExORL-walket-walk
3.5
-2
0.99
0.95
32
ExORL-waller-stand
4.5
-2
0.99
0.99
32
ExORL-walker-run
4.5
0,99
0.9
ExORL-quadruped
ExORL-quadriped-ru
0.99
0.9
MxORL-jaco--ccac-top-eee
0.99
0.9
32
ExORLL-saO-Raachcop----ig
1
0.99
0.9
32
ExORRl-cheethh-run
0.99
0.9
LUORL-quadruped-nui -onedd---u--ueee run-backward
0.99
walker walk
cheetah run
walker walk
cheetah run
1000
300
$80.0
1990
210.2
Performance
Performance
900
$56.0
250
0.8
2 beta
3.5
3 beta
5
Expectile
Expectile
Cheetah-Run
Cheetah-Run backward
209
217
203
264
253
106
121
158
144
114
2
98
101
128
128
115
1
78
101
123
119
95
1
72
87
106
102
105
1.5
10
1.5
5
10
W
W
Figure 5: Top-left: ablation on 9 and k; Top-right: ablation on expectile т; Bottom: ablation on w and B
OGBench offline-tooonline learning curves. We also present the training curves of DIPOLE, in-
cluding both the 1M offline oradient and IM online oradient stens, shown in Fioure
Humanoidmaze-mediuu
ad: Antsoccer-arena
Ilit
Cube-double
ing
Scene-play
Perforaacce
iev
Performance
aftt
Performance
OC
Rerfommncc
Step
146
Stop
140
Stop
1e5
Figure 6: Offline-to-online visualization. DIPOLE presents stable fine-tuning process OGBench.
19

---

Published as a conference paper at ICLR 2026
Table 7: OGBench full results.
Gaussian Policy
Ditfusion/Flow Policy
Task Category
IQL.
ReBRAC
IQQI
IFQI.
FQI.
DIPOLE
32:17
16:19
111
69:119
19::12
63:16
humanoidmaze-
41.99
18.116
111
85.11
94.13
91
humanoidmaze-mie
25
36:1:3
0::1
49:1:4
74
88:14
0=11
15
1::1
1:11
3:144
1:11
humamoidmmze--me wwtp:///weeeiiunnnniiiieeeeeee sewediuunnniiigieessiiileaaaaaae
66.14
24
L11
98
97.12
96
humancidmaze htw.r/ddmwwwaaaaggaaaaaa----iiie sage--aaiiiaac-ssiggllaaattttt-
2:11
0::0
6:h2
7
20:
humancidmaze- haatmaaae------eeaaae Sunge-naviiaee-singleuus--aaa
0=:0
0.1:0
0.::
0.10
0.:0
0.10
bumanoidmaze waareee--rrnnaaaeee large-navigate-sinng maaiigate-singlletssstaaaae
7=33
8:h4
3+:1
48-=10
11:77
7
humnaoidmaae-- harreaaraee--e------e---- large-naviig aagge--aaiaattte wiiiilatesingletus---ae
1.10
1.1
O.LO
1
2
111
humanoidmaze wwwaaaazz-----aaaaae large-naviig www.angga--iingllaas--------
1=1
2#:2
0#:0
0::0
1
2
untsoccer-aren navigate- singatt-iiggetss-ttsss..
14::5
0.10
44
61::25
77:14
82.17
anlsoccer arena priigte--singletae
17:17
0:11
15:12
75:15
88:13
74:15
antsocecr-ane wagggattinglttts--naaaaee
6=#4
0=:0
0::0
14::22
61#b6
55+88
anisoccer-archn aeeeeccer-rena-eee navinaaaaaaaaaae gate--inggletaa letas task4
3=:2
0::0
0::1
16:
39:26
40:100
untsoccer-arena naviiaaa--iinnggneeeaaee sagaatt-iiinnttss-asssss.
2
0:::
0::0
0::1
36:199
32:15
sube--ingge-phhe pla--singleeaskk lask
88:13
89
95:12
79
97:12
97:12
85
92-14
96:22
73-
97
98+2
sube-single-pla-- -ingg-pp-----nllttss-------
91:15
99.11
98.12
99.12
cube-single-phe play--ssngllltakk
73:16
93:1:4
79:16
94:E3
94:15
cube-single pla--paa----iinetasskk -usk5
78.19
87
90.16
77.17
93.13
96 L3
cube-double- wawuubblepla-snneeeeee play-singlerask-tasse
27:
45 65-
39:=19
35-19
61:t9
68
cube-double-play play-singletusk-tuue
1::1
7:13
16::10
9
36:1-
14
cube-double-p play--inglettaeeee task3
0::0
4:11
17:88
8
22:55
51:16
saabb--ouble-pla--eee play-singletask-taaee
0-:0
1+1
0::1
1E11
5#22
6#=2
cube-double sawuubbe-plly-iieee blay-singletask-lnn
4.13
4.22
L.1
17.16
19.110
50.18
scene-play play-singlletsseeeeee..aa
94
95-12
100
98:13
100
100
scene-play singl-siglet
12:13
50
33
0:10
76:19
96:13
sccnc-play-singee
play-singlctaskktaskk
32#77
16
94:14
54::19
98+11
99
sene--lay-singleeaase
011
313
413
0.10
$14
$.16
scene-play pingl-inglettsk--ee -taskk
0=:0
0::0
0::00
0:::
0::0
0::1
E DETAILS ON E2E AD BENCHMARKS
E. MODEL ARCHITECTURE
We employ the pretrained Florence-2-large model (Xiao et al., 2024) as the visual-language en- coder, paired with a -75M-parameter Diffusion Transformer as the action decoder. The visual input comprises images from Front, Front-Left, and Front-Right perspectives, while the language input consists of driving commands provided by the dataset. Encoder output tokens are processed by the action decoder through cross- cross-attention block, which ultimately generates the predicted trajectory.
E.2 TRAINING PROCEDURE
The training process consists of two phases: pretraining and reinforcement learning fine-tuning. In the pretraining phase, we utilize trainval frames from the NAVSIM dataset to jointly train the encoder and decoder using diffusion loss objective. During the RI. fine-tuning phase, the encoder frozen, and two Low-Rank Adaptation (LoRA) adapters-a positive adapter and a negative adapter- are incorporated into every linear projection of the attention and MLPs. The NavSim benchmark's PDMS score serves as the direct optimization target. Every 10 epochs, the replay buffer is cleared, and new model rollout trajectories and corresponding rewards are collected based on one epoch of data samples. For each data sample, the model generates g trajectories, which used train the LoRA adapters over the subsequent 9 epochs. For each trajectory, we obtain its PDMS score vector enabling estimation of the advantage function (Shao al., 2024):
(r1,72,
G(s,a) = A(s,a) Tj - (-mean())
(21)
std(r)
20

---

Published as a conference paper at ICLR 2026
E.3 INFERENCE AND EVALUATION
During inference, the DP-VLA encoder extracts features from the input images and driving com- mands. The denoising process is solved using the DPM-Solver(Lu al., 2022), with the action decoder iteratively predicting the denoised trajectory over 10 steps to produce the final clean trajec- tory. For evaluation on the NAVSIM benchmark, the proposed trajectory is fed into an LQR tracker and dynamics model to compute the posterior trajectory. The final PDM score is derived from this posterior trajectory, satisfying the benchmark's evaluation criteria (Dauner et al., 2024).
E,4 HYPERPARAMETERS
We summarize the training details of DP-VLA in Table 8. LoRA adapters are applied to all linear projections, including the QKVO projections Attentions and Linears in FFNs.
Table 8: DIPOLE hyperparameters used in DP- DL-
Hyperparameter
Value
Optimizer
AdamW
Learning rate Learning epochs
le-4
Pre-train Hyperparameters
100
Mini-batch
16
Optimizer
A
Learning rate Learning steps
le-4
DIPOLE navtrain Hyperparameters
2.069k
Mini-batch
56
Group size g
10
Optimizer
Adam W
Learning rate
le-4
DIPOLE navtest Hyperparameters
Learning steps
11.52k
Mini-batch
4
Group size
25
Rank
16
Alpha Dropout
16
LoRA Hyperparameters
0.0
Num. params(Each/Touas
6.68M/13.37M
Ratio params(Each/Total)
1.4%/2.8%
E.5 MORE CASES
We selected representative scenarios from nayTest and superimposed the trajectories of DP-VLA before and after DIPOLE fine-tuning visualize the behavioral differences in Figure 7. It can be observed that while the pre-fine-tuned model exhibits off-road driving or collisions, these rule- violating and dangerous behaviors are effectively rectified after fine-tuning.
F LIMITATION & DISCUSSION & FUTURE WORK
Here, discuss the limitations, potential solutions, and promising future directions of our work. While this paper presents policy optimization method for achieving both greedy and stable pol- icy training, we observe that performance highly dependent on the quality of the value function. Developing effective value function estimation methods for diffusion- ing remains an important direction for future research. Additionally, our approach remains within the behavior-regularized optimization framework, which inherently constrains the policy relative to a reference policy. We attempt to address this limitation through controllable greediness and greedified optimization objective. While our method has demonstrated applicability in complex autonomous driving tasks, we anticipate its potential extension to other domains.
reinforcement learn-
21

---

Published as a conference paper at ICLR 2026
Figure 7: NAVSIM Results: DP-VLA w/ DIPOLE Line-tuned model trajectory: ground truth ego trajectory; DP- imitation pretrained model trajectory.
22


Reviews:
[{"review_id": "YulTbNOXwa", "reviewer": "Reviewer_jJbf", "summary": "The paper proposes DIPOLE, a novel reinforcement learning (RL) framework for optimizing diffusion-based policies. The key idea is to reformulate the KL-regularized RL objective into a “greedified” version that can be decomposed into two dichotomous diffusion policies — one maximizing reward (positive policy) and one minimizing reward (negative policy). During inference, their score functions are linearly combined, enabling controllable trade-offs between greediness and stability.\nExtensive experiments on ExORL, OGBench, and a large-scale autonomous driving benchmark (NAVSIM) demonstrate performance gains over prior RL and diffusion-based baselines.", "questions": "The proposed method requires training two separate policy net\u0002works $\\epsilon^+, \\epsilon^-$, which effectively doubles the computational and storage costs. However, the paper does not discuss or evaluate this overhead: How long is the training time compared to single-policy baselines? During inference, although the final score is obtained by a linear combination, two models must be executed—what is the resulting latency? In the autonomous driving experiments, the authors mention using LoRA to mitigate this issue, but they do not quantify the number of additional parameters introduced by LoRA or its impact on training efficiency", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed dichotomous decomposition of the KL-regularized objective is both elegant and conceptually novel. The analogy to classifier-free guidance (CFG) provides a strong intuitive and theoretical bridge between diffusion modeling and RL optimization.\n\n2. The paper presents comprehensive experiments across multiple RL benchmarks and an ambitious large-scale 1B-parameter VLA model for end-to-end driving, showing clear improvements over strong baselines (IQL, FQL, CFGRL, etc.).\n\n3. The paper presents comprehensive experiments across multiple RL benchmarks and an ambitious large-scale 1B-parameter VLA model for end-to-end driving, showing clear improvements over strong baselines (IQL, FQL, CFGRL, etc.).", "weaknesses": "1. The reviewer is a little bit confused about why we need to train a policy that minimizes the rewards. In my opinion, to avoid the large difference between the optimized policy and the behavior policy of offline data, we can directly perform imitation learning on the second diffusion policy rather than minimizing the reward. \n2. How can we get $G(s, a)$ in the proposed method? Should we apply some special technique to learn it, such as CQL [R2]?\n3. The method can be classified as a weighted-based diffusion RL method and lacks the citation of the recent weighted-based diffusion RL method [R1].\n\n[R1] Ding S, Hu K, Zhang Z, et al. Diffusion-based reinforcement learning via q-weighted variational policy optimization[J]. Advances in Neural Information Processing Systems, 2024, 37: 53945-53968.\n\n[R2] Kumar A, Zhou A, Tucker G, et al. Conservative q-learning for offline reinforcement learning[J]. Advances in neural information processing systems, 2020, 33: 1179-1191.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "064gru5FCj", "reviewer": "Reviewer_89ky", "summary": "The paper introduces DIPOLE, a new framework for training diffusion-based policies in goal-conditioned offline reinforcement learning. Instead of using a single diffusion policy with unstable exponential advantage weighting, the authors propose a dichotomous formulation: two separate diffusion policies are trained — one favoring high-return behaviors ($\\pi^+$) and one suppressing low-return behaviors ($\\pi^-$). The final policy is synthesized at inference by combining their score functions. This approach avoids instability caused by unbounded weighting and enables a controllable trade-off between greediness and safety. Empirically, DIPOLE achieves strong performance across several offline RL tasks (ExORL, OGBench), improves stability in training, and scales to large vision-language-action models in autonomous driving (NAVSIM benchmark), outperforming prior imitation-based diffusion policies.", "questions": "Can this algorithm generalize to the online setting? There is some recent work on using KL-regularized RL and mirror descent to define the diffusion policy loss function in the online setting (see below reference), which has the same form as Equation 4. I think the same dichotomous idea may also work there. Could you please explain if this will work or not?\n\n\"Efficient Online Reinforcement Learning for Diffusion Policy\", Haitong Ma, Tianyi Chen, Kai Wang, Na Li, Bo Dai, ICML 2025", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The paper proposes a simple but effective method (DIPOLE) that trains two diffusion policies instead of one, helping stabilize learning in offline RL.\n* The method is well-motivated and theoretically justified, avoiding unstable exponential weighting by using bounded scores.\n* Strong experimental results across many tasks, including large-scale vision-language-action models for autonomous driving.\n* The paper is clearly written, well-organized, and easy to follow.", "weaknesses": "* The method is only evaluated in offline or offline-to-online settings. I am not sure why the same idea can't be applied to online RL?\n* The baselines for comparison seem random to me. Not sure what are the reasons to choose those baselines as opposed to some other diffusion-based / non-diffusion-based offline RL baselines. For example, there are plenty of model-based offline RL baselines and I think the authors primarily only choose model-free baselines. Is this intentional? What are the rationals behind choosing these baselines? I have read Sec 4.1 but I am not fully convinced by the explanation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "khWYEXodlw", "reviewer": "Reviewer_noDV", "summary": "This paper introduces DIPOLE (Dichotomous Diffusion Policy Improvement), a stable, scalable, and controllable reinforcement learning framework for training large diffusion policies. Existing RL approaches for diffusion policies either (i) directly optimize value or reward objectives, leading to high gradient variance and training instability, or (ii) approximate Gaussian likelihoods across multiple denoising steps, which are computationally expensive and often inaccurate. By revisiting the KL-regularized RL formulation, DIPOLE proposes a greedified KL-regularized objective that naturally decomposes into two dichotomous sub-policies: the positive policy that favors high-reward actions and the negative policy that models low-reward actions.", "questions": "1.\tFor DP-VLA, the reward shaping, return computation, and LoRA-based adaptation are briefly described but not rigorously analyzed.\n2.\tThe paper uses both temperature $\\beta$ and greediness $\\omega$, how do they jointly affect stability and optimality?\n3.\tThe linear combination resembles CFG, but is $\\omega$ chosen adaptively per state or fixed globally? How does this choice affect exploration–exploitation balance?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This work derives a new closed-form optimal policy under a modified KL objective with a bounded sigmoid weighting, effectively avoiding unstable exponential terms and preventing gradient explosions. The dual-policy decomposition enables learning from both high- and low-reward samples, mitigating data imbalance and overfitting to rare high-reward trajectories. Empirical results demonstrate that DIPOLE consistently outperforms strong baselines across offline, offline-to-online, and large-scale VLA tasks. Moreover, the training procedure remains simple and modular, as it only modifies the loss weights of standard diffusion objectives, maintaining full compatibility with existing architectures.", "weaknesses": "1.\tIn DIPOLE, the weight term $\\sigma(\\beta G(s,a))$ (or $1 - \\sigma(\\beta G(s,a))$) is treated as constant with respect to the diffusion model parameters. This means that the diffusion model learns to denoise under static weighting but does not explicitly learn how to adjust the action distribution to improve $G(s,a)$ directly. Consequently, there is no gradient signal guiding the modification of intermediate noisy actions to increase the expected reward, which may lead to slower convergence or plateaued performance when the current policy’s support does not already include near-optimal actions. As a result, the performance of DIPOLE heavily depends on the quality of the value estimator. If $G(s,a)$ overestimates certain actions, the weighting will amplify these errors, making policy improvement rely entirely on value accuracy rather than direct reward gradients. This issue is particularly pronounced in offline RL, where value estimates can be severely biased in out-of-distribution (OOD) regions. Thus, the diffusion process learns primarily through denoising consistency instead of reward shaping across time. Furthermore, since updates are based on weighted regression rather than policy gradient optimization, there is no stochastic gradient noise or entropy regularization to encourage exploration. In online fine-tuning scenarios (e.g., DIPOLE’s autonomous driving setup), this lack of exploratory signal could slow adaptation to unseen environments.\n2.\tThe paper derives the optimal policy formulation but does not provide rigorous convergence guarantees or theoretical error bounds for the dichotomous approximation.\n\n3.\tAlthough the parameter $\\omega$ controls the degree of greediness, the paper lacks quantitative analysis on how $\\omega$ and $\\beta$ jointly influence training stability and performance.\n\n4.\tTraining two separate diffusion models likely doubles computational and memory costs, yet the paper does not report comparisons on training time or efficiency.\n\n5.\tThe method assumes a reliable reward function or Q-estimator $G(s,a)$, but it remains unclear how performance degrades when the value estimates are noisy or biased.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RSm0tKgt5N", "reviewer": "Reviewer_ZjGt", "summary": "This paper proposes DIPOLE (Dichotomous diffusion Policy improvement), a novel reinforcement learning algorithm for training diffusion-based policies. The authors identify a key challenge in prior work: directly optimizing the standard KL-regularized RL objective,\n$$\\max_{\\pi} \\mathbb{E}[G(s,a)] - \\frac{1}{\\beta} D_{KL}(\\pi||\\mu)$$\nis difficult because its closed-form solution, $\\pi^*(a|s) \\propto \\mu(a|s) \\cdot \\exp(\\beta G(s,a))$, leads to an unstable weighted regression loss where the $\\exp(\\cdot)$ term can explode.\nTo overcome this, DIPOLE proposes a new \"greedified\" KL-regularized objective (Eq. 5). The key insight is that the optimal solution to this new objective can be decomposed into a ratio of two \"dichotomous\" policies:\n1. A \"positive\" policy $\\pi^+ \\propto \\mu(a|s) \\cdot \\sigma(\\beta G(s,a))$\n2. A \"negative\" policy $\\pi^- \\propto \\mu(a|s) \\cdot (1 - \\sigma(\\beta G(s,a)))$\n\nCritically, the authors claim these policies can be trained stably using diffusion models with bounded sigmoid weights ($\\sigma$ and $1-\\sigma$), solving the stability-optimality trade-off.\nFurthermore, the paper shows that sampling from the optimal policy $\\pi^*$ can be achieved by linearly combining the scores of the two dichotomous policies, resulting in an inference rule analogous to Classifier-Free Guidance (CFG):\n$$\\tilde{\\epsilon}(a_t, s, t) = (1 + \\omega) \\epsilon_{\\theta_1}^{+}(a_t, s, t) - \\omega \\epsilon_{\\theta_2}^{-}(a_t, s, t)$$\nThe authors demonstrate DIPOLE's effectiveness on ExORL and OGBench benchmarks and scale it successfully to a 1-billion parameter vision-language-action (VLA) model for autonomous driving.", "questions": "1. On the Premise (re: BDPO): The paper's motivation rests on the instability of the $\\exp(\\beta G)$ weight (Eq. 4). However, recent work (e.g., BDPO) shows that the standard KL-reg objective (Eq. 2) can be optimized stably via a pathwise KL decomposition, resulting in an analytic MSE penalty. Given this, what is the theoretical advantage of proposing a new \"greedified\" objective (Eq. 5)?\n2. On the Training Mechanism (re: Saturation): The core training relies on $\\sigma(\\beta G)$ (Eq. 9). This weight saturates for high $G(s,a)$ values. How can the network $\\epsilon^+$ learn to distinguish between a \"good\" action ($G=10$) and an \"excellent\" action ($G=20$) if the training signal (the weight) is nearly identical for both?\n3. On the Inference Mechanism (re: $\\omega$): Following Q2, if $\\epsilon^+$ has lost the high-reward gradient information due to saturation, how can the inference factor $\\omega$ (Eq. 10) recover this information? Is it not simply amplifying a \"clipped\" or \"binary\" signal, rather than steering the policy towards the truly optimal (e.g., highest $G(s,a)$) actions? Could the authors comment on what $\\epsilon^+$ is actually learning?\n4. On the shift factor $k$: Regarding the 'shift factor' $k$ introduced in Appendix D.2 (using $\\sigma(\\beta G + k)$): This parameter is absent from the theoretical derivation. Could the authors confirm that this is necessary to counteract the sigmoid saturation and center the function's active region over the data's value distribution? How sensitive is the algorithm's performance to the choice of $k$, and doesn't its necessity undermine the robustness and principled nature of the proposed theoretical framework?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. **Originality**: The primary strength is the novel formulation. The idea of decomposing the optimization into a reward-maximizing ($\\pi^+$) and a reward-minimizing ($\\pi^-$) policy is creative and provides a new lens for policy optimization.\n2. **Strong Empirical Results & Scalability**: The method clearly works well in practice. The successful application of DIPOLE (using LoRA) to a 1-billion parameter VLA model is a significant achievement and demonstrates the method's practical utility for fine-tuning large models with RL.", "weaknesses": "1. **Questionable Necessity (Unsound Premise)**: The paper's motivation is that the standard KL-regularized objective (Eq. 2) is unusable because its solution (Eq. 3) implies an unstable weighted regression (Eq. 4). This implicitly assumes that weighted regression is the only way to optimize this objective. This premise is challenged by recent work like BDPO (Gao et al., 2025, ICML), which tackles the exact same standard objective. BDPO shows that by decomposing the $D_{KL}$ term along the diffusion path ($D_{KL}[p_{0:N}^\\pi || p_{0:N}^\\nu]$), the objective becomes a sum of per-step, analytic KL divergences $\\sum_n D_{KL}[p_{n-1|n}^\\pi || p_{n-1|n}^\\nu]$. Since these per-step transitions are Gaussian, this penalty becomes a simple, stable, analytic MSE between the predicted noise vectors (Eq. 17 in BDPO). This suggests that DIPOLE's \"greedified\" objective (Eq. 5) is an overly complex solution to a problem that has a simpler, more direct solution within the original, standard RL framework.\n2. **Flawed Training Mechanism (Sigmoid Saturation)**: The core of DIPOLE's solution is the replacement of $\\exp(\\beta G)$ with the bounded $\\sigma(\\beta G)$ (Eq. 9). This introduces a new, critical problem: signal saturation. The sigmoid function saturates, meaning its output approaches 1 for all values above a certain threshold (e.g., $\\sigma(10) \\approx \\sigma(20) \\approx 1.0$). This means the training loss for the positive policy $\\epsilon^+$ loses all gradient information that distinguishes \"good\" actions from \"excellent\" actions. The network is not learning a fine-grained reward landscape, but rather a near-binary classification of \"good\" (weight $\\approx 1$) vs. \"bad\" (weight $\\approx 0$).\nThis saturation flaw directly contradicts the goal of the inference step (Eq. 10). The inference mechanism $\\tilde{\\epsilon} = (1+\\omega)\\epsilon^+ - \\omega\\epsilon^-$ relies on the \"greediness factor\" $\\omega$ to amplify the difference between the two policies. However, if $\\epsilon^+$ has already lost the high-reward gradient information due to saturation, $\\omega$ is merely amplifying a \"blurry\" or \"clipped\" signal. It's unclear how this can steer the policy towards truly optimal actions if the network was never trained to distinguish them in the first place.\n3. **Gap Between Theory and Practice (The $k$ parameter)**: This saturation flaw is strongly corroborated by the implementation details. The paper's theoretical derivation (Eq. 9) relies purely on $\\sigma(\\beta G)$. However, Appendix D.2 reveals the practical use of a modified weight, $\\sigma(\\beta G + k)$. This 'shift factor' $k$, which is absent from the main theory, serves as strong evidence that the sigmoid-based weighting is not robust. It implicitly confirms the saturation problem, as the model's performance is highly sensitive to the distribution of $G(s,a)$. The mechanism is therefore not as 'principled' as claimed, requiring an ad-hoc hyperparameter to manually shift the sigmoid's non-saturating region to align with the data.\n\n**References**\n\nChen-Xiao Gao, Chenyang Wu, Mingjun Cao, Chenjun Xiao, Yang Yu, Zongzhang Zhang Proceedings of the 42nd International Conference on Machine Learning, PMLR 267:18630-18657, 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)