Beyond Penalization: Diffusion-based Out-of-Distribution Detection and Selective Regularization in Offline Reinforcement Learning
Authors: Qingjun Wang
Hongtu Zhou
Hang Yu
Junqiao Zhao
Yanping Zhao
Chen Ye
Ziqiao Wang
Guang Chen
Venue: ICLR 2026 Poster (2026)
Keywords: Offline RL
Diffusion Model
Out-of-Distribution (OOD) Detection
Primary area: reinforcement learning
TL;DR: We propose DOSER, a diffusion-based framework for OOD detection and selective regularization in offline RL. By leveraging diffusion reconstruction errors to handle OOD actions, DOSER achieves state-of-the-art performance on D4RL benchmarks.

Abstract: Offline reinforcement learning (RL) faces a critical challenge of overestimating the value of out-of-distribution (OOD) actions. 
Existing methods mitigate this issue by penalizing unseen samples, yet they fail to accurately identify OOD actions and may suppress beneficial exploration beyond the behavioral support.
Although several methods have been proposed to differentiate OOD samples with distinct properties, they typically rely on restrictive assumptions about the data distribution and remain limited in discrimination ability.
To address this problem, we propose $\textbf{DOSER}$ ($\textbf{D}$iffusion-based $\textbf{O}$OD Detection and $\textbf{SE}$lective $\textbf{R}$egularization), a novel framework that goes beyond uniform penalization. 
DOSER trains two diffusion models to capture the behavior policy and state distribution, using single-step denoising reconstruction error as a reliable OOD indicator. 
During policy optimization, it further distinguishes between beneficial and detrimental OOD actions by evaluating predicted transitions, selectively suppressing risky actions while encouraging exploration of high-potential ones.
Theoretically, we prove that DOSER is a $\gamma$-contraction and therefore admits a unique fixed point with bounded value estimates. 
We further provide an asymptotic performance guarantee relative to the optimal policy under model approximation and OOD detection errors.
Across extensive offline RL benchmarks, DOSER consistently attains superior performance to prior methods, especially on suboptimal datasets.

Published as a conference paper at ICLR 2026
BEYOND PENALIZATION: DIFFUSION-BASED OUT-OF- DISTRIBUTION DETECTION AND SELECTIVE REGU- LARIZATION IN OFFLINE REINFORCEMENT LEARNING
Qingjun Wang', Hongtu Zhou', Hang Yu', Yu', Junqiao Zh
Yanping Zhao', Chen Ye', Ziqiao Wang', Guang Chen
School of Computer Science and Technology. Tongji University, Shanghai, China
"MOE Key Lab of Embedded System and Service Computing, Tongji University, Shanghai, China Shanghai Innovation Institute
{2432069, zhouhongtu, 2432034, 2 edu. {2534018, yechen, ziqiaowang, quangchen edu. en
ABSTRACT
Offline reinforcement learning (RL) faces a critical challenge of overestimating the value of outto--diitribution (OOD) actions. Existing methods mitigate this is- sue by penalizing unseen samples, yet they fail to accurately identify OOD actions and may suppress beneficial exploration beyond the behavioral support. Although several methods have been proposed to differentiate OOD samples with distinct properties, they typically rely restrictive assumptions about the data distribu- tion and remain limited in discrimination ability. To address this problem, we pro- pose DOSER (Diffusion-based OOD Detection and SElective Regularization), a novel framework that goes beyond uniform penalization. DOSER trains two dif- fusion models capture the behavior policy and state distribution, using single- step denoising reconstruction error as reliable OOD indicator. During policy optimization, further distinguishes between beneficial and detrimental OOD actions by evaluating predicted transitions, selectively suppressing risky actions while encouraging exploration of high-potential ones. Theoretically, we prove that DOSER is 7-contraction and therefore admits unique fixed point with bounded value estimates. We further provide an asymptotic performance guaran- tee relative the optimal policy under model approximation and OOD detection errors. Across extensive offline RL benchmarks, DOSER consistently attains su- perior performance to prior methods, especially on suboptimal datasets.
INTRODUCTION
Offline reinforcement learning (RL) has emerged as powerful paradigm for learning policies ex- clusively from static datasets, eliminating the need for potentially costly or risky online interac- tions (Levine et al. 2020). This capability renders it particularly appealing for real-world domains where exploration is constrained, such as robotics, healthcare and autonomous systems. However, directly applying standard off- aff-policy RL algorithms to offline dataset pose fundamental challenge of distribution shift. When the learned policy generates actions that deviate substantially from the training data distribution, value functions tend extrapolate erroneously. leading to severe value overestimation and ultimately catastrophic performance degradation (Fujimoto et al. 2019).
Existing approaches fall into two categories: 1) Policy constraint methods enforce the learned policy remain close to the behavior policy to avoid OOD actions (Kumar et al., 2019 Wu et al., 2019 Fujimoto & Gu 202T Kostrikov al. (02T), typically relying variational auto-encoders (VAEs) (Kingma & Welling 2013) for behavior modeling. While effective in principle, these meth- ods struggle capture the multi-modal nature real-world behaviors, often collapsing diverse tion distributions into suboptimal averaged outputs within low-density regions (Wang et al.) 2022). 2) Value regularization methods offer an alternative by learning conservative Q-functions that pe-
Corrreponniigg Author

---

Published as a conference paper at ICLR 2026
VAE-based Detsction and Penalization
Diffusion model captures multi-modal behavior policy
OOD Action Detection Action
Penalize high-potential OOD actions
Actions
Adding notse
Small Distance
Incorrectly identify 000 samples as
Deploy
Denobeing
Action 2
Actions
Predicted transiioos
Denoising Network
Large Distance
OOD Action Classification
So
High-value Region
ID Region
S'
OOD Region
VAE-basee
8'
Behavior Modaling
Benus!
Dynamica Model
High-value Region
Parelty!
Dynamica Model
S' Low-ralue Region
Figure I: VAF-based behavior modeling methods (left) misidentify OOD actions, whereas uniform penalties suppress high-potential OOD actions. DOSER (right) models multi-modal behavior pol- icy via diffusion model and uses reconstruction error as an OOD indicator, further distinguishing detrimental from beneficial actions for selective regularization.
nalize OOD actions (Kumar et al. 2020 Wu et al. 2021 Bai et al. 2022 Mao et al. 2023). Their effectiveness depends on the underlying OOD identification mechanism, which is challenging task due to the limited representation capacity of the models used to characterize data distribution. Furthermore, they usually apply uniform penalties across the entire out-of-support region, without considering valuable explorations that could enhance policy performance (Figure [ left).
Recent efforts have sought to mitigate excessive pessimism by controlling the level of conservatism in fine-grained -gained manner. CCVL (Hong et al., (002) conditions the Q-function on a confidence level to learn spectrum of conservative value estimates, enabling adaptive policies that dynamically adjust pessimism during online evaluation. ACL-QL (Wu et al. 2024) models the behavior pol- icy as Gaussian distribution and introduces learnable weighting functions to adaptively modulate conservatism at the late-action level. DoRL-VC (Huang et al. (2024) employs VAE-based detec- tor to separate OOD from ID actions, and further distinguish OOD actions with different properties. Nevertheless, such approaches either rely Q-ensemble learning achieve varying degrees of con- servatism, incurring additional training overhead, or inherits strong Gaussian assumptions regarding the behavior policy, which fundamentally limit their ability reliably identify OOD samples.
To address these challenges, we present DOSER (Diffusion-based OOD Detection and SElective Regularization), advancing OOD handling through two key innovations (Figure [ right). First, we utilize diffusion models to achieve OOD detection. By deploying two diffusion models for behavior policy approximation and state distribution modeling. we establish reconstruction errors as theoreti- cally rigorous metrics, avoiding strong Gaussian assumptions while maintaining well-calibrated de- tection performance. Second, we introduce an adaptive discrimination mechanism that goes beyond binary classification of in-distribution (ID) and OOD. By integrating learned dynamics model, we distinguish between beneficial OOD actions (those with potential to improve performance while staying within state distribution) and detrimental OOD actions (those likely to induce state distri- bution shift or value degradation). This fine-graing grained discrimination enables selective regularization, discouraging hazardous actions while encouraging promising explorations, which yields robust framework that maintains necessary conservatism while facilitating policy improvement.
The key contributions of this paper OOD detection in offline RL, using reconstruction error as a theoretically grounded metric. 2) We introduce a dual regularization strategy that adaptively adjusts its treatment of OOD actions based on predicted outcomes, suppressing detrimental actions while encouraging beneficial ones. 3) Extensive experiments on D4RL benchmarks demonstrate superior or competitive performance compared to prior methods, with detailed ablations verifying the effectiveness of each component.
follows: 1)
We propose diffusion-bart based approach for
2 PRELIMINARY
Offline RL. We consider the RL problem formulated by the Markov Decision Process (MDP), which is defined tuple (S,A, P, R, 7, do), with state space S, action space A, transition dynamics
2

---

Published as a conference paper at ICLR 2026
A 10, 1], reward function R
A -
Rmin: Rmax] discount factor Y [0. 1),
1998). The goal of RL is to learn a policy S ((A) that maximizes the expected discounted return (x) = -I R(s,, a,) ]- For any policy we define the value function as (T)
and initial state distribution dy : S - [0. 1] (Sutton et al.
B 7
s0) s). and the Q- R(, 7
|r)]ss Sy = 8. = a]. Given that rewards are bounded, the Q-function must lie between Qiin = Rmin/ (1 7) and Qmax Rmmax - 7). In offline RL, the agent is limited to learn from static dataset D {s,, a,, r, s') collected by behavior policy Te,
function as (s) EI y R(s,, a.)
without any interaction with the environment (Lange et al. (012). We denote the empirical behavior
policy ng. which depicts the conditional action distribution observed in D.
Diffusion Models. Diffusion models (Sohl-Dickstein et al., 2015; Ho et al. 2020 Song et al, 2020) have emerged as a powerful class of generative models that excel in capturing complex data distributions. The core idea revolves around a forward diffusion process that gradually perturbs data into noise and reverse process that learns to reconstruct the original data. Given clean sample 2o ~ (at)) with standard deviation data. the forward process constructs a sequence of increasingly noisy samples at; ~ p( σe) by adding i.i.d. Gaussian noise with standard deviation of that increases along the schedule Jwin Co < 01 <... < ON = max Commonly, min chosen sufficiently small that Pmin (z) Paata(x), while max is large enough that the final distribution approximates isotropic Gaussian noise, i.c., Pmax (x) R N(x; 0, oi
In the original DDPM (Ho et al. 2020) formulation, this process is modeled as a discrete Markov chain. Subsequent works reinterpret it through the lens of stochastic differential equations (SDEs) (ong et al. 2020), describing the evolution of σ, over continuous time t [0, as:
dx; f t) dt + g(t dw;
where f(,1t) and g(t) are the drift and diffusion coefficients, and we is a standard Wiener process.
The EDM framework (Karras al. 2022) refines this paradigm by reparameterizing the diffusion path with differentiable noise schedules (t). The reverse process is governed by corresponding probability- ODE derived from the forward SDE, which is formulated as:
dæ; = (t)o
where att) do is the time derivative of noise schedule controlling the noise change rate,
dr
VR, log Pa (2r,) ) is the score function of the marginal distribution pa (xs). The score is approximated
by neural network ea (æx; trained via denoising score matching (Vincent) 2011). The denoising
model Eg is trained to predict the true clean sample æo from its noisy version Dt minimizing the reweighted L2 loss:
2o + OrE by
(()) Ror Px [A(a c ( 1
where A(o,) is the loss weight. Compared to the original DDPM that requires thousands of denoising steps, EDM accelerates sampling by introducing optimized noise schedules and higher-order ODE
solvers, achieving high-quality generation within only few dozen steps.
DIFFUSION-BANE OOD DETECTION AND SELECTIVE REGULARIZATION
In this section, we present the technical framework of DOSER. We begin by introducing three main components that enable precise detection and classification of OOD actions, then demonstrate the complete integration of these components into a unified algorithmic framework, detailing the prac- tical implementation. Figure 2 provides an overview of the proposed method. For comprehensive theoretical analysis. please refer to Appendix A
3.1 DIFFUSIONBBSSED BEHAVIOR AND STATE MODELING
The foundation of our approach is to establish two diffusion models that jointly capture the under- lying distributions of the offline dataset. We first construct a conditional diffusion model that learns the empirical behavior policy distribution (te(a))) ()) by training denoising network Ea, (a,, or, 8) to reconstruct the original action as through the following optimization objective:
L [(0a) (e (o.. [A(at)][an co. (0,, Zt: s)]
(4)
3

---

Published as a conference paper at ICLR 2026
000 Action?
000 State?
Detrimental
Orarget Quix
%
Dv,
Qg)
Pv
Volse VoCifo
Insample reconstruction error
Voloc) Va (sio
Pretrain demoising network
Select the best ID action
| > To
action
Pg
520
Ocorget (1( (p. 6,)
Beneficial 000 action
OOD Action Detection
OOD Classffication
Figure 2:
Overview of the proposed method: (a) Diffusion -bandd OOD action detection, (b) Inte-
grating the detector to achieve OOD action classification.
where ag + E is the noisy action with noise scale os, A(az) balances loss scales across noise levels and (0, I).
In parallel, we develop diffusion model to capture the state distribution do(s) of the dataset. The corresponding denoising network Ea, (st, oe) is trained recover the original states so from its noisy version 8,, using the following reconstruction objective:
((O.) = En,, [X(o,) |0 - Eq, (8t,
(5)
3.2 OOD DETECTION VIA RECONSTRUCTION ERROR
Our detection mechanism leverages the denoising capabilities of pretrained diffusion models to iden- tify OOD samples based on reconstruction errors. Given a state-action pair (s, an) encountered during policy optimization, compute its OOD score through two-step procedure.
First, we sample a noise scale o, from the training noise schedule and perturb the action as a, t,,c where A(0 I). The OOD score is then defined as the L2 reconstruction error between the original action and its denoised counterpart:
Eg (s, ao) |a. co, (a,, or, 1l2
Analogously, for state inputs, measure the reconstruction error between the original state so and its denoised version:
(so) || co, (sc, 02)
where s; denotes the norse-cor state.
Formally, the OOD indicator functions are given by:
Tood (ao) (E, (8, ao) > Ta); To0)(80) (E, > TT),
(8)
where the thresholds T. and T, are as the p-th percentiles of the reconstruction errors the training dataset D, with controlling the level of conservatism.
This reconstruction-bast method offers three key advantages: Reconstruction error provides a likelihood-fii surrogate for distributional alignment, directly measuring conformity to the data manifold without explicit density estimation. 2) Diffusion models naturally capture multi-modal distributions, avoiding the restrictive unimodal Gaussian assumptions of conventional approaches. 3) Detection is efficient, requiring only single forward pass per sample. Moreover, evaluating enrors across multiple randomly sampled diffusion timesteps rather than fixed noise level improves robustness, since different noise scales correspond to varying levels of information bottleneck data distribution.
3.3 ADAPTIVE OOD ACTION CLASSIFCCATON
Building on the detection framework, we introduce an adaptive classification mechanism to handle OOD actions during policy optimization. Unlike conventional methods that indiscriminately pe- nalize all deviations, our approach distinguishes between beneficial and detrimental OOD actions through two-stage assessment process.
4

---

Published as a conference paper at ICLR 2026
Algorithm (Diftusion-Based OOD Detection with selective regularization (DOSER)
Initialize Q-network Qo. V-network Vo. diffusion behavior model cB., diffusion state model CB,> policy network T.. dynamics model pv, and target networks Qo., Vir, Te
# Model Pretraining
Pretraining dynamics model Pv by minimizing [3
Pretraining difffusin models εe, and Ee, by minimizing and (5)
Calculate OOD detection thresholds Ta and Ts based on in-sample reconstruction error for each iteration do
Sample transition minibatch {(s, a,r, s'}} from D
# Critic Learning
Generate action 0g ~ Ft. (8) and predict the next state 8' - Pvo (8, ax)
Select the best ID action a and predict the next state 8/a pvl (s, a
Calculate the reconstruction enrors of policy action and next state by and
Calculate the adaptive bonus dy
(x (s
Update Qo and Vo by minimizing
10 and 12
Actor Learning
Update T& by minimizing 14
Target Network Update
8' x pA (1 p)0', & po (1 p)
end for
For each policy-generated OOD action aood in state 8, we first predict the subsequent state 8 using the learned dynamics model pv ( |s,, s, a). pretrained via supervised learning on the offline dataset D. Since value estimation for OOD states is inherently unreliable, we then evaluate the outcome of Oood along two dimensions: 1) Whether s lies outside the training distribution, determined by the proposed OOD detection mechanism; 2) If s is in-distribution, whether Vs*',, exceeds V V( where &y denotes the predicted next state after executing the optimal in in-distribution action.
Formally, the classification rule for OOD actions is given in Definition
Definition (Beneficial and detrimental OOD action sets), Let the beneficial OOD action set A and the detrimental OOD action set Abod be subsets of the action space A. Then:
A := {a A Es(s')) S Te V(s²,) V(saa))
A == {a E (s(,)) T V(*()) < V((aa)
where 8'z ~ P (-18,, Good), 8/4 ( ) a arg max Q(s, a) is the optimal in- distribution action at state 8, (.) the state reconstruction error defined in 7 and Ts is the state OOD threshold.
Accordingly, detrimental OOD actions are penalized to mitigate overestimation. Conversely, to encourage exploration beyond dataset support, beneficial OOD actions receive an adaptive bonus dy V V(*,, (s(a)) This compensates for extrapolation errors in value estimation and guides the policy towards high-value regions, even when Q-value estimates for OOD actions remain imperfect.
Therefore, we minimize the following loss for policy evaluation:
(())= = ((,...e' (Qa(s, a) (R(s, a) + TB ( Qor a.) 2
Standard Bellman
+ 3 E am)))()ss) Ila Abaa) (Qo(s, a) - Quin )²
(10)
Penalty detrimental
+ A AEEDD a+))())))) Ilaa Atoa) (Qa(s, a) (Qu aia) + 8v))²
Bonus for beneficial OOD actions
where Qa is the target Q-network, Qmin = Rmin /1 7) is the theoretical minimal Q-value of the MDP. In practical implementation, we approximate aj as:
àj arg max Q(s, a;) for i 1- 1, N
(11)
with N = empirically balancing computational cost and performance across all tasks.
5

---

Published as a conference paper at ICLR 2026
Delese:
Nocel
Drspaoy:
CAsE Reconarusiler
00
mamatad OOD betction Mocric
-10
+10
-10
-16
Modkam Dalrad.
(iikk Phoorsoociin Drow
Nozel boommbee
MC Dhagoy
Cyee Ausomarvciar Eng
0.1
03
-10
-1.0
-10
-10
-1.
starE
(c)
(b)
(c)
(c)
(e)
Figure 3: OOD detection experiments ID navigation task, where higher OOD detection metric (reconstruction error or uncertainty estimation) indicates a greater likelihood of being OOD. (a) Two offiine datasets with distinct data distributions: expert (top) and medium (bottom). (b) OOD scores across the entire state-action space, evaluated using diffusion- based reconstruction error. (c) OOD scores based on model ensemble uncertainty. (d) OOD scores based on MC dropout uncertainty. (c) OOD scores derived from CVAE-based reconstruction error.
3.4 PRACTICAL IMPLEMENTATION
In this section, we provide the practical implementation of algorithm.
Value Learning. Similar to IQL., we perform expectile regression to train the value network.
(()) E(o.r,"">> (r - >V' - [=()
(12)
where L5(u)= - I(w < 0)/u² denotes the asymmetric L2 loss, and Vy is the target V-network. Dynamics Model. With the quadruples (s, a, s') in offline dataset D, we train the dynamics model via supervised regression:
((v) = E "=..a.")~=llpp((( a) - s'12
(13)
Policy Learning. enhance exploration, we optimize the actor network with maximum entropy regularization:
(())= (a [a log TT - Q0(s.a))
(14)
where a is dynamically adjusted maintain target entropy.
Overall Algorithm. Putting everything together, we summarize our implementation in Algorithm
4 EXPERIMENTS
In this section, we conduct series of experiments to validate the effectiveness of our proposed method. We aim to answer the following key questions: 1) Is diffusion-based reconstruction er- ror better than existing approaches in detecting OOD samples? 2) How does DOSER perform on standard offline RL benchmarks compared to prior SOTA methods? 3) Does each component in DOSER contribute meaningfully to the overall performance? 4) How sensitive is DOSER to its key hyperparameter? More experimental details and results are provided Appendix B and
4.1 OOD DETECTION
To evaluate the effectiveness of diffusion- based reconstruction error for OOD detection, we design simple 1D navigation task, the discrete state-action space is defined over position s E [-10, 10] and step size a E [- 1). The reward function is defined as the negative distance the target state 0, such that rewards increase the agent approaches the target. By perturbing optimal actions with
"oode available at /

---

Published as a conference paper at ICLR 2026
CVAE-based Action Detection
Difuuson-based
Détection
halfcheetah medium
halfcheetah medium replay
halfchestah medium expert--22
00 40 20-
50
Normal
20
25
hopper medium--
hopper-medium medium cepplayepllyy.....
hopper expartiv2
800-
100
Normal Score Normal tred
75
40
20
walker2d
walker2d medium replay
walkerr2d medium expert
W M
75
13
50
25
Steps (x1e6)
Steps (x1e6)
Steps (x1e6)
Figure 4: Comparison of OOD action detection performance between CVAE-based reconstruction emror and the proposed diffusion-basee method in the D4RL MuJoCo domain.
noise varying scales, we generate two offline datasets, expert and medium. We then compare our diffusion-based approach against three representative baselines:
1) Model ensemble. An ensemble of dynamics models is trained to capture epistemic uncertainty, with OOD samples identified based high prediction variance across ensemble members.
2) MC dropout. Monte Carlo dropout is applied during inference to approximate model uncertainty, where actions with high estimated uncertainty are flagged as OOD.
3) CVAE-based reconstruction error. A conditional VAE (CVAE) is trained to model the behavior distribution, and the reconstruction error is used the OOD indicator.
As shown in Figure 3 our diffusion-based method effectively separates ID and OOD samples across the entire state-action space, whereas baseline methods fail achieve reliable identification even in this simple setting. In particular, the model ensemble approach frequently misclassifies OOD samples as in-distribution due to its inability to disentangle epistemic and aleatoric uncertainty. Similarly, MC dropout tends to conflate these two sources of uncertainty, while also introducing un- desirable stochasticity at inference. Although the CVAE-based reconstruction error baseline shows stronger discrimination than the other two methods, its performance primarily stems from recon- struction ability, while its limited capacity to model multi-modal distributions remains a fundamental limitation (Wang al. 2022). For more experimental details, please refer Appendix B.4
We further compare the OOD action detection performance using CVAE-based reconstruction er- with proposed diffusion-based approach in the D4RL MuJoCo domain (Fu et al. 2020), with results presented in Figure 4 Both methods rely solely on reconstruction error as the detec- tion metric, without incorporating any additional classification compensation. As illustrated, the CVAE-based method struggles to reliably identify OOD samples in high nigh-dimensional continuous control tasks, which is attributed its tendency to produce over-smoothed reconstructions, thus di- minishing sensitivity to anomalous action inputs. In contrast, our proposed diffusion-bart OOD detection consistently delivers superior performance across all evaluated datasets.
4.2 COMPARISONS ON D4RL BENCHMARKS
We evaluate the policy performance of DOSER the standard D4RL benchmark, covering diverse set of continuous control tasks with varying dataset qualities.
We compare DOSER against broad range of baselines, including conventional algorithms and SOTA diffusion-baate based approaches. For policy constraint methods, we include TD3+BC (Fujimoto
& Gu 2021), IQI. (Kostrikov al. 2021) and A2PR (Liu et al. 2024). For value regularization
7

---

Published as a conference paper at ICLR 2026
Table 1: Evaluation results on D4RL benchmark. We report the average normalized scores the last training iteration over 4 random seeds. Note that m-medium, m-r=medium-replay, m m-e=medium- expert. Bold indicates the values within 95% of the maximum value.
Conventional methods
Diffusion-bssed methods
Dataset
TD3+BC
IQL A2PR
CQL
SVR
ACL-QL
DQL
SIBC
IDQL
QGPO SRPO
DIQL
DOSER (Ours)
halfchectah-m
48,3
47.4
68.6
44.0
60.5
69.8
51.5
45.9
51.0
54.1
60.4
57.9
67.5 + 0.5
hopper-m
59.3
66.3
100.8
58.5
103.5
97.9
90.5
57.1
65.4
98.0
95.5
99.6
104.0 0.5
walker2d-m
83.7
78.3
89.7
72.5
92.4
79.3
87.0
77,9
82.5
86.0
84.4
89.4
86.7 : 1.2
halfchectah-m--r
44.6
44.2
56.6
45.5
52.5
55.9
47.8
37.1
45.9
47.6
51.4
50.9
63.9 + 1.1
hopper-mn-r
101.3
86.2
92.1
101.2
walker22d mi
81.8
73.9
94.4
77.2
95.6
96.5
95.5
65.1
85.1
84.4
84.6
88.5
94.4 1: 1.3
halfcheetah-ma.
91.6
96.8
92.6
95.9
92.2
0.4
hopper-m-c
98.0
91.5
112.1
105.4
111.2
107.2
111.1
108.6
108.6
108.0
100.1
109.3
111.5 + 1.6
walker26-m--
110.1
109.6
114.6
108.8
109.3
113.4
110.1
109.8
112.7
110.7
114.0
110.0
110.9
0.2
MuJoCo-F2 Average
75.3
83.3
93.0
77.6
91.4
89.6
88.0
75.6
82.1
86.6
87.1
88.7
93.2
pen- humanu
$4.9
71.5
35.2
73.1
72.8
73.9
64.1
87.8 =: 14.7
pen-cloned
63.8
37.3
27.2
70.2
57.3
54.2
81.3
79.3 d: 8,9
Adroit-v1 Average
59.4
54.4
31.2
71.7
65.1
64.1
72.7
83.6
methods, we compare against CQL (Kumar et al., 2020), SVR (Mao et al. 2023) and ACL-QL (Wu ct al. 2024). For diffusion-based methods, we consider approaches that also leverage diffusion mod- els for behavior cloning, such as DQL (Wang et al. 2022). SÈBC (Chen et al. 2022 IDQL (Hansen-) Estruch et al., 2023), QGPO (Lu et al. (023). SRPO (Chen et al. (023) and DTQL (Chen et al. 2024). Baseline performance is taken from original papers recent literature. Some baselines did not report results on the pen tasks, and key hyperparameters for reproduction are unavailable, so we mark these entries as "_".
As shown in Table I DOSER consistently achieves strong performance, outperforming prior meth- ods both Gym-MuJoCo and Adroit tasks. Its advantage is particularly pronounced in the more challenging "medium" " and "medium-replay" settings, where the datasets contain a significant pro- portion of suboptimal and heterogeneous behaviors. This highlights the effectiveness of our pro- posed ceftusion-based OOD detection mechanism and its ability of selective regularization. While existing diffusion- based baselines already exhibit improved performance over traditional approaches due to their expressive modeling capacity, DOSER further improves upon them by explicitly classi- fying OOD actions, which allows for more refined value estimation and better policy improvement. Note that methods such as SVR and A2PR also incorporate behavior modeling into their frame- works, either for value regularization or policy constraint. Specifically, SVR employs a CVAE to approximate the support of the behavior policy and imposes uniform penalties to actions that fall outside this estimated support. Similar to the motivation of DOSER, A2PR introduces an action discrimination mechanism to guide policy optimization. However, A2PR's discriminator is solely applied to in-distribution actions identified by an enhanced CVAE, thereby restricting policy learning to potentially inaccurate approximation of the dataset support. In contrast to these CVAE-based approaches, DOSER leverages the expressive power of diffusion models for more accurate OOD detection and employs a selective regularization strategy targeted OOD actions. This enables the leamed policy to extrapolate to high-value regions beyond the offline dataset, ultimately contributing to superior empirical performance.
4.3 ABLATION STUDY ON COMPONENTS IN DOSER
To systematically validate the effectiveness of each component in the DOSER framework, we con- duct ablation studies on two variants.
1) DOSER w/o AC and VC. This variant removes both OOD action classification (AC) and value compensation (VC). It relies solely on diffusion- based reconstruction error to detect OOD actions, applying uniform penalty without distinguishing between beneficial and detrimental cases. This serves as a direct test of the core capability of diffusion models in OOD detection.
2) DOSER w/o VC. Building the baseline above, this variant further differentiates OOD tions by incorporating both next-state distribution modeling and value estimation. Specifically, identifies OOD actions that either (i) lead to OOD states or (ii) yield lower value outcomes than op-
8

---

Published as a conference paper at ICLR 2026
Table Components ablation across MuJoCo-v2 tasks.
Method
halfcheetah
hopper
walker2d
BM-C
mhe
DUSER wdo and VC
65.4 1.1
588 + 1.
94.0
102.1 1.7
104.2 + 1.3
108.3 + 2.5
X5.4 0.4
94.3 + 1.5
1108 0.4
DOSER VC
67.2
0.9
6199 1
960
99.4 1
1052 1 188
11124 32
858 1: 0.6
93. 1.0
111.1
0,5
DUSER
0.5
104.0
timal ID actions as detrimental, penalizing only those. All other OOD actions are retained without regularization, enabling more nuanced treatment of OOD behavior.
We keep all hyperparameters fixed across these variants and evaluate performance on MuloCo loco- motion tasks. 2 reports the average normalized scores of DOSER and its two ablated variants across nine datasets. The complete learning curves provided in Appendix C.9
The results show that even the baseline variant (DOSER w/o AC and VC) already performs com- petitively with existing SOTA methods, confirming the strong effectiveness of diffusion models for OOD action detection. However, uniform penalization strategy excessively suppresses po- tentially beneficial OOD actions, leading to noticeable performance degradation. In contrast, the classification--asce variant (DOSER w/o VC) alleviates this issue by selectively regularizing only detrimental OOD actions, resulting in smaller performance drops. Overall, these findings strongly validate the effectiveness DOSER's fine-grained classification and compensation mechanism in better balancing conservatism and exploration during policy optimization.
4.4 SENSITIVITY ANALYSIS
We compare different OOD detection thresh- olds in Figure S(a) set as the p-th percentile of in-distribution reconstruction error.. A smaller threshold implies more samples will be iden- tified as OOD, which beneficial for row behavior distributions like in the "medium- expert" dataset, where a larger threshold might overlook OOD samples. For more diverse datasets like "medium" and "medium-replay" larger thresholds are preferred to prevent ID samples from being misclassified as OOD.
Sco Doc 3mme
Graclent Stepe le
Gridlet steps Steps Ocles)
01
Graclert Stepe Inlee)
(a) OOD detection thresholds Ta Ts.
medium
mid Mmpeypla
med mroium-erpert
Homatzee Scre
2-13-
We also investigate the impact of the penalty
2-13--
coefficient 3, varying it from 10-5 5 to 1, as shown in Figure ((b) Datasets with narrow dis- tributions require a larger /3 to prevent value overestimation, while more diverse datasets benefit from smaller 3 to avoid suppressing beneficial OOD actions.
3-1
Grdden
(b) Penalty coefficient B.
medium
mediummnaper
Nomatoed Scrre
An ablation study on the compensation coeffi- cient À in Figure ((c) shows that DOSER per- forms well across a wide range of X values on the more diverse datasets. Setting yields stable performance across datasets, while excessively large values can amplify the com- pensation effect, leading to value overestima- tion and disrupting the learning process.
0.001
Gracient Stops (x2e6)
Cradiant Staps (#2ed)
Grackent Stops (e
(c) Compensation cocfficient X.
Figure 5: Ablation study on hyperparameters for halfcheetah tasks.
RELATED WORKS
OOD Detection. Reliable identification OOD samples is critical for the robustness of ma- chine learning systems. Existing methods primarily fall into two categories: cenerative-based and reconstruction- based. Generative-based methods leverage probabilistic models to estimate the likelihood of test samples under the learned distribution (Ren al. 2019). but models such as

---

Published as a conference paper at ICLR 2026
Glow (Kingma & Dhariwal 2018) and VAEs (Kingma & Welling 2013) often assign higher likeli- hoods to OOD samples than to ID data (Hendrycks et al. 2018 Nalisnick et al. 2018)) Although improvements like likelihood ratios (Ren et al.) 2019) and typicality tests (Nalisnick et al. 2019) have been proposed, their reliance on likelihood estimation remains a fundamental limitation. In contrast, iiconstructionnbasee methods (Denouden et al. 2018 Zong et al. 2018) directly measure reconstruction quality, based on the premise that models trained on ID data reconstruct familiar pat- terns well, while exhibiting significant errors on anomalous inputs. Traditional autoencoders (Lyud] chik 2016) and more recent diffusion- results in this regard, with diffusion models leveraging iterative refinement to further enhance ID re- construction. Consequently, reconstruction error provides a more reliable signal of distribution shift than likelihood- metries, offering improved discriminability between ID and OOD samples.
models (Graham et al. 2023) have shown promising
Offline RL presents additional challenges for OOD detection due to the lack of online interaction. To mitigate the risk of extrapolation error, BCQ (Fujimoto et al.) 2019) and SVR (Mao et al. 2023) employ VAEs to approximate the behavior policy, constraining the learned policy to remain within behavior support. However, VAEs often fail to capture multi- modal distributions accurately (Wang et al. 2024), resulting in oversimplified generations. Another line of work quantifies uncertainty to identity OOD samples. Model ensemble methods (Laksh- minarayanan al.. (017) identify OOD state-action pairs via predictive variance, with algorithms such as MOPO YYu et al. (020) incorporating this uncertainty as a penalty into the reward function. Similarly, Monte Carlo (MC) dropout offers a computationally efficient approximation Bayesian inference (Gal & Ghahramani] 2016), and has been applied in offline RL for uncertainty-awant aware OOD detection (Wu et 20.1).21). While effective some extent, both approaches often conflate epistemic and aleatoric uncertainty, which may lead to erroneous identification of OOD actions (Zhang et al.) 2023). Alternatively, CQL (Kumar al. 2020) avoids explicit density estimation by regularizing the Q-function to assign lower values to all unseen actions. This implicit OOD detection eliminates the need for behavior modeling but risks being overly conservative, potentially suppressing valuable actions that lie outside the behavior support but could lead to improved performance.
OOD Detection in Offiine RI.
Diffusion Models in Offline RL. Diffusion models have recently emerged as powerful paradigms in RL for modeling multi-modal distributions. This capability is particularly valuable in offline RL settings, where capturing the diversity of behaviors is essential for deriving robust policies. Meth- ods such as Diffusion LL (Wang et al., 2022) and DAC (Fang al., 2024) incorporate Q-function guidance into the reverse diffusion process, shaping action generation toward higher-value regions. In contrast, IDQL (Hansen-Esttuunn ditional diffusion model to generate multiple action candidates for a given state, and subsequently resample according to Q-values select the best action for execution. Notably, while these ap- proaches effectively integrate diffusion models with value functions for policy improvement, their use of diffusion remains largely limited to guiding or selecting actions, none of them fully exploit the inherent properties of diffusion models, such as reconstruction fidelity or noise sensitivity, to directly assess whether state-action pairs lie within the support of the training distribution.
al., 2023) and SfBC (Chen et al. 2022) first pretrain a con-
CONCLUSION
In this work, we proposed DOSER, a framework that mitigates distribution shift through diffusion- based reconstruction error. Unlike prior methods that rely on heuristic uncertainty measures or unreliable likelihood estimates, DOSER leverages the expressive power of diffusion models to com- pute theoretically grounded reconstruction errors for both behavior policy and state distributions. This provides robust detection metries that overcome the multi-modality limitations of Gaussian- based approximators. Crucially, DOSER introduces a selective regularization mechanism that clas- sifies OOD samples into beneficial and detrimental actions, enabling suppression of detrimental extrapolations while compensating promising explorations via value-difference bonuses. Extensive experiments demonstrate that DOSER achieves superior or competitive performance compared to state-of-the-art methods, partieularly on suboptimal datasets.
Nonetheless, DOSER has two key limitations: 1) its reliance on the accuracy of the diffusion-bart reconstruction and the learned dynamics model, and 2) the computational overhead of the iterative diffusion sampling. Future work could focus on enhancing the robustness of dynamics model and improving efficiency via model distillation and accelerated sampling techniques.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work is supported by the Shanghai Municipal Science and Shanghai Automotive Industry Sci- ence and Technology Development Foundation (No. 2407).
REFERENCES
Kingma DP Ba J Adam et al. A method for stochastic optimization. arXiv preprint arXiv: 1412.698,, 1412(6), 2014.
Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhihong Deng, Animesh Garg. Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-drivet offline reinforcement learning. arXiv preprint arXiv:2202. 1166, 2022.
Stefan Banach. Sur les opérations dans les ensembles abstraits leur application aux équations intégrales. Fundamenta mathematicae, 3(1):133-181, 1922.
Liron Bergman and Yedid Hoshen. Classification based anomaly detection for general data. arXiv preprint arXhv:2005.02359) 2020.
Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. arXiv preprint arXiv:2209. /4548, 2022.
Huayu Chen, Cheng Lu, Zhengyi Wang, Hang Su, and Jun Zhu. Score regularized policy optimiza- tion through diffusion behavior. arXiv preprint arXiv:2310.07297, 2023.
Tianyu Chen, Zhendong Wang, and Mingyuan Zhou. Diffusion policies creating a trust region for offline reinforcement learning. Advances in Neural Information Processing Systems, 37:50098- 50125, 2024.
Yunqiang Chen, Xiang Sean Zhou, and Thomas S Huang- One-class svm for learning in im- age retrieval. In Proceedings 2001 international conference image processing (Cat. No. 01CH37205), volume 1, pp. 34-37. IEEE, 2001.
Taylor Denouden, Rick Salay, Krzysztof Czamecki, Vahdat Abdelzad, Buu Phan, and Sachin Vernekar. Improving reconstruction autoencoder out-of-distribution detection with mahalanobis distance. arXiv preprint arXiv:1812.02760 2018.
Francois Dufour and Tomas Prieto-Rument Finite linear programming approximations of con- strained discounted markov decision processes. SIAM Journal Control and Optimization, 51 12:1129-13244 2013.
Linjiajie Fang, Ruoxue Liu, Jing Zhang, Wenjia Wang, and Bing- Jing. Diffusion actor-critic: Formulating constrained policy iteration as diffusion noise regression for offline reinforcement learning. arXiv preprint arXiv:2405.20555, 2024.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. International conference on machine learning, pp. 2052-2062. PMLR, 2019.
Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059. PMLR, 2016.
Henry Gouk, Eibe Frank, Bernhard Pfahringer, and Michael Cree. Regularisation of neural net- works by enforcing lipschitz continuity. Machine Learning, 116 2021.
11

---

Published as a conference paper at ICLR 2026
Mark S Graham, Walter HL Pinaya, Petru-Daniee Tudosiu, Parashkev Nachev, Sebastien Ourselin, and Jorge Cardoso. Denoising diffusion models for out-of of-ssriibbttion detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
2023.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abheel, et al. Soft actor-critic algorithms and appli- cations. arXiv preprint arXiv:1812.05905, 2018.
Philippe Hansen- -atsennEsttuuch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idgl: Implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304. 10573, 2023.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. arXiv preprint arXiv: 18/2.046000 2018.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Joey Hong, Aviral Kumar, and Sergey Levine. Confidence-conndttione value functions for offline reinforcement learning. arXiv preprint arXiv:2212.04607, 2022.
Zhenbo Huang, Jing Zhao, and Shiliang Sun. De-pessimism offline reinforcement learning via value compensation. IEEE Transactions on Neural Networks and Learning Systems, 2024.
Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline ri? In International conference on machine learning, pp. 5084-5096. PMLR, 2021.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in neural information processing systems, 35:26565-26577, 2022.
Diederik P Kingma and Max Welling. arXiv:1312.6114, 14, 2013.
Auto-encoding variational bayes.
arXiv preprint
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions. Advances in neural information processing systems, 31, 2018
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. arXiv preprint arXiv:2110.06166 2021.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off off-policy q-learning via bootstrapping error reduction. Advances in neural information processing systems, 32, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in neural information processing systems, 33:1 13:1179-1191, 2020.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances neural information processing systems, 30, 2017.
Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. Reinforce- ment learning: State-of-the-ari, pp. 45-73. Springer, 2012.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives open problems. arXiv preprint arXiv:2005.0164; 2020.
Tenglong Liu, Yang Li, Yixing Lan, Hao Gao, Wei Pan, and Xin Xu. Adaptive advantage-guides policy regularization for offline reinforcement learning. arXiv preprint arXiv:2405.19900 2024.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:/600 1608. 003983, 2016.
12

---

Published as a conference paper at ICLR 2026
Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu. Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforeement learning. In Inter- national Conference Machine Learning, pp. 22825-22855 PMLR, 2023.
Olga Lyudchik. Outlier detection using autoencoders. Technical report, 2016.
Yixiu Mao, Hongchang Zhang, Chen Chen, Yi Xu, and Xiangyang Ji. Supported value regulariza- tion for offline reinforcement learning. Advances in Neural Information Processing Systems, 36: 40587-40609, 2023.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan, Do deep generative models know what they don't know? arXiv preprint a:Xi::/810.009136,, 2018.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, and Balaji Lakshminarayanan. Detect- ing out-of-distribution inputs to deep generative models using typicality. arXiv preprint arXiv:1906.02994 2019.
Yuhang Ran, Yi-Chen Li, Fuxiang Zhang, Zongzhang Zhang, and Yang Yu. Policy regularization with dataset constraint for offline reinforcement learning. In International conference on machine learning. pp. 28701-221 287117 PMLR, 2023.
Jie Ren, Peter J Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out-of-distribution detection. Advances in neural information processing systems, 32, 2019.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learn- ing. pp. 2256-2266 pmir, 2015.
Yang Song, Jascha Sohk-Dicks,ein, Diederik Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint W.Xiv:2011.13456, 2020.
Kingma, Abhishek Kumar, Stefano Ermon, and Ben
Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998.
Pascal Vincent. connection between score matching and denoising autoencoders. Neural compu- tation, 23
2011. 2011.
Mianchu Wang, Yue Jin, and Giovanni Montana. Learning on one mode: Addressing multi-mode multi-modality in offline reinforcement learning. arXiv preprint arXiv:2412.03255, 2024.
Zhendong Wang, Jonathan Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.061 06193, 2022.
Kun Wu, Yinuo Zhao, Zhiyuan Xu, Zhengping Che, Chengxiang Yin, Chi Harold Liu, Qinru Qiu, Feifei Feng, and Jian Tang. AcI-ql: Adaptive conservative level in q-learning for offline reinforce- ment learning. IEEE Transactions Neural Networks and Learning Systems, 2024.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning- arXiv preprint
2019.
Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua Susskind, Jian Zhang, Ruslan Salakhutdinov. and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. arXiv preprint 10X.v:2105.088 /40, 2021.
Huaqing Xiong, Tengyu Xu, Lin Zhao, Yingbin Liang, and Wei Zhang. Deterministic policy gra- dient: Convergence analysis. In Uncertainty in Artificial Intelligence, pp. 2159-2169. PMLR, 2022.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129-14142. 2020.
13

---

Published as a conference paper at ICLR 2026
Shuangfei Zhai, Yu Cheng, Weining Lu, and Zhongfei Zhang. Deep structured energy based models for anomaly detection. In International conference on machine learning, pp. 1100-1109, PMLR, 2016.
Hongchang Zhang, Jianzhun Shao, Shuncheng He, Yuhang Jiang, and Xiangyang Ji. Darl: distance- aware uncertainty estimation for offline reinforcement learning. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 37. pp. 11210-1 1218, 2023.
Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Daeki Cho, and Haifeng Chen. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In Inter- national conference on learning representations, 2018
14

---

Published as a conference paper at ICLR 2026
APPENDIX
A THEORETICAL ANALYSIS
In this section, we provide the formal definitions and theoretical analysis in the paper.
A.1 DEFINITIONS
Definition 2 (In-sample Bellman operator). The in in-sample Bellmun operator is defined as:
(s, a) == R[s, a) at
(15)
where R the empirical behavior policy in the dataset.
Based Definition 2 DOSER operator is defined follows.
Definition 3 (DOSER operator). From optimization perspective, (TO lead to the DOSER policy evaluation operator:
[in a) (Qadj(s, (q
as)
TDOSER Q(s,
(16)
otherwise
where Quaj(s, a) is the adjusted Q-target for OOD actions:
if a E Aood Qudi(s, a) = (Q(ss + dv) if Apod
[Qmin
(17)
Therefore, DOSER guarantees that the Q-values of ID actions remain unbiased, while underestimate those detrimental OOD actions. By applying value compensation By to beneficial OOD actions, it incentivizes exploration toward eigh-potential state-action pairs.
A.2 THEOREMS
Theorem (Contraction mapping property). For arbitrary Q-functions Q1 and Q2 defined on the whole state-action space S x A, the DOSER operator TDOSER constitutes a "y-contraction mapping in the Lox norm:
IDDOSRRQ Q1 -abosERQllls S rl1Q1 Q21loo
(18)
By the Banach fixed-point theorem (Banach unique fixed point from any initial Q-function.
1922). repeatedly applying TDOSER converges
Theorem 2 (Bounded value estimation). For any policy n, let QOSER denote the unique fixed point of the DOSER operator TDOSER- Then, QOSEE satisfies the following boundedness property for all (s, a):
Qmin S @DOSER a) I Q (s, atd + nōv, where Vi (s, ala) maXa~na((le) (s, a) is the optimal Q-value by iterating the in-sample Bellman operator Tin-
(19)
Since QT, corresponds to the fixed point of the in-sample Bellman operator, yields reliable value estimates within the data distribution. Therefore, Theorem 2 implies that DOSER incurs controlled value overestimation while enabling exploration to high-value regions via the compensation meeh- anism. The upper bound tightens as the compensation target weight 7 and value difference dy decrease, since smaller values of these parameters directly constrain the magnitude of the value adjustment for OOD actions, aligning the value estimate closer the in-distribution baseline QT,
By dynamically adjusting how OOD actions are treated based on their predicted outcomes, the pro- posed selective regularization mechanism balances safety and performance, avoiding the pitfalls of binary classification. Crucially, our method preserves standard Q-learning convergence guarantees while enabling safer exploration beyond the behavior policy support.
15

---

Published as a conference paper at ICLR 2026
Theorem 3 (Bounded critic deviation). Let Nee denote the reference policy obtained with the true environment dynamics P and without OOD detection error, with Qa being its corresponding action-value function. Let it denote the learned policy of DOSER under dynamics model approx- imation error Edyn and an OOD detection misclassification probability Edet- Then, the deviation of the learned critic from Q is bounded follows:
Q2 Trot 1100 I
7
(Qmax (ClEdyn C2Edet) njvv,,
(20)
--
Rimax
and C1. C2 are constants that capture the sensitivity of the policy optimiza- where Quax 1-7
tion process to dynamics and detection errors respectively.
Theorem (Performance gap of DOSER). Let से be the policy learned by DOSER through iterative application TDOSER, and let n² denote the optimal policy. Suppose ö, represents the function approximation error. Then the performance gap between πt" and π. satisfies
S CLp.-max (Creayu + C2Edet);
- -
(21)
1-7
where C1 C2 are positive constants, and Lp is the Lipschitz constant of the environment dynamics.
Consequently, the performance gap is influenced by three key components: the function approxima- tion error öy, the OOD detection error édet, and the dynamics model approximation error Edyn- In our setting, the diffusion model provides reliable mechanism for OOD detection, as confirmed by extensive experiments, which keeps det small. Meanwhile, the learned dynamics model maintains stable predictive performance, ensuring that Edyn remains bounded. Therefore, when the diffusion reconstruction error becomes negligible and the dynamics model is sufficiently well fitted, together with small ō,, then the right-hand side of the bound vanishes, implying J(T) + J(x").
A.3 PROOFS
A.3.1 PROOF OF THEOREM I
Proof.
Let TDOSER denote the DOSER operator acting on bounded Q-functions defined S x A. Assume
Q Q-functions lie in the Banach space (B, || Ilso) of bounded real functions on S x A with the sup-norm;
the compensation coefficient satisfies 1 S <;;
the colpencompnnnntion term dy is a scalar that does not depend on the Q-function being evaluated (i.e., treated as fixed when comparing two Q-functions; ōy depends on Q. then it must be Lipschitz continuous in Q with sufficiently small Lipschitz constant to preserve contraction; here we assume it is fixed for simplicity and clarity).
Let Q1,Q2 E be two arbitrary Q-functions. We will bound [/TDOSER - TDOSER Q21le by considering the three types of actions that DOSER treats differently: 1) in-distribution actions, 2) detrimental OOD actions, and 3) beneficial OOD actions.
1) In-distribution actions. For any (s, a) with Ea a) S Ta. DOSER reduces to the in-sample Bellman operator
TOOSER ((8, a) T(n R(s, a) Ey Ps a'>As(())) (' (22)
16

---

Published as a conference paper at ICLR 2026
Hence, the contraction property follows from standard Bellman operator properties:
ITDOSER (s, a) Q2 Q26, a)
Q1, - Q2) lc
(R(s, ( + [Qi a'))) (R(s, + ta 1Q2(s', , a
-ymax [a [Q) a') Q2(s a')ll
8,a
(23)
y max Es a Qi a') (2(5',a'))
8,8
S ymax |Q1 Q21lxc
8,8
7 || Q2 oc
Thus for all in-distribution (s, a), we have
[iDOERRQ TDOSERQ2Ie Q2 S r1111 - Q211o
(24)
2) Detrimental OOD actions. For detrimental OOD actions a AboD Q-target are set constant Qmin (independent of the current Q):
ThosEn((s. Qmin
(25)
The difference vanishes for any Q-functions Q1, Q2:
[ a) TDDoSEnQ(ss, as = ||Q - iunullso = 711Q: -2Q1=
(26)
3) Beneficial OOD actions. For beneficial OOD actions
AoOD DOSER applies value
compensation, quantified as the difference between the value of the next state 8'x reaching by taking the beneficial OOD action a and 8/ after executing the best ID action an
TROSERQ(* a) (0(,, aia) + 5v)
(27)
Q(s, a) + V - V(s(aa) = max (-
where by assumption &y is treated as fixed scalar w.r.t. the Q-function comparison. Thus l[TDosERQiss,O a) T2 a)
|| (Qi(s, a.a.)) + óv) - 1/ (Q2(8, a(a,)) + dv) 1lx0
1/ max IQ a - Q2(8, a
8.a
(28)
17 max
max Q2 a) max Q2(s.
S 1 max ||01 - 1211a
8.0
< Q1 - Q21
Combining all three cases, we have I
- S <11Q T
Q21loo
By the Banach fixed-point theorem (Banach 1922). TDOSER admits a unique fixed point in B and iterative application of TDOSER from any initial Q-function converges to that fixed point at rate most
A.3.2 PROOF THEOREM 2
Proof. Suppose the DOSER operator TDOSER admits a unique fixed point QĪOSER (Theorem Assume the compensation term &y is a scalar that does not depend on the Q-function being evaluated (if by is estimated from Q Lipschitz assumption on this estimator must be made).
We reason by cases according DOSER's treatment of actions. TOSER exists and for each (s, a) satisfies
By Theorem the fixed point
Tia ThnQoSER (s,
if if a) S Ta (in-distribution)
@OSER (s, a)
Qmin
if AOoD
(detrimental OOD)
(29)
( (8, a%a) + 5y)
if E AtoD (heneficial OOD)
We show the two inequalities (lower and upper bounds) by treating each action type.
17

---

Published as a conference paper at ICLR 2026
1) Lower bound: QoSER (s, 2 Qmin.
In-distribution actions. For Ea(s, a) Tax we have the in-sample Bellman fixed-poin -eed-ppintt
relation
DOSER (8, a) R(s, a) + ((. a'~ ( (5', 'a))
(30)
Since R(s, a) 2 Rmin and the fact that for all successor pairs @oSER (8', aa) 2 Qmins we obtain
DOSER (8, a) 2 Rmin + 7Q Qmin
(31)
Detrimental OOD actions. The operator directly assigns OoSER a) Qmin for a AOOD the lower bound holds with equality.
Beneficial OOD actions. For AooD they receive value compensation weighted by 'I E (0,1) 1). Given that the fixed-point value ØBSER (8, azg) is at least Qmin- dy 2 is
satisfied for beneficial OOD actions, and Qmin is strictly negative, have:
@osEn (s, a) = "7QoosE aia)) + 8v) 2 1) Qmin 2 Qmin
(32)
Combining the three subcases establishes the global lower bound QoSSER (8:a) Quin for all state-action pairs (s, a).
2) Upper bound: MOSER (s. I QT, (s. + nōv. Let Q denote the fixed point of the in- sample Bellman operator Ti, by construction QDOSER (s, a) Q (s, a) for every ID state-action pair (s, a). This upper bound guarantees that DOSER incurs only limited overestimation.
In-distribution actions. If E, a) S Tax then
@DSSER (s, a) Qi, (s. a) s Qh. ataa) S (T, a,a) + nōv
(33)
since nōv 2 0.
Detrimental OOD actions. For a E AñoDi
@OSER a) Qmin S (T, aia) nōv
(34)
Beneficial OOD actions. For a E A
@DOSER (s, a) (QQOSER (s, ala) + 5v)
(35)
Note that ats is an in-distribution action, hence
@bosEn(s.d; ama))
(36)
Substituting yields
(soSER(s.c) a) = (s. 5v) < Qi, and + nōv
(37)
Putting together the three cases yields the desired upper bound @OSER (s, a) s Qi, (s, agd) + nov for all (s, a).
Combining the lower and upper bounds above, we obtain for any state-action pair (s, a)
Quan < @OSER (s, a) S QT (8, a(a)) + 1jp
(38)
which shows the fixed-poin values uniformly bounded and that DOSER prevents uncontrolled value overestimation while permitting strategic exploration of beneficial out-of-distributin regions.
18

---

Published as a conference paper at ICLR 2026
A.3.3 PROOF OF THEOREM 3
We begin by introducing three key assumptions and an auxiliary lemma that will be used in the proof.
Assumption 1 (Dynamics model error bound). There exists a constant Edyn 2 0 such that the
learned dynamics model P(- s, a) is uniformly close to the true transition kernel P(: 8, a) in the (nnormm satisfying for all (8, a) E x A:
|P((1 8, P(. (,,18,)) S Edyn-
(39)
Assumption (OOD detector enoor bound). There exists constant Edet 2 0 such that the misclas- sification probability of the Out-of- Distribution detector uniformly bounded:
Pr Pr|detector misclassifies (8, a)] S Edet for all (s, a) E ESxA
(40)
Assumption 3 (Policy deviation bound). There exist constants C2 C2 O, characterizing the sensirivity of the policy optimization to dynamics model and OOD detection errors respectively, such that for all states 8 S:
| Teef( |)]|TV Cleayn C2adet
(41)
where Edyn and Edet defined in Assumptions Dand? 2
Lemma 1. Let μ and v be two probability distributions over a finite set X, and let X R be bounded function with of S M. Then,
[B [[(x]] - [x S 2M Ilu-V
(42)
where vl/rv sup (p(A) v(A) is the total variation distance. For a finite set X, this equivalent to || u|lrv = ExEx (m(x) - v(x))
Proof. The expectation difference can be written as:
[,, E.[
Σ [())u(t) - E ()))(()
DEX
ZEA
Σ ((oKu(()--(an)
EE
(43)
| (m(x) - u(x))
ZEX
S M Σ [m(x) v(x)
ZEX
2M - 1|ln-
The last equality follows from the definition of total variation distance.
Now we start the proof of Theorem
Proof. We begin by defining the Bellman operator associated with the reference policy Tre
Q( a) == r(s, a) + Pa (Q((', Denote the fixed point of the reference operator as Qura so that
(44)
Quret Tre Teet
(45)
DOSER critic constructs modified Bellman target due to three factors: (i) dynamics model error, (ii) detector misclassification, and (iii) value adjustment. Accordingly, the DOSER Bellman operator can be expressed as:
TOOSER Q(+, a) r(s,a) a) F. -1 b(s' ,a)))
(46)
19

---

Published as a conference paper at ICLR 2026
where 7 differs from Tee due dynamics model error and OOD detector error, represents the value adjustment applied to the target Q.
Now compare the difference between the two operators when applied to Quret, Define
(ss, a) := [ foo (s, a) T n (s.
(47)
Substituting the operator definitions yields:
S >>((T) + (IT) (ss, a) > E p. IQ Tre + b] Ep Ep [Qucet] where the components correspond
(48)
(1) = p (ppps1Q0 [Q - E P. [0) (11) := 16 p [6,
(49)
Bound on (I): We decompose (I) into the dynamics model approximation error and policy distribu- tion bias:
(T) | [ut]] E (Ta + Ep (Qrre] Ep [Q
(50)
< |pps [ - Ep Q Tm] I1 + 1Bp==[Q - EP [Q
For the dynamics model error, consider the function f(s') Ba (a)) 12 10 (s', aa)] Since TQ I Q maxx the function is bounded by Il I Qmax. Applying Lemma [ with distribu-
tions μ P(- 8, a) and P(. 8. a) gives:
Bpp [Q= Ep [Q [0]]
E p[f(s')()]]))))) Ef'ss(((s')]
(51)
< 20 max || 8, P(. 8, a,allry
Using the definition of TV distance ||p- 1/|TV = 21111=11 1112 S Eayn and Assumption
we have:
=pp=(Q=mm - Ep, [p $ 2Qmax Edyn Q max
(52)
For the policy distribution bias, apply similar argument in the action space A: & ( [oo]]
E paae [roor]] s 2Q || 88) meer(: s llVl
(53)
2Q (CiEdyn + C26det)
Therefore, the combined bound for (I) is:
(1) Qmax ((1 + 2C1)) 2C2Edet)
(54)
Bound on (II): Given [b] S ,jōv, it follows directly that:
(II) Ep, [b] < Ep S nōv
(55)
Thus, for all (s, a),
(ss, a) S > (Q (1 + 2C1)) + 2C2Eder) + nōv) Consequently, the operator difference is bounded in the supremum norm by:
(56)
[[(TOOSER TM m| < <Q + 2C, ( 202edet) + nσv)
(57)
By Theorem in the main paper, the DOSER critic converges to the fixed point of TOOSER. Thus:
Q TDOSER
(58)
We now bound the final approximation error:
Q Teel 1o || TTOOSER Q Tree Tief Tiee 10
< || ITDOSER Q - TDOOSE Q 11 + [I(TOOSER - -Trot)Q
(59)
< @ QT 11 y (Qmax ((1 + 2C, (aam 2C nōv)
Rearranging terms:
(1 2 Q Trरd 11ا0 S (Q ((1 + 2C Eayy 2C26det) + nōv)
(60)
Absorbing the constants into Cj and C2 yields the final result:
TQ-QFrre aloc 5 ? (Qmaxe (CrEdy + Czedet) + njov)
(61)
This completes the proof.
20

---

Published as a conference paper at ICLR 2026
A.3.4 PROOF OF THEOREM 4
We first make several common continuity assumptions about the learned Q function and the transi- tion dynamics P. which is frequently employed in the theoretical analysis of RL (Gok et al. 2021
Dufour & Prieto-Rumeau 2013).
Assumption 4 (Lipschitz Q) For all E and a1, a2 A, the learned value function is Lo- Lipschitz, then
[IQ a1) ((s, a2)1l S Lollar a2ll- Assumption 5 (Lipschitz P). For all E S and a1, 02 A, the transition dynamics is Lp-
(62)
Lipschitz, then
|P( s, s1) P(- s, a2)ll S Lplla1 - a2ll.
(63)
Lemma 2. Under Assumptions
5 the following inequality holds:
IV( S CLp max [[T(()) To (s)]l
(64)
where C. is a positive constant and d= is the state occupancy under policy
dt (8) (1
7 Er [[[ 8]]
(65)
X=
Proof. Please refer to Lemma Xiong et al.
2022 Lemma A.5 in (Ran et al. 2023
Now we start the proof of Theorem
Proof. The proof proceeds by decomposing the overall performance gap between the optimal policy n and the learned policy ग into manageable components. then bounding each term individually.
Similar to Theorem 3 let Tref denote the ideal reference policy, then
[J(m") J(7) = [J(x') ((Tret) [(Trett - J(R) < J((x)) - J(Tret) + [J(mTett) - J(H)
(66)
The first term captures approximation error due to function approximation, we denote as 5,. Under the asymptotic regime where the empirical fitting errors vanish, this term be arbitrarily small. Hence we focus on the second term, which quantifies the performance gap between the learned policy and the reference policy.
[(Trrt) [(
1
1
E iard [r(()))))(
-1 E Eandt [r(s)]
Σ (dFrc(()) - (d))rr()())
1 ?
id (s) d (s)]l=(8)
(67)
Rmax TV (s)lldd (s))
1-7
CLpBmax
max [ier(s) #(s)l
CLpRmax
(CiEdyn + Czedet)
1-7
Combining both enror terms yields the overall performance guarantee:
Rmax (CyEdyu + Czedet)
[J(z)) - ( Sd 1-7
(68)
21

---

Published as a conference paper at ICLR 2026
B EXPERIMENTAL DETAILS
B. DIFFUSION MODEL FRAMEWORK
We adopt the EDM framework (Karras et al. 2022) to leverage the advantages of continuous-time diffusion models for offline RL. EDM builds upon the continuous-time formulation derived from diffusion processes, which allows us to use an optimized ODE solver for sampling. This solver adaptively determines the steps along the noise level trajectory, significantly reducing the computa- tional load and accelerating generation speed, while maintaining high sample quality compared to sampling with fixed discrete schedule.
Noise schedule. In the DOSER framework, the noise schedule is crucial component of the diffu- sion model, defining how the noise levels vary time. Following the insights from the EDM pa- per, the noise schedule de sampled from a log-logistic distribution oN ~ log-logistic danu: 8), where log data serves as the shape parameter and s as the scale parameter. Using this schedule, noisy action a, is constructed as Og + o,6, with e ~ (0, I). The parameters are configured as follows: daata 0.5, and the noise schedule is clamped between miin 0.02 and mmax 80.
Training loss. The EDM framework precondition the neural network with or-dependent skip connection to improve numerical stability. Specifically, the denoising network for behavior policy modeling is defined as follows:
60_ (at, 0t, 8) Cakip ( + Coue (os Fo, (oe); Croise (o1)]s)
(69)
Similarly, the denoising network for state distribution modeling is defined as:
Eo, (st, σr) Cakp( + Coul (o,) Fo. (Cin(ore): Croise (ot))
(70)
where Fe. and Fø, the neural networks be actually trained, (a,) modulates the skip connection, (az( and (az) scale the input and output magnitudes respectively, and Croise (or.) maps noise level o, into conditioning input for Fe, and Fe.,
We can equivalently express the loss with respect to the raw network output Fe. in
E...,,a.e
M(rr) (o:)) (cm( (a <); (o)
(oi) (a Cskip (oi) (a + +))l
Similarly, the loss (5) can be expressed based on 70):
Bay ((o (o))l (Ci(ar) (s <); Cauise (ot)
(s Cakip(0)) (s c))
cou((r,)
(72)
According to the variance normalization principles, we follow the practical implementation of EDM in parameter choice:
Cakip oanaa/ (ot + Tzua)
Cour(oz)
= Ge d Vo? + oatn
(or(
1/(0? + -oban)
(73)
Cnoise (ae)
4 II((o:)
A(ox)
(0, +12 (/, d
B.2 NETWORK ARCHITECTURE
Behavior policy and state distribution modeling. Following Chen et al. (2024). we implement both our behavior policy and state distribution as MLP-based diffusion models. The denoising net- work for behavior policy Eg, (a,, s) is conditional diffusion model that predicts actions given noisy action vector a,, diffusion timestep (encoded via sinusoidal positional embedding), and state condition In contrast, the denoising network for state distribution Eg, (st, t)) is unconditional
22

---

Published as a conference paper at ICLR 2026
diffusion model that predicts states from noisy state 8 and timestep embedding. Both models share the same base architecture, which consists of a 4-layer MLP with Mish activations and 256 hidden units per layer. The main difference lies in their input dimensions, the behavior policy net- work additionally concatenates the state condition 8, while the state distribution network operates without conditioning.
Critic Networks. Following the implementation of SVR (Mao et al. 2023). the critic network comprises four Q-networks and V-networks, each implemented 3-layer MLPs with 256 hidden units per layer and ReLU activation functions.
Actor Network. The actor network adopts Tanh-Gaussian policy structure similar to SAC (Haarnoja et al. 2018). It is implemented as 3-layer MLP with 256 hidden units and ReLU activations in all hidden layers. The network supports both deterministic and stochastic action sam- pling, while preserving entropy regularization for effective exploration.
Dynamics Model. The dynamics model is implemented as 3-layer MLPs with 256 hidden units and ReLU activations, which takes concatenated state-action pairs as input and predicts both the next state and reward.
B.3 HYPERPARAMETERS
Diffusion models and networks share the same hyperparameter settings across all tasks. The detailed configurations are provided in T
Table 3: Hyperparameters for all tasks.
Hyperparameter
Value
Optimizer
Adam AAdm et al. 2014)
Learning rate
3e-4
Learning rate decay
Cosine (oshchilov & Hutter 2016
Batch size
256
Discounted factor
0.99
Target update rate
0.005
Policy update frequency
2
Target network update frequency
2
Number of sampled actions
10
Compensation coefficient
0.001
Compensation target weight 77
0.9
To accommodate varying data distributions across different tasks, employ task-specific hyperpa- rameters including the penalty coefficient /3 for detrimental OOD action penalty, the OOD detection thresholds Ta and т, for actions and states, the expectile regression factor and the lower bound of Q-value Qmin with their specific values for each task configuration detailed in Table 4
B.4 EXPERIMENTAL DETAILS ON TOY EXAMPLE
For the ID navigation task illustrated in Figure 6 a), the state space [-10, 10) represents the agent's current position, while actions correspond to step sizes within 1]. The reward function is the negative distance to the target state O. Based this reward function, the ground truth Q-function is calculated and depicted in Figure b(). To evaluate the performance of different methods, we generate an expert dataset and a medium dataset, each containing 500,000 transitions. The expert dataset is constructed by perturbing the optimal action derived from the ground truth Q-value with small noise ~ 41[-0.05, 0.05), while the medium dataset generated by adding larger noise 0[50.5 0.5]. The score network in this toy example is implemented as a 4-layer MLP with Mish activations and 256 hidden units per layer.
For the model ensemble method, we employ 5 independently trained neural networks with identical architectures to quantify predictive uncertainty. Each model a 3-layer MLP with ReLU activations and 128 hidden units. All models trained in a supervised manner for 100 epochs using the Adam optimizer with a fixed learning rate of le-3. During inference, the ensemble estimates epistemic uncertainty by computing the normalized variance across model predictions.
23

---

Published as a conference paper at ICLR 2026
Table 4: Task-specific hyperparameter settings.
Task
Ta
Qmin
0.001
99th
99th
0.9
-366
sarncheethh-medduu--repaaa
0.001
99th
99th
0.9
-366
halicheetal-mmddiun-experryyy
0.05
80th
80th
0.7
-366
hallchheetah-<xxeert-22
0.05
80th
80th
0.7
-366
halfcheetah--rnddom-22
0.001
99th
99th
0.9
-366
hopper-
meppum--
0.001
99th
99th
0.9
-125
hopper-mediun--repaayy222
0.001
99th
99th
0.9
-125
hopper-meeduum-expeettvv2
0.05
80th
80th
0.7
-125
hopper-expert-v2
0.05
80th
80th
0.7
-125
hopper-random--22
0.001
99th
99th
0.9
-125
walker20-mcdiumm-22
0.001
99th
99th
0.9
4411
walker-dd-medium--rplla
0.001
99th
99th
0.9
471
wallker2d-modduu--eppert22
0.05
99th
99th
0.7
471
waller2dd-expert-v22
0.05
99th
99th
0.7
471
walker2ddrandom--22
0.001
99th
99th
0.9
471
pen-
nen-cloned-vl
1
60th
60th
0.7
-715
pen- ne--human-vl
20
80th
80th
0.7
-715
Ground truth Q O-function
Action
-10
10
-40
(8) 1D navigation task
(b) Ground truth Q Q-unettion heatmap
Figure 6: Toy environment and ground truth Q-function heatmap visualization.
For the MC dropout framework, we adopt a Q-network architecture consisting of 3-layer MLP with 256 hidden units and ReLU activations. Dropout layers with fixed probability of 0.1 are incor- porated to introduce stochasticity during inference. This configuration enables the model to ap- proximate Bayesian inference by maintaining dropout activation during both training and evaluation phases. The Q-network undergoes supervised training for 1,000 epochs using the Adam optimizer with consistent learning rate of 1e-3. For uncertainty quantification, performs 20 stochastic forward passes per state-action pair with dropout enabled, computing the epistemic uncertainty as the normalized variance across these Monte Carlo samples.
For the VAE-based method, we adopt a conditional VAE (CVAE) architecture to model the behavior policy distribution and quantify oui-t--distribution actions using reconstruction error. The decoder reconstructs the original action through single output head. The model is trained for 1,000 epochs with the Adam optimizer at learning rate le-3. During inference, the reconstruction error computed for actennction pairs by comparing the reconstructed action to the original input action.
B.5 EXPERIMENTAL DETAILS ON D4RL BENCHMARKS
For all MuloCo locomotion tasks, we pretrain the diffusion models for both the behavior policy and state distribution for 100,000 gradient steps using the Adam optimizer with learning rate of 3e-4 and batchsize of 1024. The dynamics models are also pretrained for 100,000 gradient steps with the same learning rate and batch size. Our algorithm is then trained for million gradient steps to ensure convergence, with policy evaluation performed every 20,000 gradient steps. Results are reported as the average normalized scores over 40 random rollouts, comprising independently trained models and 10 evaluation trajectories per model across all tasks. All experiments
24

---

Published as a conference paper at ICLR 2026
Trrhheeta--meddmms naconsttaacin accionn Erior
hallcheetah medium m-expert-22 Recorstraution Erior
State-Action
-15
(a) saffcheetaa-mediime (ID) vs. (OOD))
(OOD).
hroppee
Siate-Aciion Diatr butiooiioo
Becconad: ucllion baror
Parcor surccictiiinn Ersor
Ertor Datribat im
-30
-00
-5
28 45
Reconstrucion Eve
(b) hopper-medium- expert--xeepeeeeeeee (ID) hopper-mcdium-relayye (OOD).
Wel mollber2 Semeclut--eeppppe
wakerr
Recongtruction
Ditribution ReGSsedduun
Secomm 2
18
Reconatructon Erer
(c) walker22d-mediuu--etpay--22 (ID) vs walker2dl-medium-222 (OOD).
Figure 7: Diffusion-based reconstruction error distribution across datasets. Diffusion models were trained exclusively on in-distribution (ID) data. From left to right: t-SNE embedding of the state- action distributions; reconstruction errors of ID samples; reconstruction errors of OOD samples; and density plots of error distributions for both ID and OOD samples. The color bar indicates the magnitude of reconstruction error in the second and third columns.
ducted on four NVIDIA GeForce RTX 3090 GPUs, with each experiment taking approximately 30 hours to complete, including both training and evaluation.
C ADDITIONAL EXPERIMENTAL RESULTS
C.1 OOD DETECTION PERFORMANCE ON D4RL BENCHMARKS
To evaluate the ability of our diffusion- models to distinguish OOD samples, we conduct ex- periments on the D4RL. benchmarks, designating certain datasets as in-distribution (ID) and others OOD. Specifically. formance on the OOD datasets drawn from the same cavironment. For each dataset, we randomly sample 5,000 state-action pairs ensure balanced comparison. The reconstruction error distribu- tions for the actions are visualized via color-mapped scatter plots and histograms in Figure
pretrained diffusion models the ID datasets, and evaluated their per-
Across all environments, OOD datasets consistently exhibit significantly larger reconstruction rors compared to their ID counterparts. This pronounced diserepancy is visually evident in both the color-mapped scatter plots and the histogram plots. In the scatter plots, ID samples consis-
25

---

Published as a conference paper at ICLR 2026
hallcheetsh med.mm mel--m--mppaaaae Stata-AAdoor Disribution
baticheedah boltchhethhmmcilu--ie medium---pll-lll Original Reconssuuddn Ertor
to Pecheeca--meeiiim ayy DGD Reconsiracion
Disitbution ongiial recouraciio Ene
orgra aooendore
Eror
Recomutsuctan
(a) Noise scale = 0.5.
halfcheetah StativAciion lon Distribution
fehhcethh
fichcetthh DCD Retoneiratiion Baror
Orig ial Bac cocconniiiiinn Enror
Enroe Distr bulion cda Recrrmtacioo
(b) Noise scale 1.0.
hallchoctah modium roplay
medium restay Original Beconetr Frtor
StaledAction Distribation
DOI Recorstmetioon Frine
Frice Distribation
200 date
Recomntrrtion
Rectnemuritin Einor
(c) Noise scale 5.0.
Figure 8: Diffusion-baan based reconstruction eπror distributions on original ID datasets and synthetic OOD datasets.
Table 5: OOD detection metries synthetic OOD datasets.
Noise Scale
TP
TN
FP
FN
Accuracy
Precision
Recall
F1-Score
AUROC
0.5
2910
4957
43
2090
0.7867
0.9854
0.5820
0.7318
0.9637
1.0
4832
4957
43
168
0.9789
0.9912
0.9664
0.9876
0.9980
5.0
5000
4957
43
0.9957
0.9915
1.0000
0.9957
1.0000
tently associated with low reconstruction errors, whereas OOD samples display markedly high error values. Similarly, the histogram plots reveal a distinct shift in the error distributions between ID and OOD samples, with OOD data showing a heavier tail toward higher error values. These results strongly suggest that iirtusion-based reconstruction enror serves as robust and effective indicator for OOD detection this setting.
To provide more comprehensive quantitative analysis, we construct synthetic OOD datasets as fol- lows. We first sample 5,000 state-action pairs from the original D4RL dataset, and for each pair, we generate a corresponding OOD sample by perturbing the action with standard Gaussian noise using noise scales of 0.5, 1.0, and 5.0, respectively. We evaluate the OOD detection capability our diffusion- based reconstruction error these datasets, using the 99-th percentile of reconstruction errors computed from ID samples as the detection threshold. Based on this threshold, we report the counts of true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN),
26

---

Published as a conference paper at ICLR 2026
fch medium-rep 8DC Curve (AUROC- 0.99677
saffchetahh meduum-rolle ROC Curve LAUROC-
halfcheetah
RDC Curve (AURDC-10000
1.-
1--
022
32-
18
144
Tabe Pauitivn Rate
Table Poultive Plate
Tabe Pauitivn Rate
(a) Noise scale 0.5.
(b) Noise scale = 1.0.
(0) Noise scale 5.0.
Figure ROC curves for diffusion-based OOD detection under different noise scales.
Table Validation on OOD detection benchmarks.
KDDCUP
KDDCUP-Rev
Arrhythmia
Method
Precision
Recall
Fi
Precision
Recall
Fi
Precision
Recall
Fi
OC-SVM
0.7457
0.8523
0.7954
0.7148
0.9940
0.8316
0.5397
0.4082
0.4581
DCN
0.7696
0.7829
0.7762
0.2875
0.2895
0.2885
0.3758
0.3907
0.3815
DSEBM-r
0.1972
0.2001
0. 0.1987
0.2036
0.2036
0.2036
0.1515
0.1513
0.1510
DAGMM
0.9297
0.9442
0.9369
0.9370
0.9390
0.9380
0.4909
0.5078
0.4983
GOAD 0.9862
0.9840
0.9890
0.5200
Ours
0.9937
0.9899
0.9476
0.9144
0.9307 0.9545
0.9545
0.9545
gether with standard classification metrics including precision, recall, Fl-score, and AUROC. These results are summarized in Table and Figure 8 presents the empirical distributions and histograms of reconstruction errors for both ID and OOD samples under different noise scales, while the corre- sponding ROC curves are shown in Figure 9
The results show that diffusion- reconstruction error is highly effective for OOD action de- tection across different levels of perturbation. When the noise scale is relatively small, the method achieves high precision but moderate recall, indicating that mildly perturbed OOD actions are more difficult to detect. As the noise scale increases, both recall and F1-score improve substantially, reaching nearly perfect detection performance at large perturbations. The AUROC also increases consistently and reaches 1.0 for the largest noise setting, demonstrating that the reconstruction error provides a reliable and discriminative signal for distinguishing ID and OOD actions under challeng- ing distribution shifts.
C.2 VALIDATION ON OOD DETECTION BENCHAMRKS
To further investigate the effectiveness and generalizability of our proposed diffusion-based OOD detection mechanism beyond the reinforcement learning domain, we conducted additional exper- iments on three widely used anomaly detection benchmarks: KDDCUP, KDDCUP-Rew, and Ar- rhythmia. Following the procedure described in the main paper, we compute the OOD score of each sample using the single-step denoising reconstruction eтror produced by a trained diffusion model. We compare our approach against several faate-of-the-aee art deep learning methods, including OC-SVM (Chen et al. 2001), DCN (Jin et al.] 2021), DSEBM-r (Zhai et al. (016)). DAGMM (Zong et al. 2018), and GOAD (Bergman & Hoshen 2020). Our experimental setup follows GOAD, and the baseline results are taken directly from the respective original publications.
Table 6 summarizes the precision, recall, and Fl-scores across all benchmarks. The results demon- strate that our diffusion-based approach consistently achieves high detection accuracy and outper- forms or matches existing baselines across all datasets. This robust performance further validates the diffusion model's 's superior capability in modeling the in in-distribution data manifold and confirms the reliability of using the reconstruction error as a general indicator for OOD detection, even across varying data distributions and task contexts.
27

---

Published as a conference paper at ICLR 2026
Diffusion Error vs GMM NLL
Diffusion Error vs CVAE NLL Correlation 0.7848
Correlation 0.9718
GMM True Negative Log-Likellnod (NLL)
1.0
CVAE Negative Log-Likelihood (NLL)
1.0
0.8
0.8
0.0
0.8
0.4
0.4
0.2
0.2
0.0
0.0
0.2
Diffusion-based Reconstruction Error
Diffusion-based Reconstruction Error
(a) Diffusion-hase reconstruction error vs. true NLII on GMM dataset.
(b) Diffusion-hasee reconstruction error vs. CVAE-
based NIJL on D4RI, dataset.
Figure 10: Correlation analysis between diffusion- based reconstruction error and negative log- likelihood (NLL).
C.3 QUANTITIVE ANALYSIS BETWEEN DIFFUSION-BASED RECONSTRUCTION ERROR AND NEGATIVE LOG-LIKELIHHON (NLL)
To further examine whether diffusion-based reconstruction error serves as a meaningful proxy for likelihood estimation, conduct quantitative correlation analyses both synthetic and D4RL. datasets. Scatter plots illustrating the relationship between diffusion reconstruction error and nega- tive log-likelihood (NLL) are shown in Figure T0
We first validate the relationship in Gaussian mixture setting, where the ground-truth density is analytically available. Specifically, we construct four-component symmetric Gaussian mixture and uniformly sample 10,000 points. Using diffusion model trained on this distribution, we compute the reconstruction error for each sample and compare against the true NI.I. derived from the underlying GMM. The result indicates a very strong Pearson correlation (p 0.9718), confirming that the diffusion- based reconstruction error is highly consistent with the true likelihood when the underlying distribution is accurately modeled.
We further evaluate this relationship on the sulfichetaa-mmdiiu--eeeee behavior policy density is not directly accessible. To approximate the behavior support likelihood, we train a CVAE on the dataset as a reference model and compute NLL as a surrogate likelihood estimator. The diffusion reconstruction eтrоr again exhibits strong positive correlation with the CVAA--bsee NLL (p = 0. 78.888). indicating that reconstruction error retains a substantial degree of statistical consistency with likelihood However, since CVAES are known to struggle with accurately modeling multi-modal behavior dis- tributions, the resulting NLL values from the reference model may introduce estimation bias and should therefore be interpreted as only rough approximation for the true likelihood.
dataset, where true
in high-dimensional continuous control environments.
C.4 ACTION TYPE PROPORTIONS DURING POLICY OPTIMIZATION
To gain deeper insight into how the action distribution induced by the learned policy evolves over time, track the proportions ID, beneficial OOD, and detrimental OOD actions throughout the training process. At each training iteration, the statistics are calculated over a sampled batch of size 256. We present the results for three halfcheetah tasks in Figure
Across all datasets, the proportion of ID actions consistently increases as training progresses, accom- panied by corresponding decline in the overall proportion of OOD actions. This trend suggests that the learned policy progressively aligns more closely with the behavioral support during optimization. However, the relative magnitudes of the ID proportions vary considerably across the three datasets.
28

---

Published as a conference paper at ICLR 2026
ba fuftheeioo-neddur myd ay
1D Action Proportion
Benefidal 0GD Adtion Proportion
Detrimental OGD Action Proportion
Benefida// Detrieets 000 Rabo
2.7
www.
Mag w/y
14-
Hornazzed
64-
24-
12
VMd Mo
8.0
3.0
0.5
0.6
0.6
10
30
04
1.0
00
Gradien: Staps (xbet)
Gusient Stepa
Gradient Stops
Grad
Figure 11: Proportions of different action types during policy optimization.
distribution halfchectah www.eetta-mdelurr Denettoil Oo0 Actions In 24844 Dennimenta 3OD Artoon 653)
actions Ino 1854)
Oraa us Didtrbe
Detenveal DE0
1200 demeenn
Figure 12: Q-value distributions for different action types.
Specifically, the medium-replay dataset exhibits the largest action ratio, followed by the medium dataset, whereas the medium-cxpert dataset yiclds the lowest ID proportion. We also visualize the ratio beneficial to detrimental OOD actions in the rightmost column. For both the medium-replay and medium datasets, the proportion of beneficial OOD actions substantially exceeds that of detri- mental ones. Conversely, on the medium-expert dataset, beneficial and detrimental OOD actions appear in almost equal proportion.
These differences can be attributed to the inherent characteristics of the datasets. The medium-cxpert dataset has relatively narrow support concentrated around near-optimal trajectories. As result, the learned policy more frequently generates actions that fall outside this narrow support, leading higher overall OOD proportion. Despite this, since the dataset already contains expert demon- strations, the potential performance gain from extrapolation is limited, leading to only a comparable proportion of beneficial and detrimental OOD actions. In contrast, the medium-replay and medium datasets exhibit more diverse distributions of generally suboptimal behaviors. This broader support enables the learned policy to benefit from moderate extrapolation, where slight deviations outside the data manifold can lead to meaningful performance improvements, which is consistent with our empirical results.
C.5 VISUALIZZATIO OF Q-VALUE DISTRIBUTION
To investigate whether beneficial OOD actions indeed lead the policy toward higher-value regions, visualize the learned Q-value landscape. Specifically, we randomly sample 10,000 states from the offline dataset. For each evaluation state, we generate categorize it as an ID action, a beneficial OOD action, or a detrimental OOD action based on the dif- fusion reconstruction error. We then apply t-SNE to embed the corresponding state-action pairs into a two-dimensional space, where each point colored according to its Q-value estimate. In addition, plot the Q value distributions for the three categories to enable a direct statistical comparison.
action using the learned policy and
As illustrated in Figure 12. beneficial OOD actions exhibit a clearly right-shifted Q-value distribu- tion, indicating that the actions identified as beneficial by our method correspond to regions where the critic consistently predicts higher returns. In contrast, detrimental OOD actions predominantly occupy the low-value region, with their Q-value distribution concentrated near the lower tail. The critic assigns persistently low values to these actions, implying that they are unlikely to yield perfor- mance gains and should therefore be suppressed during policy improvement.
29

---

Published as a conference paper at ICLR 2026
halfchettah-meiumm22
balfchhetah-mmedllu-rreeaaxx
Normal bed Score
50-
40-
30
20
20
10.
10-
DOSBR Saynamis-200
0.0
0.2
0.8
1.0
0.0
0.2
0.4
0.6
1.0
Gradient Steps (x1e6)
Gradient Steps (x1e6)
Figure 13: Sensitivity analysis of the dynamics model error.
C.6 ADDITIONAL SENSITIVITY ANALYSIS
C.6.1 DYNAMICS MODEL ERROR
To evaluate how the accuracy of the learned dynamics model influences the performance of OOD action classification, we conduct an additional ablation study. Specifically, we pretrain the dynamics model for 100k gradient steps and save the intermediate checkpoints at 10k and 20k steps. These carly-stage models exhibit substantially higher prediction error compared to the final checkpoint, providing controlled mechanism to examine the impact of model inaccuracies. In the subsequent experiments, we replace the fully trained dynamics model with the selected checkpoint while keep- ing all other components unchanged. We evaluate these variants two halfcheetah datasets, with the corresponding training curves illustrated in Figure
Our results indicate that employing dynamics models derived from the early checkpoints consis- tently deteriorates policy performance relative to the fully trained model. Furthermore, on the maalichheta-mmdiu---rrpppaa medium-replay--2 dataset, we observe that when the dynamics model poorly trained, the performance may fall below that of the DOSER w/o AC and VC variant introduced in Section 4.3 which intentionally excludes dynamics modeling. This finding highlights that if the dynamics model fails to produce reliable next-state predictions, the resulting misclassification of OOD actions can be more detrimental overall performance than omitting OOD action classification entirely.
In addition, the observed performance gap between the early-stage and fully trained dynamics mod- els provides further evidence regarding the model's ability to generalize beyond the dataset support. This indicates that the final checkpoint captures meaningful structural regularities of the environ- ment rather than merely memorizing in-distribution transitions. As result, a well-trained dynamics model can provide sufficiently reliable predictions for moderate OOD actions, which is crucial for the selective regularization mechanism of DOSER.
C.6.2 THE NUMBER OF CRITIC NETWORKS
In our main experiments, employ four critic networks for Q-function learning. This design choice follows the implementation of SVR, upon which our training pipeline is partially built. The use of multiple critics has been shown to reduce overestimation bias and stabilize value learning. To examine whether this choice confers any unintended advantage, we perform additional ablation which DOSER is trained with only two critic networks while keeping all other components and hyperparameters fixed. We evaluate both settings the seathhettta-mmiiuma- and halfchectah- medium-replay-v2 datasets, the corresponding learning curves are presented in Figure [4
Empirically, we observe that using two critics achieves comparable final performance to the four- critic setting across both tasks, with only a slight difference within an acceptable range. This indi- cates that DOSER does not noly rely the increased critic ensemble size obtain its performance gains. While additional critics can enhance robustness during training, the core algorithmic contributions of DOSER remain effective under the standard two-critic setup commonly used in offline RI.
30

---

Published as a conference paper at ICLR 2026
halfchettah-meiumm22
halfchhetah-mmellum-rreeaaxx
Normal bed Score
50-
30
30-
20
10.
critic (DOSER) critic
10
c-cricc IDOSER)
crbc
0.0
0.2
1.0
0.2
0.4
0.6
1.0
Gradient Steps (x1e6)
Gradient Steps (x1e6)
Figure 14: Sensitivity analysis of the number critic networks.
haifcheetah mediumm-2
mallcheetah-mmeiimm-eee replay
70
60-
60-
Nomallzed Score
50-
40
20-
0-03
--03
4.0
9-
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Gradient Steps (x1e6)
Gradient Steps (x1e6)
Figure 15: Sensitivity analysis of the compensation target weight
C.6.3 COMPENSATION TARGET WFIGHT 7
We set the default value of the compensation target weight 17 to 0.9 in all main experiments. This hyperparameter controls the weight of the target Q-value of beneficial OOD actions, smaller 1) re- duces the extent of compensation and makes DOSER more conservative. To evaluate the sensitivity of DOSER to this hyperparameter, we conduct an ablation study by varying 7) E {0.8, 1.0} while keeping all other components unchanged.
As shown in Figure 15| DOSER maintains consistently stable performance across this range. How- ever, when setting 7 = 1.0, observe slight degradation in performance halfcheetab-mmddumm replay-v2. This is primarily due to mild value overestimation introduced by fully adopting the target Q-values of beneficial OOD actions without discounting. Overall, the default choice 7) 0.9 effec- tively mitigates such overestimation while still enabling meaningful policy improvement.
C.6.4 THE NUMBER OF SAMPLED IN-DISTRIBUTION ACTIONS N
In the OOD action classification stage, DOSER estimates the optimal in-distribution action by sam- pling N candidate actions from the offline dataset and selecting the one with the highest Q-value as the reference. To assess the robustness of DOSER to the chioce of N, we conduct experiments on the halfcheetah tasks with N (5, 10, 20).
The results in Figure 16 indicate that DOSER maintains strong performance across different values of N. larger N provides more accurate approximation the optimal ID action but comes with increased computational cost, whereas small N may introduce randomness in the estimation. Since the optimal ID Q-value used only to construct an optimistic Q-target that guides beneficial OOD actions toward higher-value regions, DOSER does not rely heavily the precise accuracy of this estimate. Therefore, we choose N = 10 as a reasonable trade-off between computational efficiency and estimation accuracy in our main experiments.
31

---

Published as a conference paper at ICLR 2026
halfcheetah-medium-22
balfchhetah-meellu--reeaaaa
60
Normalbod bedl Score
50-
40-
40-
s0-
30-
10
100
0.0
0.2
0.6
1.0
0.0
0.2
0.4
0.6
10
Gradient Steps (x1e6)
Gradient Steps (x1e6)
Figure 16: Sensitivity analysis of the number of sampled in-distribution actions N.
haffchettah-medumm22
baffchettah-meddiu--rreaaaxz
Nomrllized Score
50-
40-
20-
30-
20
10-
Ona 366(D0SER)
10
366]DOSER)
Own -
Q.e
Gradient Steps (x1e6)
Gradient Steps (x1e6)
Figure 17: Sensitivity analysis of the value of Qiin-
C.6.5 Q-VALUE LOWER BOUND Qmin
DOSER employs a lower bound Qmin when penalizing detrimental OOD actions. In our main experiments, this value is not treated as a tunable hyperparameter. Instead, it is derived directly from the environment dynamics as Qmin Ruin which corresponds to the standard minimum achievable I-7 return under the given discount factor For the halfcheetah environment, setting to 0.99 yields Qmin -666. To further examine the impact of this parameter, we conduct an ablation study in which Quin is set to 0 while keeping all other components unchanged. This alternative setting corresponds to less conservative penalty on potentially detrimental OOD actions. We evaluate this variant on the halfcheetah tasks, with the resulting learning curves reported in Figure 17
The results show that replacing the original value Qmin with leads to performance degradation across both datasets. This outcome can be attributed to the fact that higher lower bound reduces the penalization applied to detrimental OOD actions, thereby weakening the mechanism designed to mitigate value overestimation. Nevertheless, even with this suboptimal setting, the overall perfor- mance remains competitive with existing offline RL baselines.
C.7 ENSEMBLE-GUIDED GATING MECHANISM
We further introduce an ensemble-gen guided uncertainty gating mechanism on top the learned dy- namics model, which is designed to prevent unreliable next-state predictions from influencing the classification of OOD actions. We construct an ensemble of K = 5 independently initialized dy- namics models, each trained on the original offline dataset. For any state-action pair (s, a), the ensemble produces multiple next-state predictions {&/, &/,....... 3. We calculate the prediction variance across ensemble members measure of epistemic uncertainty:
K
Var(s')
Σ k=1
8 - 8'112. 2
where s'
Σ k=
(74)
K
K
32

---

Published as a conference paper at ICLR 2026
halfchettah-meiumm22
balfchhetah-meellu--reeaaaa
Normal bed Score
50
50
40-
30-
20-
10-
Gusted
0.0
0.2
0.8
1.0
0.2
0.4
0.6
1.0
Gredient Steps (x1e6)
Gradient Steps (x1e6)
Figure 18: Comparison of DOSER with and without ensemble-guiee guidded gating.
halfcheotah- medi um um-reppay-v2
haffcheatah-mediume
ID Action Proportion
B Beneficial O0D Action Proportion
Confident Action Proportion
M
Nomallzze Proportion
0.4
0.4
0.6
0.0
02
0.4
0.8
1.0
0.0
0.5
0.0
Action Proportion
Detrimental Proportion
Uncertain Proportion
1.0
Normalbzed Propartion
0.8
0.8
0.6
0.2
0.4
0.6
1.0
0.0
0.2
0.4
0.8
1.0
0.0
0.5
0.8
Gradient Stops (x116)
Gradiont Steps (x1e6)
Gradient Stops (x
Figure 19: Proportions of different action types with ensemble-guided gating.
To determine whether predicted next state is reliable, estimate the empirical distribution of these variances on the offline dataset and use the 99-th percentile reliability threshold Tvar- Only when the prediction variance for an OOD action falls below this threshold do we trust the predicted next state and apply the value-based beneficial/deerimentta classification; otherwise, the action is conservatively categorized detrimental.
However, Figure 18 demonstrates that incorporating this ensemble-guiee guided gating mechanism into DOSER brings no noticeable performance improvement in the halfcheetah environments. We further analyze the action type proportions during training (Figure 19). defining confident actions as those with prediction variance below the reliability threshold, and uncertain actions as those filtered out by the gate. The results indicate that fewer than 5% of actions exceed the threshold and are consequently filtered out.
This result again suggests that the pretrained dynamics model already generalizes reasonably well to the moderately OOD regions. also possible that the chosen ensemble size of 5 is insufficient to fully capture epistemic uncertainty, and larger more expressive ensembles might provide stronger gating effects. We leave the exploration more sophisticated uncertainty quantification methods to future work.
33

---

Published as a conference paper at ICLR 2026
Table 7: Additional performance comparison on Gym-MuJoCo expert and random datasets. We report the mean and standard deviation over seeds for DOSER.
Dataset
BC
BCQ
BEAR
DT
AWAC
OneStep
TD3+BC
CQL
IQL
DMG
DOSER (Ours)
halfchectab-e
92.9
89.9
92.7
87.7
81.7
96.7
96.3
95.9
0.6
hopper-e
110.9
109.0
54.6
94.2
109.5
106.9
107.8
96.5
109.4
111.5
111.6
0.5
walker2d-e
107.7
106.3
106.6
108.3
110.1
110.7
110.2
108.5
109.9
114.7
111.2
balicheetall-r
2.6
2.2
2.3
2.2
6.1
11.0
17.5
13.1
28.8
32.8
1.5
hopper
4.1
7.8
3.9
5.4
9.2
8.5
7.9
20.4
31.2
0.1
walker2d.r
1.2
4.9
12.8
2.2
0.2
1.6
5.1
4.8
2.3
Average
C.8 ADDITIONAL. EXPERIMENT RESULTS ON D4RL. BENCHAMRK
To further validate DOSER's ' performance across wider range of dataset qualities, conduct ad- ditional experiments on the Gym-MuJoCo expert and random datasets, as shown in Table 7 Across the expert datasets, DOSER achieves competitive performance relative prior offline RL meth- ods. More notably, on the random datasets, where the behavior data is highly suboptimal, DOSER exhibits stronger performance, indicating its robustness under poor- poor-quality offline data.
C.9 LEARNING CURVES
Learning curves on D4RL tasks are provided in Figure 20 Figure 21 and Figure 22 The curves averaged over 4 random seeds, with the shaded area representing the standard deviation across seeds.
D THE USE OF LARGE LANGUAGE MODELS (LLMs)
We acknowledge the assistance of GPT-5 in proofreading and polishing the manuscript. The authors bear full responsibility for the content and presentation this paper.
34

---

Published as a conference paper at ICLR 2026
halfcheetah-mediium-22
hallchhettahmmediiu-meeaaaay
halichetta--msdium--eeerrr
70
100
50-
40
60
30
DOSER we AC VC
0-
DOSER
10-
0-
0.2
0.0
hopper-medium-v2
hopper-medium-rellaye
hopper-medium-exprtt
100
Normalized
60-
40
40
20
0-
0.0
0.2
0.6
0.8
1.0
0.2
0.4
1.0
0.0
0.4
0.6
0.8
walker2ddmediimg
walker2d walleer2d-mdduum--ppaaaauu
walkerz2d-medlumeexper
120
100
60
80-
Normalized
40
60
40
20
20-
20
0
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Gradlent Steps (x1e6)
Gradlent Steps (x1e6)
Gradient Steps (x1e6)
Figure 20: Learning curves of the component ablation study on Gym-MuJoCo tasks.
pen-human-v1
pen-cloned-ve
120
100
Normalized
20
40
0.8
1.0
0.0
0.6
0.8
1.0
Gradient Steps (x1e6)
Gradient Steps 1e6)
Figure 21: Learning curves Adroit tasks.
35

---

Published as a conference paper at ICLR 2026
halfcheetah--xpeertv2
hopper-expert-v2
walker2d-experttv22
100
Normalized Score
80
60
Norralized
60
M
40
0.4
1.0
0.0
0.6
0.2
0.6
Gradient Steps (x1e6)
Gradient Steps (x1e6)
Gradient Steps (x1e6)
halfchhettahranddmm-22
hopper-random--2
walkerr2draanoomw22
17.5
15.0
34
12.5
10.0
32
2.5
28
0.0
0.2
Steps (x1e6)
Gradient Steps (x1e6)
Gradient Steps (x1e6)
Figure 22: Learning curves on Gym-MuloCo expert and random tasks.
36


Reviews:
[{"review_id": "VSCChDB0Jk", "reviewer": "Reviewer_zxuo", "summary": "This paper proposes DOSER (Diffusion-based OOD Detection and Selective Regularization), an offline reinforcement learning (RL) framework that aims to mitigate overestimation caused by out-of-distribution (OOD) actions.\nThe method trains two diffusion models to model the behavior policy and the state distribution, respectively, and uses the single-step denoising reconstruction error as an OOD indicator.\nThen, during policy optimization, DOSER further classifies OOD actions into “beneficial” and “detrimental” using a learned dynamics model and value comparison, selectively penalizing or rewarding them.\nThe authors provide theoretical analysis (γ-contraction and bounded convergence) and empirical results on D4RL MuJoCo benchmarks, claiming consistent improvements over existing VAE-based and diffusion-based offline RL baselines.", "questions": "How stable is the beneficial vs. detrimental classification in practice? Can the authors report the proportion of actions identified as each type and the accuracy of this classification on a synthetic dataset where ground-truth OOD actions are known?\n\nDoes the diffusion reconstruction error correlate quantitatively with true behavior support likelihood (e.g., log-likelihood from a reference model)?\n\nHow sensitive is DOSER to errors in the learned dynamics model? Would performance degrade sharply if the model mispredicts next states for unseen regions?\n\nCould the authors show qualitative visualizations (e.g., state-action heatmaps) to verify that “beneficial” OOD actions indeed lead to higher-value regions?\n\nHave the authors tested on harder datasets (Adroit, AntMaze) to demonstrate generalization beyond MuJoCo locomotion?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Interesting integration of diffusion models into OOD detection. The idea of using denoising reconstruction error as a likelihood-free OOD signal is conceptually appealing and avoids Gaussian assumptions used in VAE-based behavior modeling.\n\nSelective regularization is an intuitive extension to uniform penalization — encouraging “good” OOD exploration rather than blindly suppressing all deviations.\n\nClarity and structure: the paper is generally well written and easy to follow, with clear diagrams (e.g., Fig. 1–2) and algorithmic pseudocode.\n\nComprehensive empirical comparison: the authors evaluate against both conventional (CQL, IQL, ACL-QL) and diffusion-based (DQL, SRPO, DTQL, etc.) baselines, and perform ablation and sensitivity studies.", "weaknesses": "Questionable reliability of OOD classification.\nThe core contribution—distinguishing beneficial from detrimental OOD actions—relies heavily on a learned dynamics model and single-step value predictions.\nIn offline RL, both are notoriously unreliable outside the data distribution, making this classification potentially unstable or self-reinforcing. The paper lacks evidence (e.g., accuracy, calibration, or failure cases) showing that such classification can be trusted.\n\nCircular dependency between detection and evaluation.\nThe method uses diffusion-based reconstruction error to decide OODness and then uses a dynamics model (also trained on the same data) to judge whether the OOD transition is beneficial.\nWithout ground truth labels or uncertainty calibration, this process risks systematic bias amplification rather than true OOD discrimination.\n\nLimited novelty beyond combining existing ideas.\nDiffusion-based behavior modeling (e.g., DQL, IDQL, SRPO) and reconstruction-based OOD detection (e.g., Graham et al., CVPR 2023) are established.\nThe proposed framework largely combines these with a hand-crafted “bonus/penalty” heuristic; the theoretical part simply restates a standard contraction argument.\n\nExperimental validation does not fully support the claims.\nThe performance gains on D4RL are moderate and sometimes within variance ranges; ablations do not directly demonstrate that the OOD classification truly identifies beneficial cases.\nNo visualization, quantitative detection metrics (e.g., AUROC on synthetic OOD test), or robustness analysis are provided to substantiate the detection accuracy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XnyjG1ifx7", "reviewer": "Reviewer_DqCJ", "summary": "This paper tackles a key challenge in offline reinforcement learning: the overestimation of out-of-distribution (OOD) actions. The authors argue that existing methods, which uniformly penalize all OOD actions, are overly pessimistic and suppress potentially \"beneficial\" explorations.\nThe paper introduces DOSER (Diffusion-based OOD Detection and SElective Regularization), a novel framework that moves beyond uniform penalization. DOSER's contributions are twofold:\n1. Diffusion-based OOD Detection: It employs two diffusion models, trained on in-distribution actions $\\hat{\\pi}_{\\beta}(a|s)$ and states $d(s)$, respectively. The single-step denoising reconstruction error ($\\mathcal{E}_a$ and $\\mathcal{E}_s$) is used as a high-fidelity metric to detect OOD samples, which the authors argue is superior to VAEs for capturing complex, multi-modal behavior.\n2. Selective Regularization: When the learning policy generates an OOD action $a_{ood}$, DOSER does not immediately penalize it. Instead, it uses a learned dynamics model to predict the next state. It then classifies $a_{ood}$ as detrimental and beneficial. While the detrimental actions are heavily penalized in the Q-function loss, the beneficial actions are compensated in the Q-function loss with a value-difference bonus.\n\nThe authors provide a theoretical analysis showing the DOSER operator is a $\\gamma$-contraction and results in a value estimate that is bounded by the in-sample value plus the beneficial bonus ($\\le Q_{In}^{\\pi}(s,a_{id}^{*})+\\eta\\delta_{V}$). Empirically, DOSER achieves state-of-the-art (SOTA) performance on D4RL benchmarks, particularly on suboptimal \"medium\" and \"medium-replay\" datasets.", "questions": "1. **On the Dynamics Model's Reliability**: Could the authors comment on the \"Extrapolation Paradox\"? How can the algorithm trust the predicted next state $s'$ (and its resulting $V(s')$ and $\\mathcal{E}(s')$) when it comes from an OOD action $a_{ood}$ that the dynamics model $p_{\\psi}$ has never seen? Have the authors considered using uncertainty estimates from $p_{\\psi}$ (e.g., via an ensemble) to gate this classification?\n2. **On Practical Hyperparameter Setting**: The reliance on per-task OOD thresholds (80th vs 99th percentile) shown in Table 4 is a major concern. How would a practitioner set $\\tau_a$ and $\\tau_s$ for a new, unseen offline dataset without a massive tuning budget? Is there a more robust or adaptive way to set this threshold?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. **Conceptual Novelty**: The primary strength is the original idea of \"selective regularization.\" By classifying OOD actions into beneficial ($\\mathcal{A}_{ood}^{+}$) and detrimental ($\\mathcal{A}_{ood}^{-}$) sets, the paper moves beyond the field's preoccupation with uniform pessimism. This is a significant and valuable conceptual contribution.\n2. **Modern OOD Detection**: The use of diffusion model reconstruction error as an OOD metric is a strong, modern approach. The argument that it is better suited for multi-modal behavior distributions than VAE-based methods is compelling and well-motivated.\n3. **Strong Empirical Results**: The algorithm achieves SOTA performance, especially on the challenging and more realistic \"medium\" and \"medium-replay\" datasets. This suggests the method is particularly effective at improving upon suboptimal, heterogeneous data.\n4. **Clear Ablations**: The ablation study in Table 2 provides clear evidence that both the Action Classification (AC) and Value Compensation (VC) components are critical to the algorithm's success. This validates the paper's core design.", "weaknesses": "1. **The Extrapolation Paradox of the Dynamics Model**: The framework's core classifier (Definition 1) relies on a learned dynamics model to predict the consequence of an OOD action. This is a critical flaw. The model is trained on in-distribution data and has no ground-truth supervision for OOD actions. Therefore, its prediction is an extrapolation that is just as likely to be erroneous as the $Q$-function's. The algorithm relies on an unreliable, \"hallucinating\" component to make its most critical safety decision.\n2. **Extreme Hyperparameter Sensitivity**: The algorithm is not a general, robust method but a \"brittle\" framework that requires massive, task-specific tuning. Appendix Table 4 reveals that key hypermeters (OOD thresholds $\\tau_a, \\tau_s$, penalty $\\beta$, expectile $\\tau$) are different for nearly every single environment-dataset pair. For example, $\\tau_a$ is 99th percentile for halfcheetah-medium but 80th for halfcheetah-medium-expert. This lack of generality is a severe practical limitation and makes the SOTA claims less impressive, as they are a product of extensive tuning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zNuI5rduP1", "reviewer": "Reviewer_AWU7", "summary": "This paper proposes the DOSER method, a novel framework that goes beyond uniform penalization. DOSER uses two diffusion models to learn the behavior policy and state distribution. Meanwhile, it uses a single-step denoising reconstruction error to detach the OOD action.", "questions": "* Similar to the motivation in this paper, there are numerous other methods for achieving efficient policy optimization through action discrimination. For instance, from the perspective of policy constraints, the latest offline RL method A2PR[1] employs advantage functions to select actions that guide policy constraints toward optimization. Could you elaborate on the advantages of DOSER? Additionally, could you include experimental comparisons and discussions with A2PR?\n\n* I'm curious about the trend in the proportion of beneficial OOD actions versus detrimental OOD actions during training for different tasks. Does this significantly impact the training results?\n\n* There are too few main experiments. Could we add some navigation tasks, such as Antmate?\n\n\n\nReference:\n\n[1]Liu, T., Li, Y., Lan, Y., Gao, H., Pan, W., & Xu, X.. Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning. In International Conference on Machine Learning (pp. 31406-31424). PMLR.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* DOSER further distinguishes between beneficial and detrimental OOD actions by evaluating predicted transitions, selectively suppressing risky actions while encouraging exploration of high-potential ones.\n* DOSER establishes stable convergence with bounded value estimates under $\\gamma$-contraction guarantees.", "weaknesses": "* This method needs to train two diffusion models, which may bring more computational burden.\n* This method also trains a dynamic model and multiple samples, which may bring more computational burden.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l4tqxjh46M", "reviewer": "Reviewer_9PZb", "summary": "The proposed DOSER framework addresses limitations in offline reinforcement learning by using diffusion models to detect out-of-distribution (OOD) actions based on reconstruction error. It then selectively regularizes these actions, suppressing detrimental ones while encouraging those deemed beneficial for exploration.", "questions": "Please see Weakness", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is clearly written and well-structured, making its central contributions easy to understand.\n2. The proposed method achieves competitive performance on Gym domain.", "weaknesses": "1. The method's design is notably complex, incorporating multiple auxiliary models beyond the standard actor-critic setup (e.g., diffusion model for state, diffusion model for action, ensembled dynamics models, implicit value model, implicit target value model, Two extra critic networks). This design challenges the method's practicality and scalability for real-world applications.\n2. The method introduces a large set of algorithm-specific hyperparameters ($ \\tau_s, \\tau_a, \\tau, \\beta, \\lambda, \\eta, \\delta_{\\mathrm{V}}, N, Q_{\\min} $). The sensitivity and impact of most of these parameters on the final performance are not adequately analyzed, leaving concerns about the robustness and reproducibility of the results. For instance, the effect of perturbations in \\( Q_{\\min} \\) remains unclear.\n3. While the core idea is intuitive, the empirical performance gains appear limited according to the ablation study, with consistent improvements observed on only two tasks (halfCheetah-m and halfCheetah-mr). The proposed method is quite similar to SVR. By comparing their performances, one can draw the conclusion that the benefits are limited for other tasks as well.\n4. Could you provide the curves of the selective ratio for ID, OOD-beneficial, and OOD-detrimental actions throughout the training process?\n5. There seems to be a potential flaw in the derivation. The claim that $ \\eta Q_{\\min} \\geq Q_{\\min} $ is contradictory given the stated parameter range $ \\eta \\in [0,1) $. \n6. The provided performance bound $ Q_{\\min} \\leq Q^{\\pi} \\leq Q_{\\in}^{\\pi} + \\eta \\delta_{\\mathrm{V}} $ is arguably trivial, as it essentially states that the value is bounded by some minimum and maximum. A non-trivial bound with asymptotic analysis, characterizing the error relative to the optimal policy, is needed for a meaningful theoretical contribution.\n7. The use of four critic networks, compared to the two commonly used in general methods, may confer an unfair advantage. An ablation study demonstrating performance with only two critics is necessary to ensure an equitable comparison and to isolate the contribution of the core algorithmic innovations.\n8. The evaluation would be more comprehensive with the inclusion of the AntMaze domain.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)