Q-Distribution guided Q-learning for offline reinforcement learning: Uncertainty penalized Q-value via consistency model
Authors: Jing Zhang
Linjiajie Fang
Kexin Shi
Wenjia Wang
Bingyi Jing
Venue: NeurIPS 2024 poster (2024)
Keywords: offline Reinforcement Learning
Q value distribution
Uncertainty measure
pessimistic Q learning
Consistency model
Primary area: reinforcement_learning
TL;DR: Overcome the too conservative problem in pessimistic Q learning in offline RL by pessimistic Q value by uncertainty via consistency model.

Abstract: ``Distribution shift'' is the primary obstacle to the success of offline reinforcement learning. As a learning policy may take actions beyond the knowledge of the behavior policy (referred to as Out-of-Distribution (OOD) actions), the Q-values of these OOD actions can be easily overestimated. Consequently, the learning policy becomes biasedly optimized using the incorrect recovered Q-value function. One commonly used idea to avoid the overestimation of Q-value is to make a pessimistic adjustment. Our key idea is to penalize the Q-values of OOD actions that correspond to high uncertainty. In this work, we propose Q-Distribution guided Q-learning (QDQ) which  pessimistic Q-value on OOD regions based on uncertainty estimation. The uncertainty measure is based on the conditional Q-value distribution, which is learned via a high-fidelity and efficient consistency model. On the other hand, to avoid the overly conservative problem, we introduce an uncertainty-aware optimization objective to update the Q-value function. The proposed QDQ demonstrates solid theoretical guarantees for the accuracy of Q-value distribution learning and uncertainty measurement, as well as the performance of the learning policy. QDQ consistently exhibits strong performance in the D4RL benchmark and shows significant improvements for many tasks. Our code can be found at <code link>.

Q-Distribution guided Q-learning for offline reinforcement learning: Uncertainty penalized Q-value via consistency model
Jing Zhang HKUST
Linjiaji Fang HKUST
Kexin Shi
HKUST
uzhanggy@connet..ue ust.
Ifangad@comecct.ug ust hk
kshiaf@connect... ust hk
Wenjia Wang" HKUST (GZ.
wenjiawangehkust--e edu.
Bing- Yi Jing" SUSTech
jingby@sustech.cm edu.
Abstract
"Distribution shift" is the main obstacle to the success offline reinforcement learning. A learning policy may take actions beyond the behavior policy's 's knowl- edge, referred to as Out-of-Distribution (OOD) actions. The Q-values for these OOD actions be easily overestimated. As result, the learning policy is biased by using incorrect Q-value estimates. One common approach to avoid Q-value overestimation make pessimistic adjustment. Our key idea is penalize the Q-values of OOD actions associated with high uncertainty. In this work, we propose O-Distribution Guided Q-Learning (QDQ), which applies pessimistic adjustment to Q-values in OOD regions based on uncertainty estimation. This uncertainty measure relies on the conditional Q-value distribution, learned through a high-fidelity and efficient consistency model. Additionally, to prevent overly optimization objective
conservative estimates, we introduce an uncertainty-awer for updating the Q-value function. The proposed QDQ demonstrates solid theoret- ical guarantees for the accuracy of Q-value distribution learning and uncertainty measurement, as well the performance of the learning policy. QDQ consis- tently shows strong performance on the D4RL benchmark and achieves significant improvements across many tasks.
Introduction
Reinforcement learning (RL) has seen remarkable success by using expressive deep neural networks to estimate the value function or policy function [1. However, in deep RL optimization, updating the Q-value function or policy value function can be unstable and introduce significant bias (2). Since the learning policy is influenced by the Q-value function, any bias in the Q-values affects the learning policy. online RL, the agent's interaction with the environment helps mitigate this bias through reward feedback for biased actions. However, in offline RL. the learning relies solely data from a behavior policy, making information about rewards for states and actions outside the dataset's distribution unavailable.
"Corresponding authors: Bing- Yi Jing and Wenjia Wang.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

It is commonly observed that during offline RL training, backups using OOD actions often lead to
target Q-values being overestimated [3] (see Figure Ja)) As a result, the learning policy tends to prioritize these risky actions during policy improvement. This false prioritization accumulates with each training step, ultimately leading failure in the offline training process 4 6) Therefore, addressing Q-value overestimation for OOD actions is crucial for the effective implementation of offline reinforcement learning.
value fucition for the behsvior palicy and eesnnng policy
2 vélue fucntion for the behavior policy and learning palicy
0s.1 of benwker pelry 0g.1 offeaning palkc
0 va us behastor pollyy
va us inseample eamiig
QILI
(a)
(b)
(a) The maximum of the estimated Q-value often occurs in OOD actions due to the instability of the offline RL backup process and the "distribution shift" problem so the Q-value of the learning policy (yellow line) will diverge from the behavior policy's 's action space (blue line) during the training. (b) The red line represents the optimal Q-value within the action space of the dataset, while the blue line depicts the Q-value function of the behavior policy. The gold line corresponds to the Q-value derived from the in-sample Q training algorithm, showcasing distribution constrained by the behavior policy. On the other hand, the green line illustrates the Q-value resulting from more conservative Q training process. Although it adopts lower values in OOD actions, the Q-value within in-distribution areas proves excessively pessimistic, failing to approach the optimal Q-value.
Figure 1:
Since any bias or error in the Q-value will propagate to the learning policy, it's crucial to evaluate whether the Q-value is assigned OOD actions and to apply pessimistic adjustment address overestimation. Ideally, this adjustment should only target OOD actions. One common way to identify whether the Q-value function updated by OOD actions is by estimating the uncertainty of the Q-value 4 in the action space. However, estimating uncertainty presents significant challenges, especially with high-capacity Q-value function approximators like neural networks [4). If Q-value uncertainty is not accurately estimated, penalty may be uniformly applied across most actions (7]. hindering the optimality of the Q-value function.
3 13 [4] attempt to make pessimistic estimates of the Q-value function, most have not effectively determined which Q-values need constraining how to pessimistically estimate them with reliable and efficient uncertainty estimates. As a result, previous methods often end up being overly conservative their Q-value estimations [15 or fail achieve tight lower confidence bound of the optimal Q-value function. Moreover, some in-sample training 19 of the Q-value function may lead it closely mimic the Q-value of the behavior policy (see Figure [b)), rendering it unable to surpass the performance of the behavior policy, especially when the behavior policy sub-optimal. Therefore, in balancing Q safety for learning and not hindering the recovery of the most optimal Q-value, current methods tend to prioritize safe optimization of the Q-value function.
While various methods 7 9 10
16.
17 18
In this study, we introduce Q-Distribution guided Q-Learning (QDQ) for offline RL The core concept focuses estimating Q-value uncertainty by directly computing this uncertainty through bootstrap sampling from the behavior policy's Q-value distribution. By approximating the behav- ior policy's Q-values using the dataset, train high-fidelity and efficient distribution learner- consistency model 20 This ensures the quality of the learned Q-value distribution.
OOr code can be found at https://gihub..oo//waaarzzz

---

Since the behavior and learning policies share the same set of high- actions [5], we can sample from the learned Q-value distribution to estimate uncertainty, identify risky actions, and make the Q target values for these actions more pessimistic. We then create an uncertainty- aware optimization objective to carefully penalize Q-values that may be OOD, ensuring that the constraints are appropriately pessimistic without hindering the Q-value function's exploration in the in-distribution region. QDQ aims to find the optimal Q-value that exceeds the behavior policy's optimal Q-value while remaining pessimistic as possible in the OOD region. Moreover, our pessimistic approach is robust against errors uncertainty estimation. Our main contributions are as follows:
Utilization of trajectory-level data with sliding window: We trajectory-level data with sliding window approach to create the real truncated Q dataset. Our theoretical analysis (Theorem 4.1) confirms that the generated data has a distribution similar true Q-values. Additionally, distributions learned from this dataset tend to favor high-reward actions.
QDQ introduces the consistency model [20] as the distribution learner for the Q-value. Similar the diffusion model, the consistency model demonstrates strong capabilities in distribution learning. Our theoretical analysis (Theorem 4.2) highlights its consistency and one-step sampling properties, making an ideal choice for uncertainty estimation.
Introduction of a consistency model distribution learner:
Risk estimation of Q-values through uncertainty assessment: QDQ estimates the risk set of Q-values by evaluating the uncertainty of actions. For Q-values likely to be overestimated and associated with high uncertainty, pessimistic penalty is applied. For safer Q-values, mild adjustment based on uncertainty error enhances their robustness.
Uncertainty-awaar optimization objective to address conservatism: To reduce the overly con- servative nature of pessimistic Q-learning in offline RL, QDQ introduces an uncertainty-awan optimization objective. This involves simultaneous optimistic and pessimistic learning of the Q-value. Theoretical (Theorem 43 and Theorem (.4) and experimental analyses show that this approach effectively mitigates conservatism issues.
Background
Our approach aims temper the Q-values in OOD) areas to mitigate the risk of unpredictable extrapolation errors, leveraging uncertainty estimation. We estimate the uncertainty of Q-values across actions visited by the learning policy using samples from learned conditional Q-distribution via the consistency model. In this section, we provide a concise overview of the problem settings in offline RL and introduce the consistency model.
2.1 Fundamentals offline RL
The online RL process is shaped by an infinite-horizor Markov decision process (MDP): M {S, A. P. (o. 7). The state space is S, and A is the action space. The transition dynamic among the state is determined by P S x A (SS), where (SS) is the support of S. The reward determined on the whole state and action space is r S x A + R,, r < co, and can either be deterministic or random. po(so) is the distribution of the initial states so, (0, 1) is the discount factor. The goal of RL is to find the optimal policy π. : S + ((a) that yields the highest long-term average return:
(mx) Esorrp V(s)) = Egg =g 7 (s0: ao) Egg B aa
E
Fkk 8k, ag
(1)
where Q(s, a) is the Q-value function under policy т. The process obtaining the optimal policy is generally recover the optimal Q-value function, which maximizes the Q-value function over the whole space A, and then to obtain either an implicit policy (Q-Learning algorithm 21 22 23 ) or a
parameterized policy Actor-Critic algorithm [4 25 26 27 28
The optimal Q-value function Q' (s, a) can be obtained by minimizing the Bellman residual:
Ea-Paa [Q(s, a) (Q(s, a))²
(2)
3

---

where B is the Bellman operator defined as
(Q(,, := a) + =Bx-welmxxx ,(s',
a'out
However, the whole paradigm needs to be adjusted in the offline RL setting, as MDP is only determined from dataset D, which is generated by behavior policy æg. Hence, the state and action space constraint by the distribution support of DD We redefine the MDP in the offline RL setting as: MD = (Sp,Ap, PD. r. (40 (o.7), where SD {s|s A(s
) AD = {ala A(Ts)). Then the transition dynamic is determined by PD SD X AD + ((D)). Therefore, the well- "distribution shift" problem occurs when solving the Bellman equation Eq The Bellman residual is taking expectation in Sp x AD. while the target Q-value is calculated based on the actions from the learning policy.
2.2 Consistency model
The consistency model is an enhanced generative model compared to the diffusion model. The diffusion model gradually adds noise to transform the target distribution into Gaussian distribution and by estimating the random noise to achieve the reverse process, i.e., sampling priori sample from a Gaussian distribution and denoise to the target sample iteratively (forms the sample generation tra- jectory). The consistency model is proposed to ensures each step sample generation trajectory of the diffusion process aligns with the target sample (we call consistency). Specifically, the consistency model [20] try to overcome the slow generation and inconsistency over sampling trajectory generated
by the Probability Flow (PF) ODE during training process of the diffusion model [29 30 31
32 33)
Let Pdato (x) denote the data distribution, we start by diffuse the original data distribution by the PF
ODE:
dx.
[M ()
2 (1) log(m))))))) di,
where pa(::)) is the drift coefficient, a..) is the diffusion coefficient, (are) is the distribution of æz. Po( = Pdata (a), and (æe, [c, T}} is the solution trajectory of the above PF ODE.
Consistency model aims to learn a consistency function Ta t) that maps each point in the same PF ODE trajectory its start point, i.e., fo( we. Vi E [e, T). Therefore, VI, E [e, TJ. we have fo( 1) = Jo(2g (wer, ('), which is the "selffconssstency"" property of consistency model.
Here, fo (xe, is defined as:
(4) where (t) and Coue (t) are differentiable functions, and (skip(E) 1, Cour = 0 such that they satisfy the boundary condition fo(x,. = Je- In (4), Fo t) can be free-form deep neural network with output that has the same dimension as z
fo fo t) = Colip + Come (t)
Consistency function fo (Z,, t) can be optimized by minimizing the difference of points in the same PF ODE trajectory. If we pretrained diffusion model to generate such PF ODE trajectory, then utilize it to train a consistency model, this process is called consistency distillation. We use the consistency distillation method learn consistency model in this work and optimize the consistency distillation loss as the Definition 1 in (20)
With well-trained consistency model fo(Øe, t), can generate samples by sampling from the initial Gaussian distribution êT ~ NO and then evaluating the consistency model for & = fo T). This involves only one forward pass through the consistency model and therefore generates samples in a single step. This is the one-step sampling process of the consistency model.
Q-Distribution guided Q-learning via Consistency Model
In this work, we present a novel method for offline RL called Q-Distribution Guided Q-Learning (QDQ). First, we quantify the uncertainty of Q-values by learning the Q-distribution using the consistency model. Next, we propose a strategy to identify risky actions and penalize their Q-values based on uncertainty estimation, helping to mitigate the associated risks. To tackle the excessive conservatism seen in previous approaches, we introduce uncertainty-aware Q optimization within the Aitor-Critic learning framework. This mechanism allows the Q-value function to perform both optimistic and pessimistic optimization, fostering balanced approach to learning.

---

3.1 Learn Uncertainty of Q-value by Q-distribution
Estimating the uncertainty of the Q function is a significant challenge, especially with deep neural network Q estimators. A practical indicator of uncertainty is the presence of large variances in the estimates. Techniques such as bootstrapping multiple Q-values and estimating variance (7] have been used to address this issue. However, these ensemble methods often lack diversity in Q-values [9] and fail to accurately represent the true Q-value distribution. They may require tens or hundreds of Q-values to improve accuracy, which is computationally inefficient
5
Other approaches involve estimating the Q-value distribution and determining the lower confidence bound [10 [2 [6], or engaging in in-distribution learning of the Q-value function [ [3 16
10] 34). However, these methods often struggle to provide precise uncertainty estimations for the Q-value [1]. Stabilization methods still lead to Q-value overestimation [3], while inaccurate variance estimation can worsen this problem. Furthermore, even if the Q-value is not overestimated, there is still a risk of it being overly pessimistic or constrained by the performance of the behavior policy when using in-distribution-only training.
In this subsection, elucidate the process of learning the distribution of Q-values based on the consistency model, and outline the technique for estimating the uncertainty of actions and identifying risky actions. We have give further demonstration on the performance of the consistency model and efficiency of the uncertainty estimation in Appendix G.2 and Appendix G.3
Trajectory-level truncated Q-value. We chose to estimate the Q-value distribution of the behavior policy instead of the learning policy because they share a similar set of high-uncertainty actions 5. Using the behavior policy's Q-value distribution has several advantages. First, the behavior policy's s Q-value dataset comes from the true dataset, ensuring high-quality distribution learning. In contrast, the learning policy's 's Q-value is unknown, counterfactually learned, and often noisy and biased, leading to poor data quality and biased distribution learning. Second, using the behavior policy's 's Q-value distribution to identify high-uncertainty actions does not force the learning policy's target Q-value to align with that of the behavior policy.
To gain insights into the Q-value distribution of the behavior policy, we first need the raw Q-value data. The calculation of the Q-value operates the trajectory level, represented as T (so, ao: $1, a1. ..... with an infinite horizon (see [. In the context of offline RI., our training relies on the dataset 'D produced by the behavior policy. This dataset consists of trajectories generated by the behavior policy, which is the only available trajectory-leve data. However, the trajectory-level data from the behavior policy often faces a significant challenge: sparsity. This issue becomes even more pronounced when dealing with low-quality behavior policies, the generated trajectories tend to be sporadic and do not adequately cover the entire state-action space x A, especially the high reward region.
To address this pervasive issue of sparsity, as well as the infinite summation in Eq we present a novel approach aimed enhancing sample efficiency. Our proposed solution involves the utilization of truncated trajectories to ameliorate the sparsity conundrum and avoid infinite summation. By employing a k- step sliding window of width T, we systematically traverse the original trajectories, isolating segments within the window to compute the truncated Q-value (as depicted in Figure A.I]). For instance, considering the initiation point of the i-th step sliding window as a;), by setting T i + k. we derive the truncated Q-value of this starting point as follows:
O0
terminal, 1, otherwise.
Q7 (8i, a.)
m- am) x t(8m, Omm) 188m am) m=
(5)
The truncation of Q-values can occur either through sliding window mechanisms or task terminations. When truncation happens due to termination, the Q-value from Eq equivalent the true Q-value, Q=*((---) = (. In contrast, if truncation results from window blocking, our theoretical analysis in Theorem 4.1 confirms that the distribution of truncated Q-values has properties similar to those of the true Qavalue distribution.
Using a k-step sliding window does not compromise the consistency of the trajectory, owing to the inherent memory-less Markov property RL. This strategic truncation allows for the extraction of truncated Q-values, which can improve sample efficiency, especially for long trajectories. Moreover, this approach highlights actions with potential high Q-values, as actions from lengthy trajecto- ries those with many successful interactions- are encountered more often during Q-distribution
5

---

training. Consequently, the uncertainty of these actions is lower, reducing the likelihood of them being overly pessimistic.
Learn the distribution of Q-value. In distributional RL., the learning of Q-value distributions
is typically achieved through Gaussian neural networks 35 36), Gaussian processes 37 38 or
categorical parameterization [39] However, these methods often suffer from low precision represen- tation of Q-value distributions, particularly in high-dimensional spaces. Moreover, straightforward replacement of true Q-value distributions with ensembles or bootstraps can lead to reduced accuracy in uncertainty estimation(a critical aspect in offline reinforcement learning )) or impose significant computational burdens 7].
The idea of diffusing the original distribution using random noise has rendered the diffusion model potent and high-fidelity distribution learner. However, it has limitations when estimating uncertainty. Sampling with diffusion model requires multi-step forward diffusion process ensure sample quality. Unfortunately, this iterative process can compromise the accuracy of uncertainty estimates by introducing significant fluctuations and noise into the Q-value uncertainty. For a detailed discussion, see Appendix A.2
To address this issue, suggest using the consistency model 20 to learn the Q-value distribution. The consistency model allows for one-step sampling. like other generative models, which reduces the randomness found in the multi-step sampling of diffusion models. This results in more robust uncertainty estimation. Furthermore, the consistency feature, as explained in Theorem 4.2 accurately captures how changes in actions affect the variance of the final bootstrap samples, making Q-value uncertainty more sensitive to out-of-distribution (OOD) actions compared to the diffusion model. Additionally, the fast-sampling process of the consistency model improves QDQ's efficiency. While there may be some quality loss in restoring real samples, this is negligible for QDQ since it only calculates uncertainty based on the variance of the bootstrap samples, not the absolute Q-value of the sampled samples. Overall, the consistency model is an ideal distribution learner for uncertainty estimation due its reliability, high-fidelity, ease of training. and faster sampling.
Once we derive the truncated Q dataset Do. we train a conditional consistency model, denoted by fo(ar, (s, a)). which approximates the distribution of Q-vaues. Since the consistency model aligns with one-step sampling, we can easily sample multiple Q-values for each action using the consistency model. Suppose draw n prior noise {ÔT, :T, ... .rr,) } from the initial noise distribution T)), and denoise the prior samples by the consistency one-step forward process: fee fo(êT. Ti (s, a)), 1,2, ... n. Then the variance of these Q-values, derived by
2
v(Xe((s a))
Σ
fél,T,
1 Σ fe TT(s, [
(6)
can be used to gauge the uncertainty of Q(s
3.2 Q-distribution guided optimization in offline RL
Recover Q-value function. We propose mmcrrainty-aaaat optimization objective Cus (Q) to penalize Q-value for OOD actions as well as to avoid too conservative Q-value learning for in- distribution areas. The uncertainty-awert aware learning objective for Q-value function Qo a) is
Luw (Qe) = min + (1-a)
In Eq 7, C(Qu) represents the classic Bellman residual defined in Fq This residual is used in online RL. and encourages optimistic optimization of the Q-value. contrast, L(Q#), pessimistic Bellman residual based on the uncertainty-penalizee Q target Qz(s', , d'), defined as
1
Qn(d',a) = (1o( Qo(s', ( + (Q (
(8)
In Eq & Hala'lls))= / a')) represents the uncertainty estimate of the Q-value for action (V..
a'. The set U(Q) includes actions that may be nut-of-distribution (OOD). We use the upper 3-quantile ng (a'ls') of the uncertainty estimate on actions taken by the learning policy as the threshold for forming U()). Additionally, incorporate the quantile parameter 3 robust weighting factor for

---

the unpenalized Q-target value. This helps control the estimation error of uncertainty and enhances the robustness the learning objective. We can also free weighting factor, but we use / to reduce the number of hyperparameters.
Improve the learning policy. The optimization of learning policy follows the classic online RI. paradigm:
(T) = max [Ex.P a)] + To (a)]]
In Eq 9 an entropy term introduced to further stabilize the volatile learning process of Q-value function. For datasets with a wide distribution, which can further enhance performance. Furthermore, other policy learning objectives, such as the AWR policy objective [40], can also be flexibly used within the QDQ framework, especially for the goal conditioned task like Antmaze.
simply set the penalization factor y zero,
We outline the entire learning process of QDQ in Algorithm| D In Section 4 Theorems 4.3 and 4.4 show that QDQ penalizes the OOD region based on uncertainty while ensuring that the Q Q-value function in the in-distribution region close to the optimal Q-value. This alignment is the main goal of offline RL.
Algorithm Q-Distribution guided Q-learning (QDQ)
Initialize: target network update rate K, uncertainty- learning hyperparameter a, 3.policy training hyperparameters y. Consistency model fg. networks {Qo,, Q0,), actor Th. target networks (Qe; Qo ). target actor Ter-
Q-distribution learning:
Calculate Q dataset DQ= {Q77 (8,a) a)} scanning each trajectory T E D by Eq
for each gradient step do
Sample minibatch of Q (s, a) Do
Update v minimizing consistency distillation loss in (q. 20
end for
for each gradient step do
Sample mini-batch of transitions (s, a, r, D
Updating Q-function:
Update 0 = (0,.00 02) minimizing Luw (Qo) in Eq
Updating policy:
Update & minimizing (TT in Eq
Update Target Networks:
ø' <- KÓ + (1 k)Ø': 02 k0; (1 k)0,, 1,2
end for
Theoretical Analysis
In this section, provide theoretical analysis of QDQ. The first theorem states that if T is sufficiently large, the distribution of Q does not significantly differ from the true distribution of Q This shows that our sliding window-basee truncated Q- Q-value distribution converges to the true Q-value distribution, ensuring accurate uncertainty estimation. A detailed proof can be found in Appendix B
Theorem 4.1 (Informal). Under some mildly condition, the truncated Q-value Q converge in- distribution the true true Q-value QT
F Foz2(1) (x) (x)) +00
(10)
we analyze why the consistency model is suitable for estimating uncertainty. Our analysis shows that Q-value uncertainty is more sensitive to actions. This sensitivity helps in detecting out-of-distribne (OOD) actions. A detailed statement of the theorem and its proof can be found in Appendix C
In Theorem 4.2
Theorem 4.2 (Informal). Following the assumptions in (20). fo(x, (s, ()) is L-Lipschitz We also assume the truncated Q-value is bounded by H. The action broadly inftuences (()((ss,)) by: Bwar(X.) X.)
7

---

we give theoretical analysis that the uncertainty-awee aware learning objective in Eq can converge and the details can be found in Appendix D
In Theorem 4.3
Theorem 4.3 (Informal). The Q-value function of QDQ can converge a fixed point of the Bellman equation: Q a) = { 0),, where the Bellman operator I (*{s, a) is defined as:
a) = ro, a) ( (max d')
(1-a))1(ss)) dd)].
(11)
Theorem 4.4 shows that QDQ penalizes the OOD region by uncertainty while ensuring that the Q-value function in the in-distribution region is close to the optimal Q-value, which is the goal of offline RL. Theorem 4.4 (Informal). Under mild conditions, with probability 1
we have
- S,
(12)
where Q is learned by the uncertainty-aware loss in 7 is error rate related to the difference between the classical Bellman operator BQ and the QDQ bellman operator 3 Q.
The optimal Q-value, Q derived by the QDQ algorithm can closely approximate the optimal Q-value function, Q", benefiting from the balanced approach the QDQ algorithm that avoids excessive pessimism for in-distribution areas. Both the value and vy are small and more details in Appendix
Experiments
In this section, first delve into the experimental performance of QDQ using the D4RL benchmarks [1]] Subsequently, we conduct a concise analysis of parameter settings, focusing on hyperparameter tuning across various tasks. For detailed implementation, we refer Appendix G
5.1 Performance D4RL benchmarks for Offline RL
We evaluate the proposed QDQ algorithm on the D4RL Gym-MuJoCo and AntMaze tasks. We compare it with several strong state-of-the- (SOTA) model-free methods: behavioral cloning (BC), BCQ [42]. DT [43] AWAC 44), Onestep RL TD3+BC [46], CQL [1. and IQL [IO]. We also include UWAC 77, EDAC 9 and PBRL
which use uncertainty to pessimistically adjust the Q-value function, as well as MCQ 13 which introduces mild constraints to the Q-value function. The experimental results for the baselines reported in this paper are derived from the original experiments conducted by the authors or from replication of their official code. The reported values are normalized scores defined in D4RL
Table I: Comparison of QDQ and the other baselines on the three Gym-MuJoCo tasks. All the experiment are performed on the MuJoCo "-v2" dataset. The results are calculated over random seeds.med medium, replay, expert, ha halfcheetah, walker2d, ho=hopper
Dataset
BC
AWAC
DT
TD3+BC
CQL
IQL
UWAC
MCQ
EDAC
PBRL.
QDQ(Ours)
ha-med
42.6
43.5
42.6
48.3
44.0
47.4
42.2
64.3
65.9
57.9
14.1
ho-med
52.9
57.0
67.6
59.3
58.5
66.2
50.9
78.4
101.6
75.3
99.0203
wa-med
75.3
72.4
72.5
78.3
75,4
91.0
86.92m0
ha-med-r
36.6
40.5
36.6
44.6
45.5
44.2
35,9
56.8
61.3
45.1
63.7
ho-med-r
18.1
37.2
82.7
60.9
95.0
94.7
25.3
101.6
101.0
100.6
102.4+028
wa-med-r
26.0
27.0
66.6
81.8
77.2
73.8
23.6
91.3
87.1
77.7
93.2
ha-med-e
55.2
42.8
86.8
90.7
91.6
86.7
42.7
87.5
106.3
92.3
99.3
ho-med-e
52.5
105.4
110.8
113.5
wa-med-e
107.5
74.5
108.1
110.1
108.8
109.6
96.5
114.2
114.7
110.1
115.9s01
Total
466.7
450.7
684.6
677.4
698.5
437.4
797.4
841.1
759.4
848.0+11a
Table I shows the performance comparison between QDQ and the baselines across Gym-MuloCo tasks, highlighting QDQ's 's competitive edge almost all tasks. Notably, QDQ excels on datasets with wide distributions, such medium and medium-replay datasets. In these cases, QDQ effectively

---

avoids the problem of over werrpenalizing Q-values. By balancing between being too conservative and actively exploring to find the optimal Q-value function through dynamic programming, QDQ gradually converges toward the optimal Q-value, as supported by Theorem 4.4
Table Comparison of QDQ and the other baselines on the Antmaze tasks, All the experiment performed on the Antmaze "-v0" dataset for the comparison comfortable with previous baseline. The results are calculated over random seeds.
Dataset
BC
TD3+BC
DT
Onestep RI.
AWAC
CQL.
IQL
QDQ(Ours)
umaze
54.6
78.6
59.2
64.3
56.7
74.0
87.5
98.6+28
umaze-diverse
45.6
71.4
53.0
60.7
49.3
84.0
62.2
67 27.8
medium-play
0.0
10.6
0.0
0.3
0.0
61.2
71.2
81.5+36
medium-diverse
0.0
3.0
0.0
0.0
0.7
53.7
70.0
85.4
large-play
0.0
0.0
0.0
15.8
39.6
35.6+134
large-diverse
0.0
0.0
0.0
0.0
1.0
14.9
47.5
31.2+44
Total
100.2
163.8
112.2
125.3
142.4
229.8
378
400.1
Table 2 presents the performance comparison between QDQ and selected baselines across AntMaze tasks, highlighting QDQ's commendable performance. While QDQ focuses on reducing overly pessimistic estimations, it does not compromise its performance on narrow datasets. This is evident in its competitive results the medium-expert dataset in Table T well as its performance on AntMaze tasks. Notably, QDQ outperforms SOTA methods on several datasets. This success is due to the inherent flexibility of the QDQ algorithm. By allowing for flexible hyperparameter control and seamless integration with various policy optimization methods, QDQ achieves synergistic performance enhancement.
5.2 Parameter analysis
The uncertainty-awary loss parameter a. The parameter crucial for balancing the dominance between optimistic and pessimistic updates of the Q-value (Eq 7}. A higher a value skews updates toward the optimistic side, and we choose higher a when the dataset or task is expected be highly robust. However, the setting of a is also influenced by the pessimism of the Q target defined in Eq 8 For more pessimistic Q target value, we can choose larger a. Interestingly, both the theoretical analyses in Theorem 4.3 and Theorem 4.4 and empirical parameter tuning suggest that variability in a across tasks minimal, with a typical value around 0.95.
The uncertainty related parameter 3. The parameter 3 influences both the partitioning of high uncertainty sets and acts relaxation variable control uncertainty estimation errors. When dealing with a narrow action space or sensitive task (such as the hopper task), the value of 3 should be smaller. In these cases, the Q-value more likely select OOD actions, increasing the risk of overestimation. This means we face greater uncertainty (Eq ( and need to minimize overestimation errors. Therefore, we require stricter criteria to ensure actions are in-distribution and penalize the Q-values of OOD points more heavily. A detailed analysis of how to determine the value of 3 can be found in Appendix G.3
The entropy parameter ?, The term in Eq stabilizes the learning of a simple Gaussian policy, es- pecially for action-sensitive and narrower distribution tasks. When the dataset has wide distribution or the task shows high robustness to actions (such as in the half-cheetah task), the Q-value function generalizes better across the action space. In these cases, for actions, keeping the value of 7 as small as possible or even at O. However, when the dataset is narrow (e.g., in the AntMaze task) when the task is sensitive changes actions (like in the hopper or maze tasks, where small deviations can lead to failure), larger value of is necessary. For these tasks, simple Gaussian policy can easily sample risky actions, as it fits a single-mod mode policy. Nonetheless, experimental results indicate that the sensitivity of the parameter is not very high. In fact, 7 is relatively small compared to the Q-value, primarily stabilize training and prevent instability in Gaussian policy action sampling. See Appendix G.7 for more details.
can set more lenient requirement
We do not provide the performance of UWAC, EDAC, PBRL, and MCQ on the AntMaze task, as they do not offer useful parameter settings for this task.

---

Related Works
Restrict policy deviate from OOD areas. The distribution mismatch between the behavior policy and the learning policy can be overcome if the learning policy share the same support with the behavior policy. One approach involves explicit distribution matching constraints, where the learning policy is encouraged to align with the behavior policy by minimizing the distance between their distributions. This includes techniques based on KL-divergence [47 48
40
44 46], Gensen-Shannel
divergence [4). and Wasserstein distance 47 49]. Another line of research aims to alleviate the overly conservative nature of distribution matching constraints by incorporating distribution support constraints. These methods employ techniques such as Maximum Mean Discrepancy (MMD) distance [ learning behavior density functions using implicit (50) or explicit [51] methods, or measuring the
geometric distance between actions generated by the learning and behavior policies [52] In addition to explicit constraint methods, implicit constraints can also be implemented by learning a behavior
policy sampler using techniques like Conditional Variational Autoencoders (CVAD) 42 53 50 (54)
Autoregressive Generative Model [55], Generative Adversarial Networks (GAN) 49 normalized
flow models [56), diffusion models [57] 58 59
60].
Pessimistic Q-value optimization. Pessimistic Q-value methods offer a direct approach address the issue of Q-value function overestimation, particularly when policy control fails despite the learning policy closely matching the behavior policy 3 promising approach to pessimistic Q-value estimation involves estimating uncertainty over the action space, as OOD actions typically exhibit high uncertainty. However, accurately quantifying uncertainty poses a challenge, especially with high-capacity function approximators like neural networks [4). Techniques such as ensemble or bootstrap methods have been employed to estimate multiple Q-values, providing a proxy for
uncertainty through Q-value variance 7 9
[4], importance ratio [6]] 62] or approximate Lower
Confidence Bounds (LCB) for OOD regions
9 Other methods focus on estimating the LCB of 34 expectile regression [10 or tail risk measurement
Q-values through quantile regression [63
Alternatively, some approaches seek to pessimistically estimate Q-values based the behavior policy, aiming underestimate Q-values under the learning policy distribution while maximizing Q-values under the behavior policy distribution B 13 64 65
such as Conditional Value at Risk (cVAR) 2
Another category of Q-value constraint methods involves learning Q-values only within the in-sample
16 17 18, 19, capturing only in-sample patterns and avoid OOD risk. Furthermore, Q-value functions can be replaced by safe planning methods used in model-based RL, such planning with diffusion models [66] or trajectory-level prediction using Transformers (43). However, ensemble estimation of uncertainty may tend to underestimate true uncertainty, while quantile estimation methods are sensitive to Q-distribution recovery. In-sample methods may also be limited by the performance of the behavior policy.
Conclusion
We introduce QDQ, novel framework rendering pessimistic Q-value in OOD areas by uncertainty estimation. Our approach leverages the consistency model to robustly estimate the uncertainty of Q-values. By employing this uncertainty information, QDQ can apply judicious penalty to Q-values, mitigating the overly conservative nature encountered in previous pessimistic Q-value methods. Additionally, enhance optimistic Q-learning within in-distribution areas, we introduce an uncertainty- aware learning objective for Q optimization. Both theoretical analyses and experimental evaluations demonstrate the effectiveness of QDQ. Several avenues for future research exist, including embedding QDQ into goal-conditioned tasks, enhancing exploration in online RL by efficient uncertainty estimation. We hope our work will inspire further advancements in offline reinforcement learning.
Acknowledgements
We would like thank AC and reviewers for their valuable comments on the manuscript. Bingyi Jing's research is partly supported by NSFC (No. 123712900
10

---

References
[1] Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. brief survey of deep reinforcement learning. arXiv preprint arXiv: 2017.
[2] John Tsitsiklis and Benjamin Van Roy. Analysis of temporal -ifforeeeeif learning with function approximation. Advances neural information processing systems. 9, 1996.
[3] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. volume 2020-December, 2020.
[4] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives open problems. arXiv preprint arXiv:2005.01643 2020.
[5] Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. volume 32, 2019.
[6] Scott Fujimoto, Herke Van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. volume 4, 2018.
[7] Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. arXiv preprint arXiv:2005.081400 2021.
[8] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference Machine Learning, pages 104-114. PMLR, 2020.
[9] Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-base offline reinforcement learning with diversified q-ensemble. Advances in neural information processing systems, 34:7436-744 17443 2021.
[10] Iya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-leaming. arXiv preprint arXiv:21 10.06/69, 2021.
[11] Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit g-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304.1055 10573, 2023.
[12] Núria Armengol Urpi, Sebastian Curi, and Andreas Krause. Risk-averse offline reinforcement learning. arXiv preprint arXiv:2102.0537,, 2021.
[13] Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongging Lu. Mildly conservative q-tlivee---eeee q-learning for offline reinforcement learning. arXiv preprint arXiv:2206.0477 04745, 2022.
[14] Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhihong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning. arXiv preprint urXiv:2202.1 1556. 2022.
[15] HJ Terry Suh, Glen Chou, Hongkai Dai, Lujie Yang, Abhishek Gupta, and Russ Tedrake. Fighting uncertainty with gradients: Offline reinforcement learning via diffusion score matching. In Conference on Robot Learning, pages
PMLR, 2023.
[16] Divyansh Garg, Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learning: Maxent rl without entropy. arXiv preprint arXiv:2301.02:22 2023.
[17] Chenjun Xiao, Han Wang, Yangchen Pan, Adam White, and Martha White. The sample softmax for offline reinforcement learning. arXiv preprint arXiv:2302.14333 2023.
[18] Hongchang Zhang, Yixiu Mao, Boyuan Wang, Shuncheng He, Yi Xu, and Xiangyang Ji. In- sample actor critic for offline reinforcement learning. In The Eleventh International Conference on Learning Representations, 2022.
[19] Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, and Xianyuan Zhan. Offline rl with no ood actions: In-sample learning via implicit value regularization. arXiv preprint 2303 2023.

---

[20] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consisteney models, arXiv preprint trXiv:2303.01466 2023.
[21] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprini 13Xiv:13/2.5600) 2013.
[22] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
[23] Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International conference on machine learning, pages 1995-2003, PMLR, 2016.
[24] R.S. Sutton and A.G. Barto. Reinforcement learning: An introduction. IEEE Transactions on Neural Networks, 9, 1998. ISSN 1045-9227. doi: 10.1109/tnn. 1 1998.71219..
[25] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In International conference on machine learning. pages 387-395. PMLR, 2014.
[26] Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning, arXiv preprint
2015.
[27] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 17X.v:1707.06347, 2017.
[28] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. volume 5, 2018.
[29] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International conference machine learning, pages 2256-2265. PMLR, 2015.
[30] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances neural information processing systems, 32, 2019.
[31] Yang Song and Stefano Ermon. Improved techniques for training score-basee generative models. Advances neural information processing systems, 11112438-12445 2020.
[32] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-688 6851, 2020.
[33] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-hased generative modeling through stochastic differential equations. arXiv preprint arXiw:2011.13456, 2020.
[34] Arsenii Kuznetsov, Pavel Shvechikov, Alexander Grishin, and Dmitry Vetrov. Controlling overestimation bias with truncated mixture of continuous distributional quantile critics. In International Conference on Machine Learning, pages 5556-5566. PMLR, 2020.
[35] Gabriel Barth-Maron, Matthew W Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva Tb, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional determin- istic policy gradients. arXiv preprint arXiv: 08617, 2018.
[36] Brendan O' O'Donoghue, Ian Osband, Remi Munos, and Volodymyr Mnih. The uncertainty bellman equation and exploration. In International conference machine learning, pages 1836-3845, 2018.
[37] Jus Kocijan, Roderick Murray-Smith Carl E Rasmussen, and Agathe Girard. Gaussian process model based predictive control. In Proceedings of the 2004 American control conference, volume 3, pages 2214-2219. IEEE, 2004.
12

---

[38] Craig Knuth, Glen Chou, Necmiye Ozay, and Dmitry Berenson. Planning with learned dynamies: Probabilistic guarantees on safety and reachability via lipschitz constants. IEEE Robotics and Automation Letters, 6(3):5129-5133) 2021.
[39] Mare G Bellemare, Will Dabney, and Rémi Munos. distributional perspective on reinforce- ment learning. In International conference machine learning, pages 449-458 PMLR, 2017.
[40] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage- regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177, 2019.
[41] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 2020.
[42] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. volume 2019-June, 2019.
[43] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15044-15097, 2021.
[44] Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint wxXiv22066.0935) 2020.
[45] David Brandfonbrener, Will Whitney, Rajesh Ranganath, and Joan Bruna. Offine without off off-policy evaluation. Advances in neural information processing systems, 14:4933-4944 2021.
[46] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 14:20132-20144 2021.
[47] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning, arXiv preprint arXiv:1911.1136. 2019.
[48] Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences dialog. arXiv preprint arXiv:1907.0000 0045600000000, 2019.
[49] Shentao Yang, Zhendong Wang, Huangjie Zheng, Yihao Feng, and Mingyuan Zhou. A reg- ularized implicit policy for offline reinforcement learning. arXiv preprint arXiv:2202.06673 2022.
[50] Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang, and Mingsheng Long. Supported policy optimization for offline reinforcement learning. arXiv preprint a:Xiv:2202.062.39, 2022.
[51] Jing Zhang, Chi Zhang, Wenjia Wang, and Bingyi Jing. Constrained policy optimization with explicit behavior density for offline reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
[52] Jianxiong Li, Xianyuan Zhan, Haoran Xu, Xiangyu Zhu, Jingjing Liu, and Ya-Qin Zhang. Distance-sensitive offline reinforcement learning, arXiv preprint arXiv:2205.11022 2022.
[53] Wenxuan Zhou, Sujay Bajracharya, and David Held. Plas: Latent action space for offline reinforcement learning. In Conference on Robot Learning, pages 1719-1735. PMLR, 2021.
[54] Xi Chen, Ali Ghadirzadeh, Tianhe Yu, Yuan Gao, Jianhao Wang, Wenzhe Li, Bin Liang, Chelsea Finn, and Chongjie Zhang. Varenl-variable advantage-weighte policy optimization for offline rl. arXiv preprint arXiv:2203..m NNi::2003008... 08949, 2022.
[55] Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. Emaq: Expected-max leaarning operator for simple yet effective offline and online rl. International Conference Machine Learning, pages 3682-3691. PMLR, 2021.
13

---

[56] Avi Singh, Huihan Liu, Gaoyue Zhou, Albert Yu, Nicholas Rhinehart, and Sergey Levine. Parrot: Data-driven behavioral priors for reinforcement learning. arXiv preprint arXiv:2011.10024 2020.
[57] Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies expressive policy class for offline reinforcement learning. arXiv preprint vrXiv:2208.0619, 2022.
[58] Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. arXiv preprint 2rX1v:2209.14545. 2022.
[59] Wonjoon Goo and Scott Nickum. Know your boundaries: The necessity of explicit behavioral cloning in offline rl. arXiv preprint arXiv:2206.0066 2022.
[60] Bingyi Kang, Xiao Ma, Chao Du, Tianyu Pang, and Shuicheng Yan. Efficient diffusion policies for offline reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
[61] Xiaoying Zhang. Jumpu Chen, Hongning Wang, Hong Xie, Yang Liu, John Lui, and Hang Li. Uncertainty-aware instance reweighting for off pollpp learning. Advances in Neural Information Processing Systems, 36:73691-73718, 2023.
[62] Paria Rashidinejad, Hanlin Zhu, Kunhe Yang, Stuart Russell, and Jiantao Jiao. Optimal conservative offline rl with general function approximation via augmented lagrangian. arXiv preprint arXiv:22 1.007 16, 2022.
[63] Cristian Bodnar, Adrian Li, Karol Hausman, Peter Pastor, and Mrinal Kalakrishnan, Quantile qt-opt for risk-aware vision-basee based robotic grasping. arXiv preprint arXiv:1910.02787, 2019.
[64] Liting Chen, Jie Yan, Zhengdao Shao, Lu Wang, Qingwei Lin, Saravanakumar Rajmohan, Thomas Moscibroda, and Dongmei Zhang. Conservative state value estimation for offline reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
[65] Yixiu Mao, Hongchang Zhang, Chen Chen, Yi Xu, and Xiangyang Ji. Supported value regularization for offline reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
[66] Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205.0 209999991, 2022.
[67] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milnn and Qiao Zhang. JAX: composable transformations of Python+NumPP programs, 2018. URL
http://githube wtt.illooooogggaeeeeeeeeeeeeee cowggoogl//aax
[68] Tero Karras, Miia Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-basee generative models. Advances in Neural Information Processing Systems, 35: 26565-26577, 2022.
14

---

Appendix
A Further discussion about uncertainty estimation of Q-value by Q-distribution.
A.1 The Q-value dataset enhancement with sidling window.
In Section 3.1 we analyze the challenges associated with deriving the Q-value dataset. We propose the k-step sliding window method improve sample efficiency at the trajectory level. The imple- mentation details of the k-step sliding window within an entire trajectory are illustrated in Figure A.1
This sliding window framework not only facilitates the expansion of Q-value data but also prevents the state-action pairs from becoming overly dense, thereby mitigating the risk of Q-value homogenization. In continuous state-action spaces, Q-values tend to be homogeneity when state and action are close in trajectory, which may hinder subsequent learning of the Q-value distribution.
Step1 1
Step 2
r
Step 3
This exemplifies how the sliding window mechanism operates to augment Q data. Let's consider a sliding window with width of 50 and a step size of k = 10. For a specific trajectory, at step 1, we commence with (81, a1) and compute the truncated Q-value utilizing trajectories within the window. At step 2, the sliding window progresses k steps forward, allowing us to compute the truncated Q-value for (8144, (1+4).
Figure 1.
A.2 Drawbacks of the diffusion model for estimating the uncertainty
Suppose the score matching method proposed in [30 to learn conditional score network s0 orss, a)s,a) to approximate the score function of the Q-value distribution po (z). Then we use the annealed Langevin dynamics as in (30) to sample from the learned Q-value distribution. For
To ~ ((x) from some arbitrary prior distribution π (x), the denoised sample is:
r++1 = x; + c. (l, a) + 2222, i = 0, .......... .T. 2 N(0,1,
(A.1)
The distribution of r441 cquals PQ(x) when - 0 and T -

---

Our primary approach to quantify the uncertainty of the action with respect to the Q-value is to evaluate the spread of the sampled Q-values for each (s. a). We then use the gradient of the Q-value samples with respect to the action to assess their sensitivity to changes in action. By iteratively deriving the gradient over the sampling chain of length T, as shown in Eq A.1 we approximate the following outcome:
OIT On
T Σ
('s0(-).a) Ge
(A.2)
da
From A.2 the impact of actions on Q-value samples learned from the diffusion model shows considerable instability, especially during the iterative gradient-solving process for the score network sa(2, ( ss a). This instability often leads to gradient vanishing exploding. While the diffusion model effectively recovers the Q-value distribution with high precision, the multi-step sampling process introduces significant fluctuations and noise, making difficult accurately assess the uncertainty of Q-values for different actions.
Additionally, different prior information may yield the same target sample, and this stochastic correlation also introduces an uncontrollable impact on uncertainty of the Q-value. However, the effect of prior sample variance on the uncertainty of the sampled Q-value can not be quantified with the diffusion model. So we can not guarantee the absolute influence on the Q-value uncertainty is from the action, then the performance of the uncertainty not be guaranteed.
B Convergence of the truncated Q-value distribution.
We first give a formal introduction of Theorem as Theorem B.
Theorem B.1. Suppose the true distribution of Q-value w.r.t the behavior policy Tg is defined as FOra (I). By Eq we derive the truncated Q-value Q denote the distribution of the truncated Q-value is F7 (x)). Assume the true Q-value is finite over the state and action space, Q converge in-distribution to the true true Q-value QT+
F (x) + FQ (x), +00.
(B.1)
We will show the distribution of the truncated Q-value 27 has the same property with the true distribution of the Q-value O and give brief proof of Theorem 4.1
Suppose the state-action space determined by the offline RL dataset D is Qp = Sp x Ap. Note that
QT can be seen as an r.v. defined as: (Rp, F.P T.P π&) @' (R, B(R), (P T3) (Q**)=1), where f the o-field on 12p, P ny is the probability measure on fip and is the transition probability measure over state, (IR) the Borel o-field on I, (IP TT) (Q**)=1 = (P- ((Q=))
the push forward probability measure on IR. Same as TT Q can also be seen as a a.V.
Then we show that T7 converge to Q** almost surely: O7 o. Q= when sending T to infinity. )]. Then for any
Define the trajectory level dataset D, = {Tu/TT (8k, Oko The 8k, ak; "..). trajectory Tk D,, the true Q-value w.r.t this trajectory be rewrite without the expectation 20
as::QT(sko an.)) = 7 5 a.
j=1
Truncating Q-value by termination situation. If the terminal occurs at step ki; of the trajectory Tk- then we have r(sky: (n2) O, for >> k, Then the truncated Q-value is identical the true Q-value: T% (Sko a. So the distribution of these two distribution same for the situation when the truncation is happened due to terminal of the task.
Truncating Q-value by sliding window situation. If the Q-value is truncated by sliding window as shown in Figure A.I then for a specific k- step sliding widow of width T with starting point (si, a.)
over a trajectory т, we have Of (si, ai) = Σ m m (8m, a.).
m=i

---

Define the state action set (C) == J An(E), where AT(E) = [(0, (=..))
T=n
{). By the definition of Q-value, we have (E) = Am (E), as (T C AT+1(E) 5 AT+2(E) 5
is decreasing. So B, (E) is also decreasing. Then,
lim P(B.(E)) PP( lim B,( = IP( lim A.(E)) m-000
(B.2)
By definition, (5) = {(8, a) : 10 (s. Q a)l > {). and assume the reward function
r(:-)) is bounded as r(.,)) for some constant
( (8,
>"r(sE, as) Se
(B.3)
kmm+1
kmn+
So (E) + when sending n to infinity, as the discount factor y < 1 by definition.
Then by Eq B.2 we have lim (().((E) = 0, v > 0. In probability theory, this is equivalent Q a.e.
Furthermore, if O7 converge to Q almost surely, then 0 also converge to Q in probability, and in-distribution. Hence, we finish the proof.
Remark B.1. Theorem 4.1 suggests that for arbitrary small c, there exists a sufficiently large T, such
that Fo (x) FOr a (x] < Given that the impact of rewards diminishes exponentially after as the increasing of trajectory length, it is unnecessary to set TT to an excessively large value. It's important remember that the goal of the Q dataset is learn the Q-distribution and assess the uncertainty of different actions. Therefore, the absolute magnitude of Q is not crucial. Additionally, using too many future steps may introduce significant uncertainty into the Q-value, as predictions for the distant future can be inaccurate.
Remark B.2. During the proof, the specific starting point of the sliding window holds significance; rather, our focus lies solely on the length of the window. This is primarily due to the Markovian nature of trajectories in RL, where the current state and action are unaffected by previous ones and adhere to a memoryless property. Consequently, the starting point of the sliding window exerts minimal influence on the computation of Q-
C Robustness of consistency model for uncertainty measure.
The formal introduction of Theorem 4.2 is shown in Theorem
Theorem C.1. Follow the assumptions in (20), we assume fo(æ, Ts a)) is L-Lipschitz.
By using the partial gradient analyze the inftuence of prior samples drr, time step T and action to the variance of the denoised sample var(X.), with high probability, we have:
(1) Prior noise influence the variance V(X.)
bounded by:
8V(X.)
02g
O(L"Tn V
(Tlog()))1.
(2) Time step T influence the variance V(X.) is bounded by:
(Duar(X,
OLTT Vlgnn.
ØT
(3) Action a influence the variance VXx)) is bounded by: (Baar (onur(X.))
= O(12T
da
As discussed in Section 3.1 and A.2 while the diffusion model has shown great success in lcarning distributions and generating samples, it is less suitable for scenarios where the influence of certain parameters on sample uncertainty must be guaranteed. When estimating uncertainty, we sample multiple Q-values for each (s, a) pair, and the standard deviation of these sampled Q-values measures the uncertainty. Therefore, it is crucial that the sample spread is sensitive to changes in action to accurately judge OOD actions.
However, the multi-step forward denoising process of the diffusion model undermines the influence of actions on the sampled Q-values, compromising the robustness of uncertainty estimation. Additionally, the lack of one-to-one correspondence between prior information and target samples prevents the cancellation of prior effects the value distribution through repeated sampling.

---

In contrast, the consistency model addresses these challenges. It not only overcomes the aforemen- tioned issues, but one-step sampling significantly enhances efficiency. In the following theoretical analysis, we will demonstrate the robustness of the consistency model in estimating uncertainty.
As described in 20 and Section 2.2
consistency model fo( t) is trained to mapping the prior noise on any trajectory of PF ODE to the trajectory's 's origin 2. by: fo(r,t) t) = Ze, given I and z belong to the same PF ODE trajectory. fo t) is defined in Eq
Suppose we trained a conditional consistency model fo(2, ds,, a) with the truncated Q-value dataset De = [Q7 (s,0)) a) following the one-step sampling of consistency model, we first initial a noise fT. (0, ( 1, 2, .... n2, then do one-step forward denosing and derive n sample &., f a), where 7 is a fixed time step. The variance based on the Q sample is:
2
V(X.) n2 (ax))=n1>>>
1
n
" fo(Ør, a)) - Σ fo fo (s))
T((s.a)
(C.1)
Next, we derive the gradient of V (X..) w.n.t ọτ. T. and check how change in these variable influence the variance. As state s always in-distribution during offline RL training process and has little influence on the uncertainty of the sampled Q Q-value, we skip the analysis.
Following 20 assume that fo( tls, a) is L-Lipschitz bounded, i.e., for any x and y. Il,o( [[(,, a)) folu, ( (s) S L || - -vll2-
Gradient of the prior दT.
Proof. Let be the square difference of the i-th prior in;
2 Σ folăr, ((s,aa)
ea(āT:) (((,,a))((
(C.2)
Then we have:
(X.))
1
n de, (
(C.3)
11
j = i,
θe,( =2 dtr,
Ofo(dT, T((, dā..
1 Lffo T(, (T,,(
To(,n, Σ
fo fo a()s
dir.
im1
n 1 @fo T (s,
folāT, T((s,a))
JKr. a()
(C.4)
n
O2r.
1=1
j ti, i,
de, dār.
-2
fo fo(or,, (s)
1
Σ
[0-
1 (
n i=1
Oør.
1 10 (ss
=2
So(DT;; (sss a))
fo(ār, T(s,
(C.5)
n
n
Øār.
Thus, plugging Eq
C.4
and Eq
C.5
into
@V(X.) OFf,
yields
(X.)
De, (
Oir.
nt
3 1=1
2
0 (iT, T (s,))))
(n (n- (o-1))o(oT,,1 (())
Σ
folōr, T (s,
nnn -11 1)
gir
341
(C.6)

---

As fo(,,t|s, is L-Lipschitz bounded, we have
2
@fa( (s, a))
(o(TT,,T(sss. - -T(,a,,
(s,,
dØT.
n(
dēT.
2
2 2. L2 ov collooooo T, ST, - n
L L
Σ jهز
ST.
(C.7)
n( - 1)
where 12 3r, Vj + can be bounded by cT Vlog due on. (O, with probability at least n n- Denote the constant with Cp and apply the previous process all the prior samples completes the proof.
Gradient of the time step T.
Proof.
Note that
(V
1
de, (T)
(C.8)
0T
0T
j=1
Taking partial gradient of (T')) w.r.t T' for any j (1,2, n2,, we obtain
De,
0T
fo f0z,, (((s,a)) - 1 Σ folăn, T(()
[ M(s,)))
1
0fe(dn, (s,
=2
OT
OT
1
TT, PT. S 4L llogn. n,
(C.9)
with probability at least n- since JेT. - H.T Vj # can be bounded by eT Vlogn due to f1. ~ (O, with probability at least 11
Plugging Eq C.9 into Eq C.8 yields
(X.) 0T
4cL2T TeLLTTolooo
(C.10)
Gradient of the action a.
For the gradient of @V(X.) just need to take partial gradient of (XX)) for each dimmension of the action = {{1 .., 3 by (X.)) The result is same as those we got in Eq C.10
da,
Then take the vector form, we have [V((..) - OLLT
1, which finish the proof.
da
Remark C.I. Theorem 4.2 elucidates the diminishing impact of random prior on the variance of the denoised Q-value as the sample size increases. Leveraging the consistency of sampling, concerns regarding the influence of a priori samples on the uncertainty of final target samples. Given the fixed sampling step size T, also address concerns about its effect on the uncertainty of Q samples. However, for thorough analysis, we still include the gradient analysis of V(X.) against T. Remark C.2. The influence of actions on sample variance depends on factors like the Lipschitz factor and sample size. As the sample size increases, the impact of actions Q sample variance does diminish; instead, it becomes more sensitive. Larger Q sample variance is more likely to occur when OOD actions are present. Consequently, the consistency model proves be a reliable approach for Q-value sampling and uncertainty estimation.
mitigate
Remark C.3. Although experiments in (20 show that the performance of the consistency model is less competitive compared to the diffusion model or adversarial generators like GANs, these findings have minimal relevance to our method. Our primary focus is not on the absolute accuracy of the sampled Q-values but on the sensitivity of Q sample dispersion to OOD actions and its ability to effectively capture uncertainty in such cases. Additionally, the high sampling efficiency achieved through one-step sampling compensates for the minor performance discrepancies of the consistency model.
5

---

D Convergence of the uncertainty-aware learning objective for recovering the Q-value function.
We first give a formal introduction of Theorem below.
Theorem D.1. Updating Q-value Qo(s, a) via the uncertainty-aware objective Eg is equivalent to minimizing the L2-norm of Bellman residuals: Bx==P the Bellman operator & (s,, a) is defined as:
-- -Q a) where
:= =Q(s,o):=r((s.p + (B Amaoosssaaaaa(s (
(D.1)
In addition, assume H (a)EE))) B. Then the Bellman operator FQ is cy-contraction operator in the Lo norm, where c < 1. The Q-value function Qo(8, a) can converge to fixed point by value iteration method.
We will show the the uncertainty-awart learning objective is equivalent to minimized the Bellman equation defined by specific Bellman operator JQ firstly.
Then the we proof the Bellman operator FQ is cy-contraction operator in the Lo norm, where < 1. The Q-value function Qo can converge to fixed point by the value iteration method.
D.1 Derivation of the Bellman operator FQ.
Recall the Q-value is optimized by the uncertainty-aware learning objective by Eq Lnw (Qo) = min (a + a))((())),
(D.2)
where
L(Q)H (a) 1Q0 (BQ)
=B (Qo(saa) (r(s,a) a) + +. |max
(D.3)
[(Q) = =B -BB
(Qo(s,)) (BQ) a)l²
= BBg~ Po (0) ~ (Q) (r(s, a) PP |max Qz Q
(D.4)
The uncertainty penalized Q target (2 a') is defined as:
Qn( Ho(aals)) Qo(s', (01(),a')l ((EE(()) + 3Qu(s', a (
(D.5)
For simplicity, we ignore the estimation parameter of Qa a) and just use Q(s, a) in the following proof.
We can just take the uncertainty-awan loss in Eq as a plain regression like loss and we have:
aL(Q), + (1 -10-0000000000000000000000000000
+(1 (11--0000000---00000000000000000
+(1
- 1: +
(D.6)
where C' is factor that not related (s, since the value of (BQ) a) and (BQ) a) are
fixed as we update Q. By the definition of (BQ) a) and (BQ) (s, a) in Eq D.3 and D.4 the uncertainty-awer learning is equivalent to minimized the following Bellman equation:
E Ex (aoPpol)..e-attt))(((((000 (FQ)(s.
(D.7)
where the specific Bellman operator (F a) is defined in D. Then finish the proof.

---

D.2
Bellman operator JQ is cy-contraccion operator in the Lee norm
Then we give a brief proof of the convergence of the Bellman operator FQ by value interative optimization.
For any Q-value function Qss,a), ('(s, define [Q(s FQ'ss, we have
I= I=|r(s, + -0-
+(r(s,a) (00
-y max
Q(((,
Ey 30 [1]
Sy max
Sy max max (1- +1
=y max max (1- 10000000000000000000000000000000
of
-7 max max - (00000000000000)0000000000000110 " [')))
+(1 a
Ha(a'ss) a() Hola's ss) ((ssd))
=Y max max (o(((s',') ('('''a')) + -1- (113)((((ss - Q'(s', (a
(1-a) (Q(s' Q' (a)))())))
Ho(a'ls)
max max (Q(((' Q'( -
+ (1- (1-0))(0())d)) (a'
+ -') (a) (1-a) (1)) -'(''''''((s',, -(( (a'euvn)
Ho(d')))
(1 (1 -101::c)--Q10.0))0aaaa
ST max max ((a (1
d)) Q'(s', (1 + (a
Ha(aalss)
S7 max max + a1-a)8,a+ (1 (1 H1(('',')) Q'(s',
Ho(a')))
=7 max (1 a)3, a + (1 (1 (( [IQ( Q'('', d')lo Halaalss))
(D.8)
Since Ha 1 (c)E)(()) < 8, we have max + (1- (1-a)
Ha(alsy } a
a)
always true. As 1, then a + a)s a+ (1-
Set c= a (1 - 3, then we have
F Q((s a) (s))s S syllQ(s' a's',
(D.9)
which implies FQ is cy-cootracttoon operator with ey < 1.
Suppose Q is the stationary point of the Bellman equation in D.7 then it can shown that Q iteratively updated by Eq D.7 can converge to Q
||Q
@l =|| - F 100 I -
s.. < - - 1
- 1000
(D.10)
Sending to infinity, we can derive Q' converge to Q=, then we finish the proof. Remark D.I. The assumption Ha(a"(aT (((()) < B can be always satisfied. Roughly speaking, since a' E uQ)), the uncertainty of Q-value Ho (a'ls') on this a' has large uncertainty due to the OOD property. Furthermore, can scale the absolute value (a'a'ls)) for all the action with
7

---

same factor to guarantee Ha(a")s) (a'eu(() < 3 without hurting the relative comparison for the uncertainty. Furthermore, experiment results have shown that Ho(a')a) (aren()) < B is consistently satisfied without additional processing.
E Performance of the Q-value function (s, a) derived by QDQ.
In this section, we delve into an analysis of the performance the Q-value function derived from the QDQ algorithm. Given that the primary aim of QDQ is to mitigate the issue of excessively conservative in most pessimistic Q-value methods, our focus is directed towards scrutinizing the disparity between the optimal Q-value within the offline RI framework and the optimal Q-value function yielded by QDQ.
The following is a formal version of Theorem 4.4
Theorem E.1. Suppose the optimal Q-value function over state-action space So x AD defined by the dataset D is Q". Then with probability least 1-1), 1), the Q-value function Q learned by minimizing uncertainty-aware loss (Eq 7 can approach the optimal Q" with small constant:
||Qa -Q Se.
(E.1)
where is rate related to the difference between the classical Bellman operator BQ and the QDQ Bellman operator FQ, and 1 is determined by the probability that max, {(1 (s), a')) (()lQQ))))))) (1 Ho(ees d'))(ss (ad .maxa {(1
Before the proof, we redefine some notation to make the subsequent exposition clearer.
Denote state space Sp be the state space defined by the distribution of dataset D, Ap is the action space defined by the dataset D, and actions not belong this space is the OOD actions. The optimal Q-value Q" on Sp x Ap can be derived by optimize the following Bellman equation:
((Q) = E Po(e) (o)e) (Q())) (s) (Q(ss (r(s,a) BBs [max Q(5' a'))))
E (S (aal))))
(E.2)
where (s, a) E Sp x Ap, Q' = BQ"
We first introduced Lemma E. facilitate the proof of Theorem 4.4
Lemma E.1. For any E Sp, Ap. with probability 1 77.
5 a) 1)]1Q' (:
(E.3)
and with probability n,
(8"((s,)-=000 7(1 (1 Ho- (a')s) IlQ" (.))))))
(E.4)
Proof of Lemma E.1

---

Direct computation shows that
[BQ(s, a) - +Q{,,a))
=|r(s,0) + -
(1 a)Qu(s(s o
Sy max -00_-=_____===00000_0_111111 10
Sy (P max [1 (1__))))((000((66111100
PP max [Q(s',
(1 (1-a) ( ((
= =P max - -(0+0(1-0030 + (a + Ho(aalss)
(1-a)
= -yE Pa(w) max (1
(1 -))))((s d'' + Ho(a' s4) 1((s', a')s,a') (a)
(p 10th---onn0000--------000000-000
(1-aa a) Ha(a'ls) 1Q((s d')l
(E.5)
Then with probability 1 1 17-
IBQ" a) $ FF (s, as) Sy max) -)00(1))000((0)))
S 5 a)] (1)1)
(E.6)
and with probability
(s) Sy max max a) Ha-(a'1s) )1Q(s' a')l]
5-(1-a)( - a1 Ho. (a'ls)) (11
(E.7)
Then we finish the proof.
Next, we will give brief proof of Theorem 4.4 with the results of Theorem 4.3 and Lemma Suppose the stationary point the optimal Q-value derived based on the QDQ Bellman operator Q³, which satisfying: Q = $ A
Then with probability 1 - we have
1|0 1'11~ = =Q0 300--#Q---0 +0" BQ 1
$ $1 8Q' + +1100 -Q0'll
<c7110a - Q'1 (1-aa) a) - =))1Q0(, (loo.
Hence,
|| - Q' - - - (lx
Set c 11 (1- -1000000111))1--0 Remark E.1. The error = >(1 a) -)01))))))))-0((((>>(-->>)))
finish the proof.
can be 0 by setting 3 1. In practice, we can ensure that e converges to a small value by appropriately adjusting the parameters a and 18.
Remark E.2. Our primary focus lies on the scenario where max ((1 (0) Q(s', ( (are(o))
(1 Haa(a")))) QQ((s', ((sss,"') (a)) = max./ {(1 - ()) a')) (((Q))) This preference stems from the potential minuteness of Firstly, we can rely Theorem in 20 to ensure that the consistency model can converge to ground truth, thus guaranteeing the fidelity of the learned Q-distribution. Consequently, the accuracy of our uncertainty estimation is upheld, enabling us to effectively assess OOD points and pessimistically adjust the Q-value for such occurrences [4]. Secondly, in order to mitigate segmentation errors of the uncertainty set, we introduce the penalty factor E Finally, since actions the set (Q) are always penalized, the Q-value always takes lower value when a' U(Q). Collectively, these measures reinforce our aim to maintain small n,

---

Theorem 4.4 shows the optimal Q-value by QDQ can closely approximate the true optimal Q-value Q" over Sp x AD. Then we also provide the following corollary to show that substitute the optimal Bellman operator BQ with & Q will introduce controllable eroor at each step, and give an step-wise analysis of the convergence of QDQ operator PQ.
Let Ca(s, a) = IQ a) Q (s, be the total estimation error of Q-value learned by QDQ algorithm and the optimal in in-distribution Q-value at step of the value iteration. Let (s a) (s, $ (s. a) be the Bellman residual induced by QDQ Bellman operator FQ step k. (s, a)) be the Bellman residual for the optimal in-distribution
Assume a)) a) (s, BQ*(s
Q-value optimization.
Corollary 1. At step k of value iteration, substitute the optimal Bellman operator BQ with &Q introduce an arbitrary small error &:
(e(s,a) (0)(s, + +BBND max (8,
a'
Then introduce second Lemma E.2 to help the proof of Corollary ! Lemma E.2. For any s E SD, Ap. with probability 7 (x(8, s ()( + - a)((1 - 3) Q)) (''',d')) d'ol
(E.8)
With probability n).
5e(8, S () +1(1 a) 1
Ha(aalss) Q(1 llo.
(E.9)
Proof of Lemmal F.2
0x(5,a) = as,) -
as) (Q
<s) + (BQ(s, -
7(1 - a) -1 (' (E.10)
By Lemma E.I with probability 1 1-7,
as =I@" aQ
-s)"(s,a)- - (s, (s.aa) (B) FQ*
(s, + (BQ(s,a) [Q(s,a)
<b)(s + - a) () d')loo
(E.11)
With probability 7,
(s, a) =10 - (s,(a,
-IQ" (Q (s. a) (O)
a0 + ()(s -
<6](5, a) + - (1
Ha(a")s") ][[((5' lloo.
(E.12)
Then we finish the proof.
Next, we will give brief proof of Corollary D
(u(s, a) =s)" (s a))
=sQ" - (00 (s)
<iQ - FQ (60)
(s(
((, ((,,a))(((ssa)
=800 (Q (s)) + - (x,
(s,-)(s,a) 56((s,)+(1000((011)
(s, BQ (s,
=6(()))++10000((..0) =Q" hB (8') max Q (8, - -BRen (s)) max Q* 8
(8,
55,(s, ((80)))(((+(().. (s,aa) BQ (ss (elll+BBBBoooo BB max Q (s,a) Q"
1
(8, + +2(1 a1
Ha(a')s) ) 11Q(6' ( + BB max (4- (F.
d
10

---

By Lemma E.1 and Lemma E.2 we can easily derive, with probability 1 1 17,
(x(8,a) (8 a) + (1 a) - 3) ,'))l + yBonD max 18, (F.14)
f = - Remark E.3. During the value iteration, there are two error accumulate: 67 a) and ç. Given the convergence of optimal Q-value iteration, (8, a) tends to approach an arbitrarily small value and may even converge to 0 under optimal circumstances. The error & 71 - Qss,, a al) induced by using the Bellman operator 9 instead of the optimal Bellman operator can also approach 0 as discussed in Remark
a loo. then we finish the proof.
F
Gap expanding of the QDQ algorithm.
In this section we will show that QDQ also has Gap expanding property discussed in CQL 3 Proposition F.1. The QDQ algorithm is Gap expanding for Q-values within-distributiin and Q-values out of distribution.
(I) If a & U()) and is indeed in-distribution action, the target Q-value for the in-distribution
point (a + (1 Q. The target Q-value for the OOD point is (a + Al 8) )Q And
(1-x)
(a + (1-a) (a + in-distribution actions.
(1~a) Hotaas) ) > 0 definitely. And this suggest Q-value will prefer
(2) If & (Q)) and is indeed OOD action. In such cases, the penalty parameter 3 is introduced penalize the Q-value, resulting (a+ < Q. Consequently, misclassifications of OOD actions can be handled in pessimistic manner.
While introducing the penalty parameter 3 may slightly slow down optimization updates for true in in-distribution actions, it is crucial to prioritize control over cat-of-distribution (OOD) actions due to the potentially severe consequences with pessimistic approach to OOD actions, the focus should be on reducing the impact of OOD actions. In fact, compared to previous pessimistic Q-value methods, QDQ offers greater flexibility in managing these values. This includes incorporating an uncertainty-aware learning objective and adjusting Q-values based on their uncertainty.
exploration errors. In balancing the optimization of Q-learning
G Experiment Details and More Results
G.1 Real Q dataset generation.
In Section 3.1
we use k-step sliding window approach with length of T to traverse each trajectory in the dataset and generate the Q-value dataset Dq based on Equation A good Q-value dataset for uncertainty estimation should cover broad state and action space to accurately characterize the Q distribution and detect high uncertainty in the OOD action region. This coverage helps identify actions with significant Q-value uncertainty in OOD areas.
Choosing the right value for the sliding step l and the window length T is crucial. A large value for k may result in a smaller Q dataset, while a small value may lead to excessive homogenization of the dataset. Both situations can negatively impact the learning of the Q-value distribution. If the Q-value dataset is too small, the learned Q distribution may not generalize well across the state-action space A. Conversely, the Q-values are too homogencous, it can hinder feature learning by the distribution learner.
To illustrate this, Figure G.I shows the Q dataset distributions obtained for different values of & using the halfcheetah mediim dataset. When 1 = 1, the Q distribution is more concentrated, resulting in more homogeneous Q-values. In contrast, with &o = 50, the Q distribution becomes sparser, showing a greater inclination towards individual features.
In the experiments, we consider various factors when setting the value of k, including the trajectory length, the width of the sliding window, and the derived Q-value dataset size. Throughout all experiments, we set & to 10. Interestingly, observed that the distribution of Q-values remains
11

---

robust to minor adjustments in k, indicating that the choice of &s does not necessitate overly stringent tuning.
Density
Densiiy
Dersity Plat
0.0
0.0
3.1
k=1, cata size-800, 800,000
k-10, data size-B0.30
k-50, data 1oo0-6,0033
Figure G.1: The derived Q-value distribution when using difference sliding step and same window width to scan over the trajectory's 's on halfcheetah- medeiu dataset.Thh The width of the sliding window is set to 200. The Q-value is scaled to facilitate comparison.
When choosing the width of the sliding window, T. we must consider factors such trajectory length and the resulting size of the Q-value dataset. When T increases, the size of the Q-value dataset decreases. However, if T is too small, it might truncate essential information from the true Q-value. We give the experimental analysis of Q-value distribution for different II on the sparse reward task meemaze-medium-aoe play in Figure G.2
012
002
Figure G.2: The Q distribution of the Antmaze-medium-pe dataset with varying sliding window widths (100 to 300 steps) is shown in the figure. Widening the sliding window does change the shape of the Q distribution, even though a larger window covers more information for this sparse reward task with many short trajectories. Instead, enlarging the sliding window decreases the Q value and compresses the size of the derived Q data.
In experiments,
opted for T = 200 across all tasks. This choice considers factors such as the maximum trajectory length (1000), the decay rate of -m in Eq[5) Based on the analysis provided in Section [ T does not need to be very large. We also found that minor adjustments to T do not significantly affect the Q-value distribution, indicating that strict tuning of this parameter is unnecessary.
G.2 The distribution of Q-value function.
The Q-distributions based on the truncated Q-value and the learned distribution from the consistency model are shown in Figure G.3 for Gym-MuJoCo tasks and in Figure G.4 for Antmaze tasks. In Figure G.3, the consistency model roughly captures the main characteristics of the true Q-value distribution. However, for the Antmaze task (Figure G.4), the learned distribution shows some fluctuations and a slightly wider support compared to the true Q-value distribution, particularly in the dynamic goal task ("-diverse" task).
From the distribution Antmaze we observe that trajectories for these tasks mainly consist of suboptimal or failed experiences, this underscores the challenging nature of the Antmaze task.
12

---

Further the narrower data distribution make it casier to take OOD actions during offline training and fewer positive experiences limits the optimisation of Q, these all potentially leading failure of these kinds of tasks.
baltchetah-meedlum-d
salcchetthh-mddlmm replay--22
beltchestah medium sxpert-v2
Oceuda be Srokke sunee wik
Dowilty by Wut. Nu vulac and prreler Oveane
Dowily tht hh ta semphert
1.33
1000-
6003
0.4
hoppae--dddmmeepppy-- +2
Derely Dellar O yol are surtoidd value
Dara by flat for valaa and samgles Ousuue
Deraty fe O wdaas srrpek O vause
643
100
2/2
welkerbd medium replay-12
walker medium
Derilyy be vebe samried
Cenily sumpled Oieium
Damalty for sumpler Oreike
Destar celetaeee wilty
wmetrr pably
5CN-
1/6
125
220
4.0
3/5
1.5
200
330
300
330
20
200
4.0
301
Figure G.3: The Q-value distribution based on the truncated Q-value v.s. the sample Q-value distribution via the learned consistency model for Gym-MuloCo tasks.
antmaze- umw
antmaae www---meedddi--eeeeeeeee- -lay-v0
antmaze- antmaze-lerge-plaee Sarge---e------ee---ee-
Dariity Ptt Srakk of
Demoto wde oedphee
camplee
011%
+150
of lemeria wily
0.3-
de contoncyyeaa
0.0
044
505
3.4
1:00
000
840
-100
040
antmaze wwwzee diverse-v0
antmaze medium medium-ddverse-v diverse-vO
antmaze large diverse-vO
Denely
Detiit sempleel velu
Cemide andeemphe
be vodel
Sudler pring
035-
035
3:5
513
041
-10
-10 +100
-40
Figure G.4: The Q-value distribution based on the truncated Q-value v.s. the sample Q-value distribution via the learned consistency model for Antmaze tasks.
13

---

G.3 Efficiency of the uncertainty measure.
The uncertainty measure is crucial for guiding the Q-value towards pessimistic regime within the QDQ algorithm. this section, we will first verify how uncertainty can assess the overestimation of Q-values in OOD regions. Then, we will discuss how the uncertainty se() can be shaped using the hyperparameter B.
To understand the differences in uncertainty between in-distribution actions and OOD actions from random policy, can compare their distributions. In the left graph of Figure G.5 the red line represents the 95% quantile of the standard deviation of sampled Q-values from the learned Q-distribution for an-distribution actions. In the right graph, this 95% quantile corresponds to approximately the 75% quantile of the standard deviation for OOD actions. This indicates that OOD actions contribute heavy-tailed distribution of Q-value uncertainty. resulting in larger values compared to in-distribution actions.
Figure G.6 further illustrates this, showing that the standard deviation of sampled Q-values from the learned policy sharply increases when Q-values are overestimated and become uncontrollable. These observations suggest that the uncertainty measure in the QDQ algorithm effectively captures the overestimation phenomenon in OOD actions.
Standard deviation for the in-samp value
Standard deviation for the COD Q value
0.20
0:15
Densty
0.10
0.30
0.00
10
12
14
4
10
12
14
16
10
sampled Q
sampled
Figure G.5: The uncertainty distribution Q-value based on in-distribution action v.s. the uncertainty distribution of Q-value based on OOD actions(by ramodm policy).
value function
Standard deviation the sample value
200k
400k
600k
800k
1M
200k
400k
600k
800k
1M
Figure G.6: The Q value learned by QDQ(eft), and the standard deviation of the sample Q-value from the consistency model for same state and action pair.
14

---

Table G.1: The hyperparameter used in consistency model training.
Hyperparameter
Value
Batch size
2556
Time embedding
Gaussian Fourier features embeddings
Fourier scale factor
Shared parameter
Time embedding size
Minimum time
0.002
Maximum time
80
Group norm
True
Activation in the hidden layer
Swish
Latent dim
256
Hidden Layer
3
sample size
50
Optimizer
Adam
Learning rate SDE
5e-4
Diffusion model
KVESDE 668
sampling method
heun sampler
Number of iterations
80000
Optimizer
RAdam
Learning rate
4e-4
trajectory interval
1000
Consistency model
trajectory interval factor as 668
consistency loss norm SDE solver
12
heun
sampling method
one-step sampler
Number of iterations
160000
The uncertainty set ((Q) is derived using the upper (9-quantile of the entire uncertainty value of over the action taken by the learning policy. Section [5.2] we provide brief discussion on determining the appropriate 3 during experiments. A higher / may allow for a more generous attitude towards OOD actions, which suitable for tasks with wide data distribution or that are robust for action change. Conversely, a lower f suggests more rigid control over OOD actions, appropriate for tasks with narrow data distribution point for setting 3 can be obtained by comparing the quantiles of the uncertainty distribution based on in-distribution actions and OOD actions. For instance, shown in Figure G.4| parameter tuning 0.80.
that are sensitive. During experiments, rough starting
might begin with 3 0.75
G.4 Implementation details for QDQ algorithm.
Implementation of QDQ algorithm contains consistency distillation for the consistency model and offline RI. training. The whole algorithm implemented with jax (67. The training process of consistency distillation and offline RI. is independent, we first train the consistency model by consistency distillation and save the converged model. Then we use the pretrained consistency model as the Q-value distribution sampler and go through the offline RL training, see Algorithm for the whole training process.
The consistency model is trained using pretrained diffusion model 33 The training process follows the official implementation 3 of the consistency model (20]. Since the consistency model is designed for image data, we modified the initial architecture, the NCSN++ model better fit offline RI. data. For instance, we replaced the U-Net architecture with a multilayer perceptron (MLP) that has three hidden layers, each with 256 units. This simplified architecture used learn the consistency function fo fa t). The main hyperparameters for the consistency distillation are shown in Table G.I covering both diffusion model training and consistency model training.
Consistency Distillation.
33
hhanss//gghhbb.ommmoomm///msssss models. cifar 10
15

---

Table G.2: The hyperparameters used in Actor-Critt training.
Hyperparameter
Value
Optimizer
Adam
Batch size
256
Discount factor
0.99
Actor learning rate
3c-4
Critic learning rate
3e-4
TD3 training
Number of iterations
le6
Target update
0.005
Actor noise
0.2
Actor noise clipping
0.5
Actor update frequency
2
Actor activation
relu
Critic activation
mish
Evaluation episode length
MuloCo
100 for Antmaze
Actor hidden dim
256
Architecture
Actor layers
Critic hidden dim
256
Critic layers
3
sliding window step k
10
sliding window width T
200
uncertainty parameter |
0.8 for hopper task
QDQ specific
0.9 for other task
pessimistic parameter
0.99 for halfcheetah .hoper-medium and hoper-medium-repllyy Antmaze task
0.95 for other task
entropy parameter
for halfcheetah task except the expert dataset
for other task
Offline RL training. The offline RL training follows TD3 46). which has a delayed update schedule for both the target network and the target policy network. For the Gym-MuJoCo tasks, we use the raw dataset without any preprocessing, such as state normalization reward adjustment. For the AntMaze tasks, we apply the same reward tuning as IQL [TO), with no additional preprocessing for the state. The hyperparameters used in offline RL training are shown in Table G.2
G.5 Learning Curve
The learning curve of Gym-Mujoco tasks are shown in Figure G.7 The learning curve AntMaze tasks shown in Figure G.8
G.6 Computation efficienty of QDQ
We provide detailed discussion of the computational efficiency of the QDQ algorithm we proposed, focusing the training cost of the consistency model and computation coefficiency of QDQ (the distributtion-bseed bootstrap method) compared with SOTA uncertainty estimation methods based on Q-value ensembles.
Regarding the training cost of the consistency model, we believe is nearly negligible. Training a diffusion model 4090 GPU takes about 5.2 minutes, while training consistency model using this pretrained diffusion model takes around 16 minutes. Additionally, the consistency model can be stored and reused for subsequent RL experiments, eliminating the need for retraining.
For comparison of the computational costs of QDQ with SOTA uncertainty estimation methods in offline RI, see Table G.6 As mentioned earlier, QDQ achieves significantly faster training speed compared the ensemble-based uncertainty estimation method EDAC and other SOTA methods. The results for other methods are taken from Table 3 of EDAC (9]
16

---

Propper- hopper-meedummee petpoe-mmnduu--pppttta
welkordd ww-.opeee----------------
100
120
100
100
Normaltadd sooie
Normeltaai soots
Narmeltad stbit
600
60
60
40
40
40
20
100
Nomwabed acore
60
Nomaaladd accrs
Nomalred acores
40
80
20
20
0.0
0.2
04
0.6
0.8
1.0
0.0
0.2
0.4
0.8
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
halfchoetsh med un roplay-22
106
aepoermediumm raplay
+2
106
wolkor24 modium roslay
166
120
60
100
50
Noteeizeed score
Nomelzad scoraa
Nomested score
60
60
40
20
stop
stop
stop
Figure G.7:
Training curve of different Mujoco Tasks. All results are averaged across random seeds. The evaluation interval is 5000 with evaluation episode length 10.
Table G.3: Computational performance of QDQ and other SOTA methods.
Runtime(s/epoch)
GPU MGB
SAC
21.4
1.3
CQL
38.2
1.4
EDAC
30.8
1.8
QDQ
0.028
0.74
G.7 Ablations
Although QDQ has three hyperparameters (a, 3, and 7) for flexibility, the tuning process is straight- forward. For example, as discussed Theorem 4.4 (Appendix(E) theoretically, (1-a) should be small. Since 8 controls the size of the uncertainty set and requires flexibility across different tasks, we typically set a close to 1, tuning it between 0.9 and 0.995. This only requires few experiments to find the optimal value. Our tuning process involves sequentially fixing parameters while selecting the best a, then 3, and finally 7. Based on the characteristics of different datasets, we can set each parameter to initial value close to its optimal value. QDQ offers evidence-based guidelines for
17

---

130
Norrlbee R01
Figure G.8: Training curve of different Antmaze Tasks. All results are averaged across 5 random seeds. The evaluation interval is 100000 with evaluation episode length 100.
hyperparameter ranges, making tuning manageable. Additionally, QDQ's efficiency (see Table G.6 reduces the tuning burden.
In the ablation study, we choose four task which represent different kinds of dataset we analyzed during parameter study:
(1) Wide distribution: halfcheetth-medduu--22
Task sensitive: happer-medium-v2.
Demonstration: walker2d-mediam-eepprte
(4) Narrow distribution: unaze-diverse-v0.
We perform ablation study for parameters we discussed in Section 5.2 We compare the performances of three different settings for the three parameters respectively.
The learning curves for four types of tasks with different uncertainty-aware weights a (Eq. are and hopper-medium--2 datasets, decreasing expert-v2 dataset, a slightly lower a increases training This sensitivity occurs
shown in Figure G.9 For the harms performance. For the volatility. In contrast, the umaze-diverse-vé dataset shows high sensitivity because the Antmaze task requires combining different suboptimal trajectories, which challenges algorithms like QDQ that are not fully in-sample but in-support. For instance, value that is too high or too low can lead to overly optimistic or pessimistic Q-values, causing the algorithm to favor actions that do not align with the exact suboptimal trajectories.
The ablation study of the uncertainty-relates parameter presented Figure G.11 For the wide dataset halfcheetah--mddinee v2, higher 3 is preferred, indicating less control over the uncertainty
18

---

bepout-medlum---
100
80
60
Nomelred arore
new Masal
Nomelzzd
20
20
0.2
1.0
0.0
0.2
04
0.6
088
1.0
step
was warkerzd--edd www.pprrrr------e
umazee dieer-d-
100
Namne 5000
60
Normatzo s00e
60
20
0.99
0.99
0.05
stop
stop
Figure G.9: Training curve of four different Tasks when using different a in Eq. 7 All results are averaged across random seeds. The evaluation interval is 5000 with evaluation episode length 10 for Mujoco tasks. The evaluation interval is 100000 with evaluation episode length 100 for Antmaze task.
penalty and a smaller uncertainty set. In contrast, the performance for the umaze-diverse-vd datasets is sensitive to small changes 3 due to their task sensitivity and narrow distribution. The medier experrt-x-ee dataset, however, is robust to small changes in B.
and
We present the performance of different tasks for the entropy parameter in Figure G.12 For wide distribution in hulfcheetuh-meddimm-222 a larger 7 negatively impacts performance. The hopper-mediim--2 and umaze-diverse-vd vO datasets also show sensitivity to the parameter In the malker2d-medium mediimeexpe------e222 dataset, a very small may introduce volatility into the training process. However, the final result's convergence is not significantly affected.
Furthermore, as discussed in Section 5.2 the gamma term primarily stabilizes the learning of simple Gaussian policy, especially for aetioonsensitee and dataset- dataset-narrow tasks. We note that Gaussian policies are prone to sampling risky actions because they can only single-mode policy. To verify the impact of uncertainty-
Q-value optimization in QDQ, we compared the performance of Q-values without uncertainty control (using Bellman optimization like in online RL settings) to the QDQ algorithm on the action-sensitive hopper-mediun dataset, using identical gamma settings. Figure[G.Te 1. shows that introducing an uncertainty- -based constraint for the Q-value function QDQ significantly improves training stability, convergence speed, and overall performance. This supports
19

---

the effectiveness of QDQ's s uncertainty- aware Q-value optimization. We believe that a stronger learning policy will further reduce the need for this stabilizing term.
hopper-medium--2
100
80
Normalized score
60
QDQ
Without Q pessimistic in QDQ
40
20
0
0.0
0.2
0.4
0.6
0.8
1.0
step
1e6
Figure 1.10: The training curve of the hopper-medium dataset with QDQ's uncertainty pessimistic Q learning and without Q-value adjustments is shown. The green curve indicates that the term in Eq. 9 has limited impact on performance. Comparing the learning curves, QDQ's uncertainty pessimistic Q learning boosts performance, leading to faster convergence and greater stability.
20

---

hationeetah- med umn222
hopper-meduum-c x2
50
150
0.7
125
0.7
100
Nomsized sooe
Nomwlzee
76
20
25
0.2
0.4
0.6
0.8
0.2
0.4
0.6
1.0
step
step
walkurdd-medd ummeppart-222
Lnnaza-hharse-OD
140
0.9
80
120
0-7
Nomellaa sc0r
Normalood score
80
6- 0.8
20
20
step
Figure G.11: Training curve of four different Tasks when using different 3 Section 3.2 All
results averaged across 5 random seeds. The evaluation interval is 5000 with evaluation episode length 10 for Mujoco tasks. The evaluation interval 100000 with evaluation episode length 100 for Antmaze task.
21

---

hattonedtah med umm222
hopper-medum--c +2
70
100
yml
1-5
Nomalizo scoee
40
Nomeetee
10
Y
y
0.4
0.6
0.8
0.2
0.4
0.6
0.8
1.0
step
step
wwlkus2d-medrreeee www.patrtte
120
100
Nomallard score
Nomallzed score
40
20
200
y=1
y=1
-20
1.5
-20
y-15
xtep
1et
Figure G.12: Training curve of four different Tasks when using different y in Eq. All results are averaged across random seeds. The evaluation interval is 5000 with evaluation episode length 10 for Mujoco tasks. The evaluation interval is 100000 with evaluation episode length 100 for Antmaze task.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We explicitly discuss the contribution and research focus in the abstract and introduction section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We give limitation of this work the experiment section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
23

---

We provide the detailed assumption and proof for each theoretical result in the Appendix B
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We give the detailed implementation of the experiment in experiment section and Appendix G
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
24

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ "pipsscc/puulic//uui /tt.e//uuuddeeoooouuummiiiiiii ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We give the detailed implementation of the experiment in Appendix G
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
25

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
26

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
27

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "3g2VwtJXZw", "reviewer": "Reviewer_rDnz", "summary": "The paper proposes a method for estimating Q values in the offline/batch setting by leveraging consistency models. Via these, the uncertainty over the Q function can be estimated and used as a robust penalty to prevent distribution shift in offline RL. The authors provide both experimental validation on the standard D4RL dataset, as well as theoretical insight into the performance of their algorithm.", "questions": "Why is the set of baseline methods different on the two different benchmarks? Especially the strongest method on D4RL seems to be missing from the Ant-maze comparison.", "limitations": "none", "rating": 7, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The paper proposes an intuitive and well supported idea to improve the robustness of offline RL. Following in the well established path of using uncertainty instead of pessimism to regularize Q value estimation in the offline regime, they present a solid approach to estimate this uncertainty using consistency models.\n\nAs far as I can tell, the addition and evaluation of the consistency model as the measure of uncertainty in a Q function is novel and a solid contribution to the literature.\n\nThe empirical results place the method at the top of comparable methods.", "weaknesses": "The main weakness at the moment lies in the slightly confusing presentation. While I think that all the ideas are in principle well supported, it is very hard to follow the exact setup and intuition throughout. In addition, I am not fully convinced that the theoretical results are fully informative for the empirical implementation.\n\nThe introduction of the consistency model is very terse. For readers who are less deep in the diffusion literature, this presents a large barrier for understanding the rest of the paper. I would recommend that the authors provide a slightly more thorough introduction here.\n\nThe sliding window approach to Q estimation seems to rely heavily on a relatively dense informative reward. This should be discussed in the paper. While Theorem 4.1 does (somewhat trivially) hold, it does not suggest that partial reward sums will necessarily be informative. In cases with 0 reward across a chunk (which would happen frequently with sparse rewards) the consistency model would always predict near 0 uncertainty. This issues is currently the main reason I am leaning towards recommending rejection and I am happy to discuss it in the rebuttal in case I am overestimating the importance of the problem.\n\nIn general, the theoretical analysis is not tied strongly to the rest of the paper. It is not clear if the authors draw conclusions from it for their method, or simply present it as a justification for its validity (in which case I would expect a slightly more thorough discussion).\n\nThe method requires 3 additional hyperparameters, which the authors discuss. Given the enormous difficulty of model selection / hyperparameter tuning in offline RL and the communities lack of coherent standards here, I would like for the authors to discuss how the hyperparameters were tuned (and whether this constitutes test set training). I do concede that this is a wider issue in the community and that I cannot fully fault the authors to adhering to standards here, but I think the issue should be discussed in offline RL papers.\n\nThe empirical results do not seem vastly better than MCQ, so i would tone done the writing a small bit. It is unclear whether the authors for example allowed themselves a larger hyperparameter tuning budget.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "For weakness:\n\nFirstly, we apologize for any ambiguities. In offline RL, which aims to train policy without interacting with the environment, \"distribution shift\" is the main obstacle. A learning policy may take out-of-distribution (OOD) actions, leading to overestimated Q-values. One suggested way to tackle this overestimation in Q value function is by its high uncertainty property in the OOD regions [3]. Following this idea, QDQ targets high uncertainty of the Q-values in OOD regions. Please see more details in the motivation of the \"author rebuttal\" at the top.\n\nWhile our theoretical framework is detailed in Section 4, we reference these results in Section 3 to provide context. For instance, Theorem 4.1, mentioned in line 166 of Section 3.1, shows that our sliding window-based truncated Q-value distribution converges to the true Q-value distribution, which guarantee accurate uncertainty estimation. Each theoretical result is briefly introduced in Section 4 before being detailed, such as in lines 232-235, where Theorem 4.2 is discussed. This theorem shows that the consistency model is suitable for uncertainty estimation because it guarantees the effect of actions on the variance change of the Q-value. Theorems 4.3 and 4.4 illustrate that QDQ penalizes the OOD region by uncertainty while ensuring that the Q value function in the in-distribution region is close to the optimal Q-value, which is the goal of offline RL. Detailed descriptions, proofs, and implications of these theorems are provided in Appendices B-F. We will revise the statements further for clarity.\n\n1.\tFor the consistency model: we use the consistency model [1] for the Q-value distribution learner, which addresses the inconsistency between the diffusion model's prior information and the target distribution. It also improves sampling speed through one-step sampling, outperforming the diffusion model. In essence, the consistency model is an enhanced generative model compared to the diffusion model. The diffusion model gradually adds noise to transform the target distribution into a Gaussian distribution and by estimating the random noise to achieve the reverse process, i.e., sampling a priori samples from a Gaussian distribution (forms the sample generation trajectory) and denoise to the target sample. And the consistency model ensures each step in a sample generation trajectory of the diffusion process aligns with the target sample (we call consistency), enhancing uncertainty estimation. The consistency feature, as discussed in Theorem 4.2, ensures the accurate impact of action changes on the variance of the final bootstrap samples, making Q-value uncertainty more sensitive to OOD actions compared to the diffusion model. Additionally, the fast-sampling process of the consistency model enhances QDQ's efficiency. Despite some quality loss in restoring real samples, this loss is negligible for QDQ, as it only calculates uncertainty from the variance of the bootstrap sample, not use the absolute Q-value of the sampled samples. Overall, the consistency model is an ideal distribution learner for uncertainty estimation due to its consistency, high fidelity, ease of training, and faster sampling.\n2.\tFor the sliding window: in fact, a good Q-value dataset for uncertainty estimation should span a broad state and action space of the dataset to accurately characterize the Q distribution and detect high uncertainty in the OOD action region. This coverage of the state and action allows us to identify actions with high Q-value uncertainty in OOD regions. Even if the variance of Q-values in the in-distribution region is small, it makes uncertainty detection more sensitive, as Q-value functions typically exhibit high uncertainty in OOD regions [3], aiding in identifying OOD actions by comparison. We also give some rational on how to choose the sliding window size, please see the line 814-821 in Appendix G.1. For sparse reward tasks, we apply adjustments such as those in IQL [2] to prevent all rewards from being zero and maintain some level of variance of the bootstrap samples even in the in-distribution region. Although a wider sliding window theoretically provides richer information as suggested by the reviewer, experiments (e.g., Fig. 1 in the attached PDF) show that it has not influence the shape of distribution of the Q-value data much. Conversely, larger window width reduces the Q-value dataset size, affecting coverage of the Q-value dataset, then harm the uncertainty estimation accuracy. \n3.\tFor parameter tuning, please refer to 2. in the” author rebuttal” at the top.\n4.\tWe have updated our results during parameter tuning, as shown in Table 2. QDQ outperforms MCQ by 50 points, with notable improvements in tasks like HalfCheetah-Medium and Hopper-Medium. MCQ addresses Q-value overestimation by identifying in-distribution and OOD actions from an estimated behavior policy. While both QDQ and MCQ apply mildly constraints on the Q value, they follow different approaches. QDQ focuses on Q-value constraints introduced in [3], whereas MCQ is more aligned with the method used in policy control method. Our goal with QDQ is not merely to achieve SOTA results but to advance research on uncertainty-guided pessimistic adjustment of Q-values in offline RL. QDQ provides both theoretical support and experimental validation, aiming to improve offline RL methodology and address research gaps beyond SOTA performance.\n\n\nFor the question:\n\nIn footnote 1, we explain why we did not include the same baselines as in Table 1 of Section 5, specifically the results of UWAC and MCQ. They do not provide experiment results for the Antmaze task, and we lack information on how to set their hyperparameters.\n\n[1] Song et al. Consistency models.\n\n[2] Kostrikov et al. Offline reinforcement learning with implicit q-learning.\n\n[3] Levine et al. Offline reinforcement learning: Tutorial, review, and perspectives on open problems.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5vcAp6jKsr", "reviewer": "Reviewer_4FB7", "summary": "This paper proposes a new offline RL method, Q-Distribution guided Q-learning (QDQ), which uses a consistency model to model the distribution of Q-value for uncertainty estimation and then introduces an uncertainty-aware optimization objective for pessimistic Q-learning. This method has theoretical guarantees and exhibits strong performance in the D4RL benchmark.", "questions": "To summarize the main points/questions raised in the weaknesses section:\n\n- Could the authors provide some explanations for the confusions mentioned in the first point of weaknesses?\n- Could the authors include EDAC and PBRL in Table 1 & 2?\n- Could the authors compare the computational costs (like runtime and GPU memory usage) of their method with other methods?", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper is clearly written, allowing readers to follow the main arguments.\n- It introduces a novel approach for estimating uncertainty of Q-value via a consistency model.\n- It provides detailed training curves in the appendix, making the experimental results credible.", "weaknesses": "- Personally, there are some confusions regarding the method. The consistency model is trained to fit the distribution of $Q^{\\pi{\\beta}}$, but it is used to estimate the uncertainty of the updating Q function (as shown in Eq. 8). So, I wonder why the variance of $Q^{\\pi{\\beta}}$ can estimate the uncertainty of Q. Another confusion is why there is still a policy constraint in Eq. 9. Normally, for pessimistic value offline RL methods like CQL[1], EDAC[2], and PBRL[3], they only impose conservatism on Q-value learning and do not add any constraint on policy learning. I noticed the ablation study on $\\gamma$ shows that too small $\\gamma$ leads to poor performance on some tasks like hopper-medium-v2. Does this mean that the proposed uncertainty-aware learning objective (Eq. 7) is too weak for some tasks?\n- I think some baselines are missing in the main evaluation experiment, like EDAC and PBRL. To the best of my knowledge, EDAC is the SOTA model-free uncertainty-based method so far. So, I suggest the authors include these two algorithms in Table 1 & 2.\n- The authors point out that previous uncertainty-based offline RL methods like bootstrap or ensemble impose significant computational burdens (line 181). Although their proposed method does not need ensemble Q-value networks, it additionally brings a consistency model whose training relies on a pretrained diffusion model. So, my concern is whether this will cost more computation compared with ensemble Q-value networks.\n\n[1] Kumar et al. \"Conservative Q-Learning for Offline Reinforcement Learning\"\n\n[2] An et al. \"Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble\"\n\n[3] Bai et al. \"Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "For weakness:\n\n1.\tWe apologize for the ambiguity. Please see 1. of the “author rebuttal” at the top for the rationale behind using Q-value data from the behavior policy. Although estimated uncertainty penalizes the Q target of the learning policy, it only pessimistic the Q-value in the OOD region and does not affect the learning policy's Q-value updates and optimization in the in-distribution region. We will provide a clearer explanation in Section 3.1.\n\n2. The gamma term in Eq. 9 stabilizes the learning of a simple Gaussian policy, particularly for action-sensitive and narrower distribution tasks like hopper-medium etc. For these tasks, a simple Gaussian policy can easily sample risky actions as it fits only a single-mode policy. Indeed, the inclusion of a gamma term in QDQ depends on the task. For instance, the optimal gamma for QDQ on wide distribution data (e.g., halfcheetah-medium) is 0. Estimating uncertainty is inherently challenging. QDQ aims to estimate uncertainty directly while applying minimal control over the Q-value and using a less robust Gaussian policy. This combination increases the risk of unstable Q-value training. In fact, the gamma in Eq. 9 is very small relative to the Q-value, primarily to stabilize training and avoid instability in Gaussian policy action sampling. CQL [1], although not explicitly controlling the policy, requires the leaning the behavior policy's distribution and another policy to cover the action space in the dataset, this is some implicit action intervene. IQL [2] uses AWR policy loss to control the learning policy as well as the expectile regression to underestimate the Q value function, and PBRL [3] mentions in the penultimate paragraph of the introduction that simply using ensemble to estimate uncertainty to control Q-value is not very useful, and highlights the need for OOD data in the learning policy's action space to the success of Q value ensemble kinds methods. MCQ follows a similar approach like PBRL and not just constrain the Q value function by behavior policy’s support. To verify the impact of the uncertainty-aware Q-value optimization in QDQ, we compared the performance of Q-values without uncertainty control (Bellman optimization for the Q value function like in online RL setting) to the QDQ algorithm on hopper-medium data, using identical gamma terms setting. Figure 2 in the attached PDF shows the learning curve. The figure illustrates that introducing uncertainty-based constraint for the Q value function in QDQ significantly improves training stability, convergence speed, and learning strategy performance. This supports the effectiveness of QDQ's uncertainty-aware Q-value optimization. We believe stronger learning policy will further reduce the need for this stabilizing term.\n3.\tThank you for your reminder. We apologize for the oversight and will add EDAC[4] and PBRL[3] to our baseline(Table 2) and related work.\n4.\tRegarding the training cost of the consistency model, we believe it is almost negligible. Training a diffusion model on a 4090 GPU takes about 5.2 minutes, while training a consistency model with this pretrained diffusion model takes about 16 minutes. Additionally, the consistency model is stored and can be reused for subsequent RL experiments, eliminating the need for retraining.\nFor the comparison of computational costs, see Table 1 in the “author rebuttal” at the top. The results of other methods are taken from Table 3 of EDAC [4]. QDQ aims not only to achieve SOTA performance but also to fill gaps in uncertainty research and advance methods in Q-value constraints.\n\nFor Questions:\n\nPlease refer to our previous reply on weaknesses. \n\nTable 2: Comparison of QDQ and the other baselines on the three Gym-MuJoCo tasks. All the experiment are performed on the MuJoCo \"-v2\" dataset. The results are calculated  over 5 random seeds.med = medium, r = replay, e = expert, ha = halfcheetah, wa = walker2d, ho=hopper\n\n| Dataset  | BC    | AWAC  | DT    | TD3+BC | CQL   | IQL   | UWAC  | MCQ   | EDAC        | PBRL  | QDQ(Ours)   |\n|----------|-------|-------|-------|--------|-------|-------|-------|-------|-------------|-------|-------------|\n| ha-med   | 42.6  | 43.5  | 42.6  | 48.3   | 44.0  | 47.4  | 42.2  | 64.3  | 65.9        | 57.9  | **74.1**  |\n| ho-med   | 52.9  | 57.0  | 67.6  | 59.3   | 58.5  | 66.2  | 50.9  | 78.4  | **101.6** | 75.3  | **99.0**  |\n| wa-med   | 75.3  | 72.4  | 74.0  | 83.7   | 72.5  | 78.3  | 75.4  | 91.0  | **92.5**  | 89.6  | 86.9        |\n| ha-med-r | 36.6  | 40.5  | 36.6  | 44.6   | 45.5  | 44.2  | 35.9  | 56.8  | 61.3        | 45.1  | **63.7**  |\n| ho-med-r | 18.1  | 37.2  | 82.7  | 60.9   | 95.0  | 94.7  | 25.3  | 101.6 | 101.0       | 100.6 | **102.4** |\n| wa-med-r | 26.0  | 27.0  | 66.6  | 81.8   | 77.2  | 73.8  | 23.6  | 91.3  | 87.1        | 77.7  | **93.2**  |\n| ha-med-e | 55.2  | 42.8  | 86.8  | 90.7   | 91.6  | 86.7  | 42.7  | 87.5  | **106.3** | 92.3  | 99.3        |\n| ho-med-e | 52.5  | 55.8  | 107.6 | 98.0   | 105.4 | 91.5  | 44.9  | 112.3 | 110.7       | 110.8 | **113.5** |\n| wa-med-e | 107.5 | 74.5  | 108.1 | 110.1  | 108.8 | 109.6 | 96.5  | 114.2 | 114.7       | 110.1 | **115.9** |\n| Total    | 466.7 | 450.7 | 672.6 | 684.6  | 677.4 | 698.5 | 437.4 | 797.4 | 841.1       | 759.4 | **848.0** |\n\n[1] Kumar et al. \"Conservative Q-Learning for Offline Reinforcement Learning\".\n\n[2] Kostrikov et al. Offline reinforcement learning with implicit q-learning.\n\n[3] Bai et al. Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning.\n\n[4] An et al. Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GWfALbYfjI", "reviewer": "Reviewer_T4W8", "summary": "This paper addresses the well-known issue of OOD (Out-of-Distribution) actions in offline reinforcement learning by proposing the QDQ method, which penalizes the Q-values in regions with high uncertainty. To better estimate uncertainty, QDQ first constructs a truncated Q-value dataset for the behavior policy and learns the distribution of Q-values induced by the behavior policy using a consistency model, allowing subsequent estimation of the uncertainty for each state-action pair through one-step sampling. With this uncertainty estimation method, QDQ can perform  pessimistic value estimation for samples, thus avoiding the overestimation of Q-values for OOD actions. The paper provides a series of theoretical results to support the rationality of the QDQ method and demonstrates the advantages of the QDQ method through a series of experiments.", "questions": "The same as Weaknesses", "limitations": "This paper does not mention any potential negative impacts that may arise from its work. It is recommended that this be supplemented.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposes an innovative approach to estimate the sample uncertainty in the dataset by learning a consistency model and employing one-step sampling. This scheme is novel as it does not require maintaining multiple Q-networks to estimate uncertainty, as traditional methods do, thus incurring less computational overhead. Moreover, unlike the diffusion model, which requires multi-step denoising to obtain uncertainty, this approach ensures higher fidelity in uncertainty estimation without the need for iterative processes. Consequently, this scheme is practical in terms of implementation.\n\n2. The paper has a solid theoretical foundation, demonstrating the feasibility and convergence of the algorithm.\n\n3. The paper is well-written with clear expression, distinct structure, and logical flow.", "weaknesses": "1. First and foremost, the fundamental assumption of this paper is that a consistency model can learn a distribution of Q-values with high fidelity. However, why do Q-values need to satisfy the property of consistency? If we are learning the distribution of trajectories, it is reasonable to require the trajectory distribution to be consistent, but is this assumption also reasonable for the distribution of Q-values? This is the starting point of the paper, yet it does not argue for its necessity. Therefore, I suspect that for any type of generative model, even Variational Autoencoders (VAEs), the uncertainty of Q(s,a) could be calculated, thereby allowing the subsequent methods of this paper to be applied for learning.\n\n2. Based on my understanding, Equation (6) actually calculates the emperical version $V_{\\epsilon}(s,a)$ of uncertainty $Var(Q^{\\pi_{\\beta}})$  corresponding to the behavior policy $\\pi_{\\beta}$, rather than the uncertainty $Var(Q^{\\pi})$ of  corresponding to the current policy $\\pi$. However, it seems to me that in the subsequent use of $V_{\\epsilon}(s,a)$ and in the proof of the theorems, $V_{\\epsilon}(s,a)$ is directly used as $Var(Q^{\\pi})$, which could be mathematically problematic. The authors need to provide an explanation for this point. If indeed $Var(Q^{\\pi_{\\beta}})$ is used in place of $Var(Q^{\\pi})$, please justify the rationale.\n\n3. This paper lacks some comparisons with related work. It could be compared with methods that use ensembles to estimate uncertainty [1] and methods that make pessimistic estimates for Out-of-Distribution (OOD) actions [2].\n\n\n[1] Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning.\n[2] Supported value regularization for offline reinforcement learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "For weakness:\n    \n1.\tWe apologize for any ambiguity caused by our description of the consistency model. Consistency is a feature of the consistency model, not a requirement for the Q-value. The consistency model ensures a consistent relationship between the prior sample and the target sample during generation. This consistency is beneficial for computing Q-value uncertainty, as analyzed in Theorem 4.2, which shows that the Q-value distribution learned with the consistency model can ensure the absolute effect of action changes on the variance of the final bootstrap sample. This makes Q-value uncertainty more sensitive to OOD actions compared to the diffusion model. Additionally, the fast-sampling process of the consistency model significantly enhances QDQ's efficiency. Although one-step sampling may slightly reduce sample quality, QDQ only calculates uncertainty by the variance of the bootstrap sample, so this loss is negligible. Overall, the consistency model is highly suitable for uncertainty estimation due to its consistency, high fidelity, easier training, and faster sampling.\nRegarding the use of VAE and GAN for learning Q-value distributions, any distribution learner can be applied, but we prioritize high fidelity and efficiency. VAE often assigns excessive probability mass to less important regions by covering mode [1,2], leading to inaccuracies, especially in complex or multimodal Q-value distributions. GANs face instability and training difficulties due to their adversarial nature and the Nash equilibrium is not easy to reach, and GAN often resulting in mode collapse [3,4]. Thus, we prefer the consistency model for its stability and effectiveness.\n2.\tPlease refer to the 1. of the “author rebuttal” at the top for the reason and rationale for estimating uncertainty by using the Q-value distribution of the behavior policy. Moreover, while the estimated uncertainty influences the Q target penalty in the learning policy, it only affects the Q-value in the OOD region and does not impact the updating or optimization of the learning policy in the in-distribution region. Additionally, the uncertainty term has minimal effect on our theoretical proofs; for instance, in the proof of Theorem 4.3 (Appendix D.2), the uncertainty term is ultimately canceled out (see Eq. D.9 to Eq. D.10).\n3.\tThank you very much for your reminder, we apologize for our oversight and we will take your suggestion to add [5] and [6] to our related work.\n\nAbout limitation:\n    \nWe apologize that we have not been very clear about QDQ's limitation. Uncertainty estimation is challenging, but high-precision uncertainty estimation is crucial for the success of Q constraint kind method in offline RL [7]. We hope QDQ can serve as an uncertainty-based RL algorithm to facilitate the successful implementation Q value constraint method in offline RL. However, QDQ has limitations, particularly in using estimated uncertainty to adjust the learning policy's Q target value. Currently, QDQ employs a pessimistic adjustment by dividing the uncertainty value for OOD actions, but this approach may not be optimal. We will continue researching how to combine uncertainty estimation with the LCB of the learning policy, providing theoretical and experimental insights. We apologize for omitting this limitation and will add it to section 7.\n\n[1] Ghasemipour et al. EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL.\n    \n[2] Chen et al. OFFLINE REINFORCEMENT LEARNING VIA HIGHFIDELITY GENERATIVE BEHAVIOR MODELING.\n\n[3] Martin et al. Wasserstein generative adversarial networks.\n\n[4] Li et al. MMD GAN: Towards deeper understanding of moment matching network.\n\n[5] Bai et al. Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning.\n\n[6] Mao et al. Supported value regularization for offline reinforcement learning.\n\n[7] Levine et al. Offline reinforcement learning: Tutorial, review, and perspectives on open problems.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate the valuable comments from our three reviewers, which have helped us improve our manuscript. We have provided detailed answers to each question and included additional experiments in the attached PDF. To address any remaining confusion, we would like to introduce the motivation behind our work (QDQ).

QDQ aims to directly control overestimation in offline Reinforcement Learning (RL) for Q value functions by estimating the uncertainty of Q-values concerning the action during training [1]. Estimating Q-value uncertainty is challenging [1], and to our best knowledge, most methods address it indirectly by using ensembles of Q-value functions rather than bootstrapping. These ensemble methods suffer from lacking diversity in the Q-value[2] and fail to accurately represent the true Q-value distribution, often requiring tens or hundreds of Q-values to improve accuracy, which is computationally inefficient [2,4]. Other methods, like CQL[6] and MCQ, focus on underestimating Q-values in the OOD region by first identifying the OOD region, rather than exploiting Q-value uncertainty in that region.

QDQ aims to solve the problem of estimating Q-value uncertainty by directly computing this uncertainty through bootstrap sampling from the distribution of Q-values of the behavior policy. By approximating the behavior policy's Q-values from the dataset, we train a high-fidelity, efficient distribution learner-consistency model. This ensures the quality of the learned Q-value distribution. Since the behavior and learning policy share the same high uncertainty action set [4], we can sample from the learned Q-value distribution to estimate uncertainty, identify risky actions, and make Q target values for these actions more pessimistic. Additionally, QDQ proposes an uncertainty-aware Q-value optimization objective to avoid excessively penalizing Q-values, ensuring the Q value function’s exploratory in the in-distribution region. QDQ seeks to find the optimal Q-value that exceeds the behavioral policy's optimal Q-value while being as pessimistic as possible in the OOD region.

QDQ aims not only to achieve state-of-the-art experimental results but also to explore and advance uncertainty estimation in RL, an area with limited research. By promoting the role of uncertainty estimation in offline RL, QDQ seeks to enhance the methodology completeness of Q-value constraint. It is designed to be concise while maintaining competitive experimental performance and efficiency (Table 1). These features make QDQ flexible for integration into more complex elements, such as, enabling the incorporation of more powerful policies and enhancing exploration in online RL [1].

Next, we provide a brief description of some confusing details for QDQs:

1.	We chose to estimate the Q-value distribution of the behavior policy rather than the learning policy because they share almost the same high-uncertainty action set [4]. Using the behavior policy’s Q-value distribution offers several advantages. Firstly, the behavior policy's Q-value dataset is derived from the true dataset, ensuring high-quality distribution learning. Conversely, the learning policy's Q-value is unknown, counterfactually learned, and often noisy and biased. Poor data quality would lead to biased distribution learning. Secondly, using the behavior policy's Q-value distribution to identify high-uncertainty actions does not constrain the learning policy's target Q-value to match the behavior policy's. This aligns with our goal of a mildly constrained Q-value. Theorems 4.3 and 4.4 demonstrate that our uncertainty-aware Q-value optimization objective can train a Q-value that closely approximates the optimal Q-value in the in-distribution region, outperforming the behavior policy, as confirmed by our experimental results. We will integrate these points into the paper to clarify the QDQ algorithm.
2.	Although QDQ has three hyperparameters ($\alpha$, $\beta$, and $\gamma$) to achieve more flexible functions, the tuning process is straightforward. Take $\alpha$ as example, as discussed in Theorem 4.4 (Appendix E), theoretically $(1-\alpha)(1-\beta)$ should be small. Since beta controls the size of the uncertainty set and needs flexibility across different tasks, we typically set $\alpha$ closer to 1, tuning it between 0.9 and 0.995, which requires only a few experiments to find the optimal value. Our tuning process involves sequentially fixing parameters while choosing the best $\alpha$, then $\beta$, and finally $\gamma$. Considering the characteristics of different datasets (Section 5.2 and Appendix G.3), we can set each parameter a confidence initial value that close to its optimal value. QDQ provides evidence-based guidelines for hyperparameter ranges, making tuning manageable. Additionally, QDQ's speed (Table 1) reduces the tuning burden. This analysis as well as the tuning detail of $\beta$ and $\gamma$ will be detailed in Section 5 and Appendix G.3.

Table 1: Computational performance of QDQ and other SOTA methods

|           | Runtime(s/epoch) | GPU Mem.(GB) |
|-----------|------------------|--------------|
| SAC       | 21.4             | 1.3          |
| CQL       | 38.2             | 1.4          |
| EDAC      | 30.8             | 1.8          |
| **QDQ** | **0.028**      | **0.74**   |

[1] Levine et al. Offline reinforcement learning: Tutorial, review, and perspectives on open problems.

[2] An et al. Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble.

[3] Agarwal et al. An Optimistic Perspective on Offline Reinforcement Learning.

[4] Kumar et al. Stabilizing off-policy Q-learning via bootstrapping error reduction.

[5] Kostrikov et al. Offline reinforcement learning with implicit q-learning.

[6] Kumar et al. "Conservative Q-Learning for Offline Reinforcement Learning".

Meta-review: The paper presents a method for conservative offline RL via consistency models, that reviewers found well-motivated, intuitive, novel, and sound. The paper itself is mostly clear and the empirical evaluation is satisfactory. Some concerns were raised regarding the presentation and correctness of the method's theoretical grounding, as well as its context in the literature. The reviewers and I are satisfied with the authors' explanation of these issues and with their promise to polish the writing.
Decision: Accept (poster)