Q-Distribution guided Q-learning for offline reinforcement learning: Uncertainty penalized Q-value via consistency model

Authors: jing  zhang
kexin  shi
wenjia  wang
bing-yi  jing
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 16:57:10

Abstract: ``Distribution shift'' is the primary obstacle to the success of offline reinforcement learning. As a learning policy may take actions beyond the knowledge of the behavior policy (referred to as Out-of-Distribution (OOD) actions), the Q-values of these OOD actions can be easily overestimated. Consequently, the learning policy becomes biasedly optimized using the incorrect recovered Q-value function. One commonly used idea to avoid the overestimation of Q-value is to make a pessimistic adjustment. Our key idea is to penalize the Q-values of OOD actions that correspond to high uncertainty. In this work, we propose Q-Distribution guided Q-learning (QDQ) which  pessimistic Q-value on OOD regions based on uncertainty estimation. The uncertainty measure is based on the conditional Q-value distribution, which is learned via a high-fidelity and efficient consistency model. On the other hand, to avoid the overly conservative problem, we introduce an uncertainty-aware optimization objective to update the Q-value function. The proposed QDQ demonstrates solid theoretical guarantees for the accuracy of Q-value distribution learning and uncertainty measurement, as well as the performance of the learning policy. QDQ consistently exhibits strong performance in the D4RL benchmark and shows significant improvements for many tasks. Our code can be found at <code link>.

Main paper: 
 introduction: 
 Reinforcement learning (RL) has seen remarkable success by using expressive deep neural networks to estimate the value function or policy function [1] . However, in deep RL optimization, updating the Q-value function or policy value function can be unstable and introduce significant bias [2] . Since the learning policy is influenced by the Q-value function, any bias in the Q-values affects the learning policy. In online RL, the agent's interaction with the environment helps mitigate this bias through reward feedback for biased actions. However, in offline RL, the learning relies solely on data from a behavior policy, making information about rewards for states and actions outside the dataset's distribution unavailable.It is commonly observed that during offline RL training, backups using OOD actions often lead to target Q-values being overestimated [3] (see Figure 1(a) ). As a result, the learning policy tends to prioritize these risky actions during policy improvement. This false prioritization accumulates with each training step, ultimately leading to failure in the offline training process [4, 5, 6] . Therefore, addressing Q-value overestimation for OOD actions is crucial for the effective implementation of offline reinforcement learning. Figure 1 : (a) The maximum of the estimated Q-value often occurs in OOD actions due to the instability of the offline RL backup process and the "distribution shift" problem , so the Q-value of the learning policy (yellow line) will diverge from the behavior policy's action space (blue line) during the training. (b) The red line represents the optimal Q-value within the action space of the dataset, while the blue line depicts the Q-value function of the behavior policy. The gold line corresponds to the Q-value derived from the in-sample Q training algorithm, showcasing a distribution constrained by the behavior policy. On the other hand, the green line illustrates the Q-value resulting from a more conservative Q training process. Although it adopts lower values in OOD actions, the Q-value within in-distribution areas proves excessively pessimistic, failing to approach the optimal Q-value.Since any bias or error in the Q-value will propagate to the learning policy, it's crucial to evaluate whether the Q-value is assigned to OOD actions and to apply a pessimistic adjustment to address overestimation. Ideally, this adjustment should only target OOD actions. One common way to identify whether the Q-value function is updated by OOD actions is by estimating the uncertainty of the Q-value [4] in the action space. However, estimating uncertainty presents significant challenges, especially with high-capacity Q-value function approximators like neural networks [4] . If Q-value uncertainty is not accurately estimated, a penalty may be uniformly applied across most actions [7] , hindering the optimality of the Q-value function.While various methods [7, 8, 9, 10, 11, 12, 3, 13, 14] attempt to make pessimistic estimates of the Q-value function, most have not effectively determined which Q-values need constraining or how to pessimistically estimate them with reliable and efficient uncertainty estimates. As a result, previous methods often end up being overly conservative in their Q-value estimations [15] or fail to achieve a tight lower confidence bound of the optimal Q-value function. Moreover, some in-sample training [16, 17, 18, 19] of the Q-value function may lead it to closely mimic the Q-value of the behavior policy (see Figure 1(b) ), rendering it unable to surpass the performance of the behavior policy, especially when the behavior policy is sub-optimal. Therefore, in balancing Q safety for learning and not hindering the recovery of the most optimal Q-value, current methods tend to prioritize safe optimization of the Q-value function.In this study, we introduce Q-Distribution Guided Q-Learning (QDQ) for offline RL 2 . The core concept focuses on estimating Q-value uncertainty by directly computing this uncertainty through bootstrap sampling from the behavior policy's Q-value distribution. By approximating the behavior policy's Q-values using the dataset, we train a high-fidelity and efficient distribution learnerconsistency model [20] . This ensures the quality of the learned Q-value distribution.Since the behavior and learning policies share the same set of high-uncertainty actions [5] , we can sample from the learned Q-value distribution to estimate uncertainty, identify risky actions, and make the Q target values for these actions more pessimistic. We then create an uncertaintyaware optimization objective to carefully penalize Q-values that may be OOD, ensuring that the constraints are appropriately pessimistic without hindering the Q-value function's exploration in the in-distribution region. QDQ aims to find the optimal Q-value that exceeds the behavior policy's optimal Q-value while remaining as pessimistic as possible in the OOD region. Moreover, our pessimistic approach is robust against errors in uncertainty estimation. Our main contributions are as follows:• Utilization of trajectory-level data with a sliding window: We use trajectory-level data with a sliding window approach to create the real truncated Q dataset. Our theoretical analysis (Theorem 4.1) confirms that the generated data has a distribution similar to true Q-values. Additionally, distributions learned from this dataset tend to favor high-reward actions. • Introduction of a consistency model as a distribution learner: QDQ introduces the consistency model [20] as the distribution learner for the Q-value. Similar to the diffusion model, the consistency model demonstrates strong capabilities in distribution learning. Our theoretical analysis (Theorem 4.2) highlights its consistency and one-step sampling properties, making it an ideal choice for uncertainty estimation. • Risk estimation of Q-values through uncertainty assessment: QDQ estimates the risk set of Q-values by evaluating the uncertainty of actions. For Q-values likely to be overestimated and associated with high uncertainty, a pessimistic penalty is applied. For safer Q-values, a mild adjustment based on uncertainty error enhances their robustness. • Uncertainty-aware optimization objective to address conservatism: To reduce the overly conservative nature of pessimistic Q-learning in offline RL, QDQ introduces an uncertainty-aware optimization objective. This involves simultaneous optimistic and pessimistic learning of the Q-value. Theoretical (Theorem 4.3 and Theorem 4.4) and experimental analyses show that this approach effectively mitigates conservatism issues. 
background: 
 Our approach aims to temper the Q-values in OOD areas to mitigate the risk of unpredictable extrapolation errors, leveraging uncertainty estimation. We estimate the uncertainty of Q-values across actions visited by the learning policy using samples from a learned conditional Q-distribution via the consistency model. In this section, we provide a concise overview of the problem settings in offline RL and introduce the consistency model.The online RL process is shaped by an infinite-horizon Markov decision process (MDP): M = {S, A, P, r, µ 0 , γ}. The state space is S, and A is the action space. The transition dynamic among the state is determined by P : S × A → ∆(S), where ∆(S) is the support of S. The reward determined on the whole state and action space is r : S × A → R, r < ∞, and can either be deterministic or random. µ 0 (s 0 ) is the distribution of the initial states s 0 , γ ∈ (0, 1) is the discount factor. The goal of RL is to find the optimal policy π : S → ∆(a) that yields the highest long-term average return:EQUATIONwhere Q(s, a) is the Q-value function under policy π. The process of obtaining the optimal policy is generally to recover the optimal Q-value function, which maximizes the Q-value function over the whole space A, and then to obtain either an implicit policy (Q-Learning algorithm [21, 22, 23] ), or a parameterized policy (Actor-Critic algorithm [24, 25, 26, 27, 28] ).The optimal Q-value function Q * (s, a) can be obtained by minimizing the Bellman residual:EQUATION)where B is the Bellman operator defined asBQ(s, a) := r(s, a) + γE s ′ ∼P [max a ′ ∼π Q(s ′ , a ′ )].However, the whole paradigm needs to be adjusted in the offline RL setting, as MDP is only determined from a dataset D, which is generated by behavior policy π β . Hence, the state and action space is constraint by the distribution support of D. We redefine the MDP in the offline RL setting as:M D = {S D , A D , P D , r, µ 0 , γ}, where S D = {s|s ∈ ∆(s D )}, A D = {a|a ∈ ∆(π β )}.Then the transition dynamic is determined by P D : S D × A D → ∆(S D ). Therefore, the well-known "distribution shift" problem occurs when solving the Bellman equation Eq.2. The Bellman residual is taking expectation in S D × A D , while the target Q-value is calculated based on the actions from the learning policy.The consistency model is an enhanced generative model compared to the diffusion model. The diffusion model gradually adds noise to transform the target distribution into a Gaussian distribution and by estimating the random noise to achieve the reverse process, i.e., sampling a priori sample from a Gaussian distribution and denoise to the target sample iteratively (forms the sample generation trajectory). The consistency model is proposed to ensures each step in a sample generation trajectory of the diffusion process aligns with the target sample (we call consistency). Specifically, the consistency model [20] try to overcome the slow generation and inconsistency over sampling trajectory generated by the Probability Flow (PF) ODE during training process of the diffusion model [29, 30, 31, 32, 33] .Let p data (x) denote the data distribution, we start by diffuse the original data distribution by the PF ODE:EQUATIONwhere µ(•, •) is the drift coefficient, σ(•) is the diffusion coefficient, p t (x t ) is the distribution of x t , p 0 (x) ≡ p data (x),(x t , t) = x ϵ , ∀t ∈ [ϵ, T ]. Therefore, ∀t, t ′ ∈ [ϵ, T ], we have f θ (x t , t) = f θ (x t ′ , t ′ ), which is the "self-consistency" property of consistency model.Here, f θ (x t , t) is defined as:EQUATIONwhere c skip (t) and c out (t) are differentiable functions, and c skip (ϵ) = 1, c out (ϵ) = 0 such that they satisfy the boundary condition f θ (x ϵ , ϵ) = x ϵ . In (4), F θ (x t , t) can be free-form deep neural network with output that has the same dimension as x t .Consistency function f θ (x t , t) can be optimized by minimizing the difference of points in the same PF ODE trajectory. If we use a pretrained diffusion model to generate such PF ODE trajectory, then utilize it to train a consistency model, this process is called consistency distillation. We use the consistency distillation method to learn a consistency model in this work and optimize the consistency distillation loss as the Definition 1 in [20] .With a well-trained consistency model f θ (x t , t), we can generate samples by sampling from the initial Gaussian distribution xT ∼ N (0, T 2 I) and then evaluating the consistency model for xϵ = f θ (x T , T ). This involves only one forward pass through the consistency model and therefore generates samples in a single step. This is the one-step sampling process of the consistency model. 
q-distribution guided q-learning via consistency model: 
 In this work, we present a novel method for offline RL called Q-Distribution Guided Q-Learning (QDQ). First, we quantify the uncertainty of Q-values by learning the Q-distribution using the consistency model. Next, we propose a strategy to identify risky actions and penalize their Q-values based on uncertainty estimation, helping to mitigate the associated risks. To tackle the excessive conservatism seen in previous approaches, we introduce uncertainty-aware Q optimization within the Actor-Critic learning framework. This mechanism allows the Q-value function to perform both optimistic and pessimistic optimization, fostering a balanced approach to learning.Estimating the uncertainty of the Q function is a significant challenge, especially with deep neural network Q estimators. A practical indicator of uncertainty is the presence of large variances in the estimates. Techniques such as bootstrapping multiple Q-values and estimating variance [7] have been used to address this issue. However, these ensemble methods often lack diversity in Q-values [9] and fail to accurately represent the true Q-value distribution. They may require tens or hundreds of Q-values to improve accuracy, which is computationally inefficient [9, 5] .Other approaches involve estimating the Q-value distribution and determining the lower confidence bound [10, 12, 16] , or engaging in in-distribution learning of the Q-value function [3, 13, 16, 10, 34] . However, these methods often struggle to provide precise uncertainty estimations for the Q-value [15] . Stabilization methods can still lead to Q-value overestimation [3] , while inaccurate variance estimation can worsen this problem. Furthermore, even if the Q-value is not overestimated, there is still a risk of it being overly pessimistic or constrained by the performance of the behavior policy when using in-distribution-only training.In this subsection, we elucidate the process of learning the distribution of Q-values based on the consistency model, and outline the technique for estimating the uncertainty of actions and identifying risky actions. We have give a further demonstration on the performance of the consistency model and efficiency of the uncertainty estimation in Appendix G.2 and Appendix G.3.Trajectory-level truncated Q-value. We chose to estimate the Q-value distribution of the behavior policy instead of the learning policy because they share a similar set of high-uncertainty actions [5] . Using the behavior policy's Q-value distribution has several advantages. First, the behavior policy's Q-value dataset comes from the true dataset, ensuring high-quality distribution learning. In contrast, the learning policy's Q-value is unknown, counterfactually learned, and often noisy and biased, leading to poor data quality and biased distribution learning. Second, using the behavior policy's Q-value distribution to identify high-uncertainty actions does not force the learning policy's target Q-value to align with that of the behavior policy.To gain insights into the Q-value distribution of the behavior policy, we first need the raw Q-value data. The calculation of the Q-value operates at the trajectory level, represented as τ = (s 0 , a 0 , s 1 , a 1 , ...), with an infinite horizon (see Eq.1). In the context of offline RL, our training relies on the dataset D produced by the behavior policy. This dataset consists of trajectories generated by the behavior policy, which is the only available trajectory-level data. However, the trajectory-level data from the behavior policy often faces a significant challenge: sparsity. This issue becomes even more pronounced when dealing with low-quality behavior policies, as the generated trajectories tend to be sporadic and do not adequately cover the entire state-action space S × A, especially the high reward region.To address this pervasive issue of sparsity, as well as the infinite summation in Eq.1, we present a novel approach aimed at enhancing sample efficiency. Our proposed solution involves the utilization of truncated trajectories to ameliorate the sparsity conundrum and avoid infinite summation. By employing a kstep sliding window of width T , we systematically traverse the original trajectories, isolating segments within the window to compute the truncated Q-value (as depicted in Figure A.1). For instance, considering the initiation point of the i-th step sliding window as (s i , a i ), by setting T = i + k, we derive the truncated Q-value of this starting point as follows:EQUATIONThe truncation of Q-values can occur either through sliding window mechanisms or task terminations. When truncation happens due to termination, the Q-value from Eq.5 is equivalent to the true Q-value,Q π β T (•, •) ≡ Q π β (•, •).In contrast, if truncation results from window blocking, our theoretical analysis in Theorem 4.1 confirms that the distribution of truncated Q-values has properties similar to those of the true Q-value distribution.Using a k-step sliding window does not compromise the consistency of the trajectory, owing to the inherent memory-less Markov property in RL. This strategic truncation allows for the extraction of truncated Q-values, which can improve sample efficiency, especially for long trajectories. Moreover, this approach highlights actions with potential high Q-values, as actions from lengthy trajectories-those with many successful interactions-are encountered more often during Q-distribution training. Consequently, the uncertainty of these actions is lower, reducing the likelihood of them being overly pessimistic.Learn the distribution of Q-value. In distributional RL, the learning of Q-value distributions is typically achieved through Gaussian neural networks [35, 36] , Gaussian processes [37, 38] , or categorical parameterization [39] . However, these methods often suffer from low precision representation of Q-value distributions, particularly in high-dimensional spaces. Moreover, straightforward replacement of true Q-value distributions with ensembles or bootstraps can lead to reduced accuracy in uncertainty estimation(a critical aspect in offline reinforcement learning [4] ), or impose significant computational burdens [8, 7] .The idea of diffusing the original distribution using random noise has rendered the diffusion model a potent and high-fidelity distribution learner. However, it has limitations when estimating uncertainty.Sampling with a diffusion model requires a multi-step forward diffusion process to ensure sample quality. Unfortunately, this iterative process can compromise the accuracy of uncertainty estimates by introducing significant fluctuations and noise into the Q-value uncertainty. For a detailed discussion, see Appendix A.2.To address this issue, we suggest using the consistency model [20] to learn the Q-value distribution. The consistency model allows for one-step sampling, like other generative models, which reduces the randomness found in the multi-step sampling of diffusion models. This results in a more robust uncertainty estimation. Furthermore, the consistency feature, as explained in Theorem 4.2, accurately captures how changes in actions affect the variance of the final bootstrap samples, making Q-value uncertainty more sensitive to out-of-distribution (OOD) actions compared to the diffusion model. Additionally, the fast-sampling process of the consistency model improves QDQ's efficiency. While there may be some quality loss in restoring real samples, this is negligible for QDQ since it only calculates uncertainty based on the variance of the bootstrap samples, not the absolute Q-value of the sampled samples. Overall, the consistency model is an ideal distribution learner for uncertainty estimation due to its reliability, high-fidelity, ease of training, and faster sampling.Once we derive the truncated Q dataset D Q , we train a conditional consistency model, denoted by f θ (x T , T |(s, a)), which approximates the distribution of Q-values. Since the consistency model aligns with one-step sampling, we can easily sample multiple Q-values for each action using the consistency model. Suppose we draw n prior noise {x T1 , xT2 , • • • , xTn } from the initial noise distribution N (0, T 2 ), and denoise the prior samples by the consistency one-step forward process:xϵi = f θ (x Ti , T i |(s, a)), i = 1, 2, • • • , n.Then the variance of these Q-values, derived byEQUATIONcan be used to gauge the uncertainty of Q(s, a).Recover Q-value function. We propose an uncertainty-aware optimization objective L uw (Q) to penalize Q-value for OOD actions as well as to avoid too conservative Q-value learning for indistribution areas. The uncertainty-aware learning objective for Q-value function Q θ (s, a) is :EQUATIONIn Eq.7, L(Q θ )H represents the classic Bellman residual defined in Eq.2. This residual is used in online RL and encourages optimistic optimization of the Q-value. In contrast, L(Qθ) L is a pessimistic Bellman residual based on the uncertainty-penalized Q target Q L (s ′ , a ′ ), defined asEQUATIONIn Eq.8,H Q (a ′ |s ′ ) = V (X ϵ |(s ′ , a ′ ))represents the uncertainty estimate of the Q-value for action a ′ . The set U(Q) includes actions that may be out-of-distribution (OOD). We use the upper β-quantile H β Q (a ′ |s ′ ) of the uncertainty estimate on actions taken by the learning policy as the threshold for forming U(Q). Additionally, we incorporate the quantile parameter β as a robust weighting factor for the unpenalized Q-target value. This helps control the estimation error of uncertainty and enhances the robustness of the learning objective. We can also set a free weighting factor, but we use β to reduce the number of hyperparameters.Improve the learning policy. The optimization of learning policy follows the classic online RL paradigm:EQUATIONIn Eq.9, an entropy term is introduced to further stabilize the volatile learning process of Q-value function. For datasets with a wide distribution, we can simply set the penalization factor γ to zero, which can further enhance performance. Furthermore, other policy learning objectives, such as the AWR policy objective [40] , can also be flexibly used within the QDQ framework, especially for the goal conditioned task like Antmaze.We outline the entire learning process of QDQ in Algorithm 1. In Section 4, Theorems 4.3 and 4.4 show that QDQ penalizes the OOD region based on uncertainty while ensuring that the Q-value function in the in-distribution region is close to the optimal Q-value. This alignment is the main goal of offline RL. f η , Q networks {Q θ1 , Q θ2 }, actor π ϕ , target networks {Q θ ′ 1 , Q θ ′ 2 }, target actor π ϕ ′ . Q-distribution learning: Calculate Q dataset D Q = {Q π βT (s, a)} scanning each trajectory τ ∈ D by Eq.5.for each gradient step do Sample minibatch of q π β T (s, a) ∼ D Q Update η minimizing consistency distillation loss in Eq.( 7) [20] end for for each gradient step do Sample mini-batch of transitions (s, a, r, s ′ ) ∼ D Updating Q-function:Update θ = (θ 1 , θ 2 ) minimizing L uw (Q θ ) in Eq.7Updating policy: Update ϕ minimizing L ϕ (π) in Eq.9 Update Target Networks:ϕ ′ ← κϕ + (1 -κ)ϕ ′ ; θ ′ i ← κθ i + (1 -κ)θ ′ i , i = 1, 2 end for 
theoretical analysis: 
 In this section, we provide a theoretical analysis of QDQ. The first theorem states that if T is sufficiently large, the distribution of Q π β T does not significantly differ from the true distribution of Q πβ . This shows that our sliding window-based truncated Q-value distribution converges to the true Q-value distribution, ensuring accurate uncertainty estimation. A detailed proof can be found in Appendix B. Theorem 4.1 (Informal). Under some mildly condition, the truncated Q-value QEQUATIONIn Theorem 4.2, we analyze why the consistency model is suitable for estimating uncertainty. Our analysis shows that Q-value uncertainty is more sensitive to actions. This sensitivity helps in detecting out-of-distribution (OOD) actions. A detailed statement of the theorem and its proof can be found in Appendix C. Theorem 4.2 (Informal). Following the assumptions as in [20] , f θ (x, T |(s, a)) is L-Lipschitz. We also assume the truncated Q-value is bounded by H. The action a broadly influences V (X ϵ |(s, a)) by:| ∂var(Xϵ) ∂a | = O(L 2 T √ log n)1.In Theorem 4.3, we give theoretical analysis that the uncertainty-aware learning objective in Eq.7 can converge and the details can be found in Appendix D. Theorem 4.3 (Informal). The Q-value function of QDQ can converge to a fixed point of the Bellman equation:Q(s, a) = F Q(s, a), where the Bellman operator F Q(s, a) is defined as:EQUATIONTheorem 4.4 shows that QDQ penalizes the OOD region by uncertainty while ensuring that the Q-value function in the in-distribution region is close to the optimal Q-value, which is the goal of offline RL. Theorem 4.4 (Informal). Under mild conditions, with probability 1 -η we haveEQUATIONwhere Q ∆ is learned by the uncertainty-aware loss in Eq.7, ϵ is error rate related to the difference between the classical Bellman operator BQ and the QDQ bellman operator F Q.The optimal Q-value, Q ∆ , derived by the QDQ algorithm can closely approximate the optimal Q-value function, Q * , benefiting from the balanced approach of the QDQ algorithm that avoids excessive pessimism for in-distribution areas. Both the value ϵ and η are small and more details in Appendix E. 
experiments: 
 In this section, we first delve into the experimental performance of QDQ using the D4RL benchmarks [41] . Subsequently, we conduct a concise analysis of parameter settings, focusing on hyperparameter tuning across various tasks. For detailed implementation, we refer to Appendix G.We evaluate the proposed QDQ algorithm on the D4RL Gym-MuJoCo and AntMaze tasks. We compare it with several strong state-of-the-art (SOTA) model-free methods: behavioral cloning (BC), BCQ [42] , DT [43] , AWAC [44] , Onestep RL [45] , TD3+BC [46] , CQL [3] , and IQL [10] .We also include UWAC [7] , EDAC [9] , and PBRL [14] , which use uncertainty to pessimistically adjust the Q-value function, as well as MCQ [13] , which introduces mild constraints to the Q-value function. The experimental results for the baselines reported in this paper are derived from the original experiments conducted by the authors or from replication of their official code. The reported values are normalized scores defined in D4RL [41] . Table 1 shows the performance comparison between QDQ and the baselines across Gym-MuJoCo tasks, highlighting QDQ's competitive edge in almost all tasks. Notably, QDQ excels on datasets with wide distributions, such as medium and medium-replay datasets. In these cases, QDQ effectively avoids the problem of over-penalizing Q-values. By balancing between being too conservative and actively exploring to find the optimal Q-value function through dynamic programming, QDQ gradually converges toward the optimal Q-value, as supported by Theorem 4.4. Table 2 presents the performance comparison between QDQ and selected baselines3 across AntMaze tasks, highlighting QDQ's commendable performance. While QDQ focuses on reducing overly pessimistic estimations, it does not compromise its performance on narrow datasets. This is evident in its competitive results on the medium-expert dataset in Table 1 , as well as its performance on AntMaze tasks. Notably, QDQ outperforms SOTA methods on several datasets. This success is due to the inherent flexibility of the QDQ algorithm. By allowing for flexible hyperparameter control and seamless integration with various policy optimization methods, QDQ achieves a synergistic performance enhancement.The uncertainty-aware loss parameter α. The parameter α is crucial for balancing the dominance between optimistic and pessimistic updates of the Q-value (Eq.7). A higher α value skews updates toward the optimistic side, and we choose a higher α when the dataset or task is expected to be highly robust. However, the setting of α is also influenced by the pessimism of the Q target defined in Eq.8.For a more pessimistic Q target value, we can choose a larger α. Interestingly, both the theoretical analyses in Theorem 4.3 and Theorem 4.4 and empirical parameter tuning suggest that variability in α across tasks is minimal, with a typical value around 0.95.The uncertainty related parameter β. The parameter β influences both the partitioning of high uncertainty sets and acts as a relaxation variable to control uncertainty estimation errors. When dealing with a narrow action space or a sensitive task (such as the hopper task), the value of β should be smaller. In these cases, the Q-value is more likely to select OOD actions, increasing the risk of overestimation. This means we face greater uncertainty (Eq.8) and need to minimize overestimation errors. Therefore, we require stricter criteria to ensure actions are in-distribution and penalize the Q-values of OOD points more heavily. A detailed analysis of how to determine the value of β can be found in Appendix G.3.The entropy parameter γ. The γ term in Eq.9 stabilizes the learning of a simple Gaussian policy, especially for action-sensitive and narrower distribution tasks. When the dataset has a wide distribution or the task shows high robustness to actions (such as in the half-cheetah task), the Q-value function generalizes better across the action space. In these cases, we can set a more lenient requirement for actions, keeping the value of γ as small as possible or even at 0. However, when the dataset is narrow (e.g., in the AntMaze task) or when the task is sensitive to changes in actions (like in the hopper or maze tasks, where small deviations can lead to failure), a larger value of γ is necessary. For these tasks, a simple Gaussian policy can easily sample risky actions, as it fits a single-mode policy. Nonetheless, experimental results indicate that the sensitivity of the γ parameter is not very high. In fact, γ in Eq.9 is relatively small compared to the Q-value, primarily to stabilize training and prevent instability in Gaussian policy action sampling. See Appendix G.7 for more details. 
related works: 
 Restrict policy deviate from OOD areas. The distribution mismatch between the behavior policy and the learning policy can be overcome if the learning policy share the same support with the behavior policy. One approach involves explicit distribution matching constraints, where the learning policy is encouraged to align with the behavior policy by minimizing the distance between their distributions. This includes techniques based on KL-divergence [47, 48, 40, 44, 46] , Jensen-Shannon divergence [49] , and Wasserstein distance [47, 49] . Another line of research aims to alleviate the overly conservative nature of distribution matching constraints by incorporating distribution support constraints. These methods employ techniques such as Maximum Mean Discrepancy (MMD) distance [5] , learning behavior density functions using implicit [50] or explicit [51] methods, or measuring the geometric distance between actions generated by the learning and behavior policies [52] .In addition to explicit constraint methods, implicit constraints can also be implemented by learning a behavior policy sampler using techniques like Conditional Variational Autoencoders (CVAE) [42, 53, 50, 54] , Autoregressive Generative Model [55] , Generative Adversarial Networks (GAN) [49] , normalized flow models [56] , or diffusion models [57, 58, 11, 59, 11, 60] .Pessimistic Q-value optimization. Pessimistic Q-value methods offer a direct approach to address the issue of Q-value function overestimation, particularly when policy control fails despite the learning policy closely matching the behavior policy [3] . A promising approach to pessimistic Q-value estimation involves estimating uncertainty over the action space, as OOD actions typically exhibit high uncertainty. However, accurately quantifying uncertainty poses a challenge, especially with high-capacity function approximators like neural networks [4] . Techniques such as ensemble or bootstrap methods have been employed to estimate multiple Q-values, providing a proxy for uncertainty through Q-value variance [7, 9, 14] , importance ratio [61, 62] or approximate Lower Confidence Bounds (LCB) for OOD regions [8, 9] . Other methods focus on estimating the LCB of Q-values through quantile regression [63, 34] , expectile regression [10, 11] , or tail risk measurement such as Conditional Value at Risk (cVAR) [12] . Alternatively, some approaches seek to pessimistically estimate Q-values based on the behavior policy, aiming to underestimate Q-values under the learning policy distribution while maximizing Q-values under the behavior policy distribution [3, 13, 64, 65] . Another category of Q-value constraint methods involves learning Q-values only within the in-sample [16, 17, 18, 19] , capturing only in-sample patterns and avoid OOD risk. Furthermore, Q-value functions can be replaced by safe planning methods used in model-based RL, such as planning with diffusion models [66] or trajectory-level prediction using Transformers [43] . However, ensemble estimation of uncertainty may tend to underestimate true uncertainty, while quantile estimation methods are sensitive to Q-distribution recovery. In-sample methods may also be limited by the performance of the behavior policy. 
conclusion: 
 We introduce QDQ, a novel framework rendering pessimistic Q-value in OOD areas by uncertainty estimation. Our approach leverages the consistency model to robustly estimate the uncertainty of Q-values. By employing this uncertainty information, QDQ can apply a judicious penalty to Q-values, mitigating the overly conservative nature encountered in previous pessimistic Q-value methods. Additionally, to enhance optimistic Q-learning within in-distribution areas, we introduce an uncertainty-aware learning objective for Q optimization. Both theoretical analyses and experimental evaluations demonstrate the effectiveness of QDQ. Several avenues for future research exist, including embedding QDQ into goal-conditioned tasks, enhancing exploration in online RL by efficient uncertainty estimation. We hope our work will inspire further advancements in offline reinforcement learning. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA]Justification: [TODO]Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: [TODO]Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: [TODO]Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A Further discussion about uncertainty estimation of Q-value by Q-distribution.A.1 The Q-value dataset enhancement with sidling window.In Section 3.1, we analyze the challenges associated with deriving the Q-value dataset. We propose the k-step sliding window method to improve sample efficiency at the trajectory level. The implementation details of the k-step sliding window within an entire trajectory are illustrated in This sliding window framework not only facilitates the expansion of Q-value data but also prevents the state-action pairs from becoming overly dense, thereby mitigating the risk of Q-value homogenization.In continuous state-action spaces, Q-values tend to be homogeneity when state and action are close in a trajectory, which may hinder subsequent learning of the Q-value distribution.Step 1Step 2 Suppose we use the score matching method proposed in [30] to learn a conditional score network s θ (x, σ|s, a) to approximate the score function of the Q-value distribution p Q (x). Then we use the annealed Langevin dynamics as in [30] to sample from the learned Q-value distribution. For x 0 ∼ π(x) from some arbitrary prior distribution π(x), the denoised sample is:x i+1 = x i + ϵ • s θ (x i , σ|s, a) + √ 2ϵz i , i = 0, 1, 2, • • • , T. z i ∼ N (0, 1). (A.1)The distribution of x i+1 equals p Q (x) when ϵ → 0 and T → ∞.Our primary approach to quantify the uncertainty of the action with respect to the Q-value is to evaluate the spread of the sampled Q-values for each (s, a). We then use the gradient of the Q-value samples with respect to the action to assess their sensitivity to changes in action. By iteratively deriving the gradient over the sampling chain of length T , as shown in Eq.A.1, we approximate the following outcome:∂x T ∂a = T i=1 c i ϵ i ∂ i s θ (•|s, a) ∂a i . (A.2)From Eq.A.2, the impact of actions on Q-value samples learned from the diffusion model shows considerable instability, especially during the iterative gradient-solving process for the score network s θ (x, σ|s, a). This instability often leads to gradient vanishing or exploding. While the diffusion model effectively recovers the Q-value distribution with high precision, the multi-step sampling process introduces significant fluctuations and noise, making it difficult to accurately assess the uncertainty of Q-values for different actions.Additionally, as different prior information may yield the same target sample, and this stochastic correlation also introduces an uncontrollable impact on uncertainty of the Q-value. However, the effect of prior sample variance on the uncertainty of the sampled Q-value can not be quantified with the diffusion model. So we can not guarantee if the absolute influence on the Q-value uncertainty is from the action, then the performance of the uncertainty can not be guaranteed.B Convergence of the truncated Q-value distribution.We first give a formal introduction of Theorem 4.1 as Theorem B.1.Theorem B.1. Suppose the true distribution of Q-value w.r.t the behavior policy π β is defined asF Q π β (x). By Eq.5, we derive the truncated Q-value Qπ β T , denote the distribution of the truncated Q-value is F Q π β T (x).Assume the true Q-value is finite over the state and action space,then Qπ β T converge in-distribution to the true true Q-value Q π β . F Q π β T (x) → F Q π β (x), T → +∞. (B.1)We will show the distribution of the truncated Q-value Qπ βT has the same property with the true distribution of the Q-value Qπ βT and give a brief proof of Theorem 4.1. Suppose the state-action space determined by the offline RL dataset D is Ω D = S D × A D . Note that Q π β can be seen as an r.v. defined as:(Ω D , F , P • π β ) Q π β ---→ (R, B(R), (P • π β ) • (Q π β ) -1), where F is the σ-field on Ω D , P • π β is the probability measure on Ω D and P is the transition probability measure over state, B(R) is the Borel σ-field on R, (P• π β ) • (Q π β ) -1 = (P • π β )((Q π β ) -1 ) is the push forward probability measure on R. Same as Q π β , Q π βT can also be seen as a r.v.. Then we show that Qπ β T converge to Q π β almost surely: Q π β T a.s.--→ Q π β , when sending T to infinity.D τ = {τ k |τ k = (s k0 , a k0 , r k0 , s k1 , a k1 , r k1 , • • • )}.Then for any trajectory τ k ∈ D τ , the true Q-value w.r.t this trajectory can be rewrite without the expectationas:Q π (s k0 , a k0 ) = ∞ j=1 γ j-1 r(s kj , a kj ).Truncating Q-value by termination situation. If the terminal occurs at step k t of the trajectory τ k , then we have r(s kj , a kj ) = 0, for j > k t . Then the truncated Q-value is identical the true Q-value:Q π β kt ≡ Q π (s k0 , a k0). So the distribution of these two distribution is same for the situation when the truncation is happened due to terminal of the task.Truncating Q-value by sliding window situation. If the Q-value is truncated by a sliding window as shown in Figure A.1, then for a specific kstep sliding widow of width T with starting point (s i , a i ) over a trajectory τ , we have Q π β T (s i , a i ) = T m=i γ m-1 r(s m , a m ).B n (ξ) := ∞ ∪ T =n A n (ξ), where A T (ξ) = {(s, a) : |Q π β T (s, a)-Q π β (s, a)| > ξ}. By the definition of Q-value, we have B m (ξ) = A m (ξ), as A T (ξ) ⊃ A T +1 (ξ) ⊃ A T +2 (ξ) ⊃ • • • is decreasing. So B n (ξ)|Q π β n (s, a) -Q π β (s, a)| = ∞ k=n+1 γ n r(s k , a k ) ≤ c ∞ k=n+1 γ n = γ n 1 -γ • c (B.3)So A n (ξ) → ∅ when sending n to infinity, as the discount factor γ < 1 by definition.Then by Eq.B.2, we have lim m→∞ P(B n (ξ)) = 0, ∀ξ > 0. In probability theory, this is equivalent toQ π β T a.s. --→ Q π β . Furthermore, if Q π β T converge to Q π β almost surely, then Q π βT also converge to Q π β in probability, and in-distribution. Hence, we finish the proof. Remark B.1. Theorem 4.1 suggests that for arbitrary small ϵ, there exists a sufficiently large T , such that|F Q π β T (x) -F Q π β (x)| < ϵ.Given that the impact of rewards diminishes exponentially after as the increasing of trajectory length, it is unnecessary to set T to an excessively large value. It's important to remember that the goal of the Q dataset is to learn the Q-distribution and assess the uncertainty of different actions. Therefore, the absolute magnitude of Q is not crucial. Additionally, using too many future steps may introduce significant uncertainty into the Q-value, as predictions for the distant future can be inaccurate. Remark B.2. During the proof, the specific starting point of the sliding window holds no significance; rather, our focus lies solely on the length of the window. This is primarily due to the Markovian nature of trajectories in RL, where the current state and action are unaffected by previous ones and adhere to a memoryless property. Consequently, the starting point of the sliding window exerts minimal influence on the computation of Q.C Robustness of consistency model for uncertainty measure.The formal introduction of Theorem 4.2 is shown in Theorem C.1. Theorem C.1. Follow the assumptions in [20] , we assume f θ (x, T |(s, a)) is L-Lipschitz.By using the partial gradient to analyze the influence of prior samples xT , time step T and action a to the variance of the denoised sample var(X ϵ ), with high probability, we have:(1) Prior noise influence the variance V (X ϵ ) is bounded by:| ∂V (Xϵ) ∂ xT | = O(L 2 T n -1 T log(n))1.(2) Time step T influence the variance V (X ϵ ) is bounded by:| ∂var(Xϵ) ∂T | = O(L 2 T √ log n).(3) Action a influence the variance V (X ϵ ) is bounded by:| ∂var(Xϵ) ∂a | = O(L 2 T √ log n)1.As discussed in Section 3.1 and A.2, while the diffusion model has shown great success in learning distributions and generating samples, it is less suitable for scenarios where the influence of certain parameters on sample uncertainty must be guaranteed. When estimating uncertainty, we sample multiple Q-values for each (s, a) pair, and the standard deviation of these sampled Q-values measures the uncertainty. Therefore, it is crucial that the sample spread is sensitive to changes in action to accurately judge OOD actions.However, the multi-step forward denoising process of the diffusion model undermines the influence of actions on the sampled Q-values, compromising the robustness of uncertainty estimation. Additionally, the lack of a one-to-one correspondence between prior information and target samples prevents the cancellation of prior effects on the Q-value distribution through repeated sampling.In contrast, the consistency model addresses these challenges. It not only overcomes the aforementioned issues, but its one-step sampling significantly enhances efficiency. In the following theoretical analysis, we will demonstrate the robustness of the consistency model in estimating uncertainty.As described in [20] and Section 2.2, a consistency model f θ (x, t) is trained to mapping the prior noise on any trajectory of PF ODE to the trajectory's origin x ϵ by: f θ (x, t) = x ϵ , given x and x ϵ belong to the same PF ODE trajectory. f θ (x, t) is defined as in Eq.4.Suppose we trained a conditional consistency model f θ (x, t|s, a) with the truncated Q-value datasetD Q = {Q π βT (s, a)}, following the one-step sampling of consistency model, we first initial n noise xTi ∼ N (0, T 2 ), i = 1, 2, ..., n, then do one-step forward denosing and derive n sample xϵi = f θ (x Ti , T |s, a), where T is a fixed time step. The variance based on the Q sample is:V (X ϵ ) = 1 n -1 n i=1 f θ (x Ti , T |(s, a)) - 1 n n i=1 f θ (x Ti , T |(s, a)) 2 . (C.1)Next, we derive the gradient of V (X ϵ ) w.r.t xTi , T , a, and check how change in these variable influence the variance. As state s is always in-distribution during offline RL training process and has little influence on the uncertainty of the sampled Q-value, we skip the analysis.Following [20] , we assume that f θ (x, t|s, a) is L-Lipschitz bounded, i.e., for any x and y,∥f θ (x, t|(s, a)) -f θ (y, t|(s, a))∥ 2 ≤ L ∥x -y∥ 2 .Gradient of the prior xTi .Proof. Let e i be the square difference of the i-th prior xTi :e i (x Ti ) := f θ (x Ti , T |(s, a)) - 1 n n i=1 f θ (x Ti , T |(s, a)) 2 . (C.2)Then we have:∂V (X ϵ ) ∂ xTi = 1 n -1 n j=1 ∂e j (x Tj ) ∂ xTi (C.3) If j = i, ∂e j (x Tj ) ∂ xTi =2 f θ (x Ti , T |(s, a)) - 1 n n i=1 f θ (x Ti , T |(s, a)) • ∂f θ (x Ti , T |(s, a)) ∂ xTi - 1 n ∂f θ (x Ti , T |(s, a)) ∂ xTi =2 f θ (x Ti , T |(s, a)) - 1 n n i=1 f θ (x Ti , T |(s, a)) • n -1 n ∂f θ (x Ti , T |(s, a)) ∂ xTi . (C.4) If j ̸ = i, ∂e j (x Tj ) ∂ xTi =2 f θ (x Tj , T |(s, a)) - 1 n n i=1 f θ (x Ti , T |(s, a)) • 0 - 1 n ∂f θ (x Ti , T |(s, a)) ∂ xTi =2 f θ (x Tj , T |(s, a)) - 1 n n i=1 f θ (x Ti , T |(s, a)) • - 1 n ∂f θ (x Ti , T |(s, a)) ∂ xTi . (C.5)Thus, plugging Eq.C.4 and Eq.C.5 into ∂V (Xϵ)∂ xT i yields ∂V (X ϵ ) ∂ xTi = 1 n -1 n j=1 ∂e j (x Tj ) ∂ xTi = 2 n(n -1) ∂f θ (x Ti , T |(s, a)) ∂ xTi   (n -1)f θ (x Ti , T |(s, a)) - j̸ =i f θ (x Tj , T |(s, a))   . (C.6) As f θ (x, t|s, a) is L-Lipschitz bounded, we have | ∂V (X ϵ ) ∂ xTi | ≤ 2 n(n -1) | ∂f θ (x Ti , T |(s, a)) ∂ xTi | j̸ =i |f θ (x Ti , T |(s, a)) -f θ (x Tj , T |(s, a))| ≤ 2 n(n -1) • L • L j̸ =i |x Ti -xTj | ≤ 2 n • L 2 • c log n • T, (C.7)where |x Ti -xTj |, ∀j ̸ = i can be bounded by cT √ log n due to xTi ∼ N (0, T 2 ) with probability at least 1 -n -1 . Denote the constant with c p and apply the previous process to all the prior samples completes the proof.Gradient of the time step T .Proof.∂V (X ϵ ) ∂T = 1 n -1 n j=1 ∂e j (T ) ∂T . (C.8)Taking partial gradient of e i (T ) w.r.t T for any j ∈ {1, 2, • • • , n}, we obtainEQUATIONwith probability at least 1 -n -1 , since |x Ti -xTj |, ∀j ̸ = i can be bounded by cT √ log n due to xTi ∼ N (0, T 2 ) with probability at least 1 -n -1 .Plugging Eq.C.9 into Eq.C.8 yields∂V (X ϵ ) ∂T ≤ 4cL 2 T log n. (C.10)Gradient of the action a.For the gradient of ∂V (Xϵ) ∂a , we just need to take partial gradient of V (X ϵ ) for each dimmension of the action a = {a 1 , a 2 , • • • , a m } by ∂V (Xϵ) ∂ai . The result is same as those we got in Eq.C.10.Then take the vector form, we have| ∂V (Xϵ) ∂a | = O(L 2 T √ log n) • 1, which finish the proof. Remark C.1. Theorem 4.2 elucidates the diminishing impact of random prior on the variance of the denoised Q-value as the sample size increases. Leveraging the consistency of sampling, we mitigate concerns regarding the influence of a priori samples on the uncertainty of final target samples. Given the fixed sampling step size T , we also address concerns about its effect on the uncertainty of Q samples. However, for a thorough analysis, we still include the gradient analysis of V (X ϵ ) against T . Remark C.2. The influence of actions on sample variance depends on factors like the Lipschitz factor and sample size. As the sample size increases, the impact of actions on Q sample variance does not diminish; instead, it becomes more sensitive. Larger Q sample variance is more likely to occur when OOD actions are present. Consequently, the consistency model proves to be a reliable approach for Q-value sampling and uncertainty estimation. Remark C.3. Although experiments in [20] show that the performance of the consistency model is less competitive compared to the diffusion model or adversarial generators like GANs, these findings have minimal relevance to our method. Our primary focus is not on the absolute accuracy of the sampled Q-values but on the sensitivity of Q sample dispersion to OOD actions and its ability to effectively capture uncertainty in such cases. Additionally, the high sampling efficiency achieved through one-step sampling compensates for the minor performance discrepancies of the consistency model.We first give a formal introduction of Theorem 4.3 below. Theorem D.1. Updating Q-value Q θ (s, a) via the uncertainty-aware objective in Eq.7 is equivalent to minimizing the L 2 -norm of Bellman residuals:E s∼P D (s),a∼π(a|s) [Q(s, a) -F Q(s, a)] 2, where the Bellman operator F Q(s, a) is defined as:F Q(s, a) := r(s, a) + γE s ′ ∼P D (s ′ ) {max a ′ [αQ(s ′ , a ′ ) + (1 -α)Q L (s ′ , a ′ )]}. (D.1)In addition, assume1 H Q (a ′ |s ′ ) 1 (a ′ ∈U (Q)) < β.Then the Bellman operator F Q is cγ-contraction operator in the L ∞ norm, where c < 1. The Q-value function Q θ (s, a) can converge to a fixed point by value iteration method.We will show the the uncertainty-aware learning objective is equivalent to minimized the Bellman equation defined by a specific Bellman operator F Q firstly.Then the we proof the Bellman operator F Q is cγ-contraction operator in the L ∞ norm, where c < 1. The Q-value function Q θ (s, a) can converge to a fixed point by the value iteration method.Recall the Q-value is optimized by the uncertainty-aware learning objective by Eq.7:EQUATIONwhereL(Q) H :=E s∼P D (s),a∼π(a|s) [Q θ (s, a) -(BQ)(s, a)] 2 =E s∼P D (s),a ′ ∼π(a|s) [Q θ (s, a) -(r(s, a) + γE s ′ ∼P D (s ′ ) [max a ′ Q θ (s ′ , a ′ )])] 2 , (D.3) L(Q) L :=E s∼P D (s),a∼π(a|s) [Q θ (s, a) -(BQ) L (s, a)] 2 =E s∼P D (s),a ′ ∼π(a|s) [Q θ (s, a) -(r(s, a) + γE s ′ ∼P D (s ′ ) [max a ′ Q L (s ′ , a ′ )])] 2 . (D.4)The uncertainty penalized Q target Q L (s ′ , a ′ ) is defined as:Q L (s ′ , a ′ ) = 1 H Q (a ′ |s ′ ) Q θ (s ′ , a ′ )1 (a ′ ∈U (Q)) + βQ θ (s ′ , a ′ )1 (a ′ / ∈U (Q)) . (D.5)For simplicity, we ignore the estimation parameter of Q θ (s, a) and just use Q(s, a) in the following proof.We can just take the uncertainty-aware loss in Eq.7 as a plain regression like loss and we have:αL(Q) H + (1 -α)L(Q) L =α[Q(s, a) -(BQ)(s, a)] 2 + (1 -α)[Q(s, a) -(BQ) L (s, a)] 2 =α[Q(s, a) 2 -2Q(s, a)(BQ)(s, a) + ((BQ)(s, a)) 2 ] +(1 -α)[Q(s, a) 2 -2Q(s, a)(BQ) L (s, a) + ((BQ) L (s, a)) 2 ] =αQ(s, a) 2 -α2Q(s, a)(BQ)(s, a) + α((BQ)(s, a)) 2 +(1 -α)Q(s, a) 2 -(1 -α)2Q(s, a)(BQ) L (s, a) + (1 -α)((BQ) L (s, a)) 2 =Q(s, a) 2 -2Q(s, a)[α(BQ)(s, a) + (1 -α)(BQ) L (s, a)] +α((BQ)(s, a)) 2 + (1 -α)((BQ) L (s, a)) 2 =[Q(s, a) -(α(BQ)(s, a) + (1 -α)(BQ) L (s, a))] 2 + C, (D.6)where C is a factor that not related to Q(s, a), since the value of (BQ) L (s, a) and (BQ)(s, a) are fixed as we update Q. By the definition of (BQ)(s, a) and (BQ) L (s, a) in Eq.D.3 and Eq.D.4, the uncertainty-aware learning is equivalent to minimized the following Bellman equation:E s∼P D (s),a∼π(a|s) [Q(s, a) -(F Q)(s, a)] 2 , (D.7)where the specific Bellman operator (F Q)(s, a) is defined in Eq.D.1. Then we finish the proof.F Q is cγ-contraction operator in the L ∞ norm.Then we give a brief proof of the convergence of the Bellman operator F Q by value interative optimization.For any Q-value function Q(s, a), Q ′ (s, a), define I = |F Q(s, a) -F Q ′ (s, a)|, we haveI =|r(s, a) + γE s ′ ∼P D (s ′ ) {max a ′ [αQ(s ′ , a ′ ) + (1 -α)Q L (s ′ , a ′ )]}- +(r(s, a) + γE s ′ ∼P D (s ′ ) {max a ′ [αQ ′ (s ′ , a ′ ) + (1 -α)Q ′ L (s ′ , a ′ )]})| =γ|E s ′ ∼P D (s ′ ) max a ′ {αQ(s ′ , a ′ ) + (1 -α)Q L (s ′ , a ′ ) -[αQ ′ (s ′ , a ′ ) + (1 -α)Q ′ L (s ′ , a ′ )]}| ≤γE s ′ ∼P D (s ′ ) | max a ′ {αQ(s ′ , a ′ ) + (1 -α)Q L (s ′ , a ′ ) -[αQ ′ (s ′ , a ′ ) + (1 -α)Q ′ L (s ′ , a ′ )]}| ≤γ max s ′ | max a ′ {αQ(s ′ , a ′ ) + (1 -α)Q L (s ′ , a ′ ) -[αQ ′ (s ′ , a ′ ) + (1 -α)Q ′ L (s ′ , a ′ )]}| ≤γ max s ′ max a ′ |αQ(s ′ , a ′ ) + (1 -α)Q L (s ′ , a ′ ) -[αQ ′ (s ′ , a ′ ) + (1 -α)Q ′ L (s ′ , a ′ )]| =γ max s ′ max a ′ |α(Q(s ′ , a ′ ) -Q ′ (s ′ , a ′ )) + (1 -α)(Q L (s ′ , a ′ ) -Q ′ L (s ′ , a ′ ))| =γ max s ′ max a ′ |α(Q(s ′ , a ′ ) -Q ′ (s ′ , a ′ )) + (1 -α)(βQ(s ′ , a ′ ) -βQ ′ (s ′ , a ′ ))1 (a ′ / ∈U (Q)) +(1 -α)( 1 H Q (a ′ |s ′ ) Q(s ′ , a ′ ) - 1 H Q (a ′ |s ′ ) Q ′ (s ′ , a ′ ))1 (a ′ ∈U (Q)) | =γ max s ′ max a ′ |α(Q(s ′ , a ′ ) -Q ′ (s ′ , a ′ )) + (1 -α)β(Q(s ′ , a ′ ) -Q ′ (s ′ , a ′ ))1 (a ′ / ∈U (Q)) + (1 -α) H Q (a ′ |s ′ ) (Q(s ′ , a ′ ) -Q ′ (s ′ , a ′ ))1 (a ′ ∈U (Q)) | =γ max s ′ max a ′ |α(Q(s ′ , a ′ ) -Q ′ (s ′ , a ′ ))1 (a ′ / ∈U (Q)) + (1 -α)β(Q(s ′ , a ′ ) -Q ′ (s ′ , a ′ ))1 (a ′ / ∈U (Q)) + α(Q(s ′ , a ′ ) -Q ′ (s ′ , a ′ ))1 (a ′ ∈U (Q)) + (1 -α) H Q (a ′ |s ′ ) (Q(s ′ , a ′ ) -Q ′ (s ′ , a ′ ))1 (a ′ ∈U (Q)) | ≤γ max s ′ max a ′ {(α + (1 -α)β)|Q(s ′ , a ′ ) -Q ′ (s ′ , a ′ )|1 (a ′ / ∈U (Q)) + (α + (1 -α) H Q (a ′ |s ′ ) )|Q(s ′ , a ′ ) -Q ′ (s ′ , a ′ )|1 (a ′ ∈U (Q)) } ≤γ max s ′ max a ′ {max{α + (1 -α)β, α + (1 -α) H Q (a ′ |s ′ ) }|Q(s ′ , a ′ ) -Q ′ (s ′ , a ′ )|} =γ max{α + (1 -α)β, α + (1 -α) H Q (a ′ |s ′ ) 1 (a ′ ∈U (Q)) }||Q(s ′ , a ′ ) -Q ′ (s ′ , a ′ )|| ∞ (D.8) Since 1 H Q (a ′ |s ′ ) 1 (a ′ ∈U (Q)) < β, we have max{α + (1 -α)β, α + (1-α) H Q (a ′ |s ′ ) } = α + (1 -α)β is always true. As β < 1, then α + (1 -α)β < α + (1 -α) < 1. Set c = α + (1 -α)β, then we have |F Q(s, a) -F Q ′ (s, a)| ≤ cγ||Q(s ′ , a ′ ) -Q ′ (s ′ , a ′ )|| ∞ , (D.9) which implies F Q is cγ-contraction operator with cγ < 1.Suppose Q ∆ is the stationary point of the Bellman equation in Eq.D.7, then it can be shown that Q iteratively updated by Eq.D.7 can converge to Q ∆ :||Q t+1 -Q ∆ || ∞ =||F Q t -F Q ∆ || ∞ ≤ cγ||Q t -Q ∆ || ∞ ≤(cγ) 2 ||Q t-1 -Q ∆ || ∞ ≤ • • • ≤ (cγ) t+1 ||Q 0 -Q ∆ || ∞ . (D.10)Sending t to infinity, we can derive Q t converge to Q ∆ , then we finish the proof. Remark D.1. The assumption1 H Q (a ′ |s ′ ) 1 (a ′ ∈U (Q))< β can be always satisfied. Roughly speaking, since a ′ ∈ U(Q), the uncertainty of Q-value H Q (a ′ |s ′ ) on this a ′ has large uncertainty due to the OOD property. Furthermore, we can scale the absolute value H Q (a ′ |s ′ ) for all the action with same factor to guarantee 1 H Q (a ′ |s ′ ) 1 (a ′ ∈U (Q)) < β without hurting the relative comparison for the uncertainty. Furthermore, experiment results have shown that1 H Q (a ′ |s ′ ) 1 (a ′ ∈U (Q)) < β is consistently satisfied without additional processing.E Performance of the Q-value function Q k (s, a) derived by QDQ.In this section, we delve into an analysis of the performance of the Q-value function derived from the QDQ algorithm. Given that the primary aim of QDQ is to mitigate the issue of excessively conservative in most pessimistic Q-value methods, our focus is directed towards scrutinizing the disparity between the optimal Q-value within the offline RL framework and the optimal Q-value function yielded by QDQ.The following is a formal version of Theorem 4.4.Theorem E.1. Suppose the optimal Q-value function over state-action space S D × A D defined by the dataset D is Q * . Then with probability at least 1 -η, the Q-value function Q ∆ learned by minimizing uncertainty-aware loss (Eq.7) can approach the optimal Q * with a small constant:Q ∆ -Q * ∞ ≤ ϵ, (E.1)where ϵ is error rate related to the difference between the classical Bellman operator BQ and the QDQ Bellman operator F Q, and η is determined by the probability thatmax a ′ {(1 -β))|Q(s ′ , a ′ )|1 (a ′ / ∈U (Q)) + (1 - 1 H Q (a ′ |s ′ ) )|Q(s ′ , a ′ )|1 (a ′ ∈U (Q)) } = max a ′ {(1 - 1 H Q (a ′ |s ′ ) )|Q(s ′ , a ′ )|1 (a ′ ∈U (Q)) }.Before the proof, we redefine some notation to make the subsequent exposition clearer.Denote state space S D be the state space defined by the distribution of dataset D, A D is the action space defined by the dataset D, and actions not belong to this space is the OOD actions. The optimal Q-value Q * on S D × A D can be derived by optimize the following Bellman equation:L(Q) :=E s∼P D (s),a∼π(a|s) [Q(s, a) -BQ(s, a)] 2 =E s∼P D (s),a ′ ∼π(a|s) [Q(s, a) -(r(s, a) + γE s ′ ∼P D (s ′ ) [max a ′ Q(s ′ , a ′ )])] 2 , (E.2)where (s, a)∈ S D × A D , Q * = BQ * .We first introduced Lemma E.1 to facilitate the proof of Theorem 4.4.Lemma E.1. For any s ∈ S D , a ∈ A D , with probability 1 -η,|BQ * (s, a) -F Q * (s, a)| ≤ γ(1 -α)(1 -β)||Q * (•, •)|| ∞ , (E.3)and with probability η,|BQ * (s, a) -F Q * (s, a)| ≤ γ(1 -α)(1 - 1 H Q * (a ′ |s ′ ) )||Q * (•, •)|| ∞ . (E.4)Proof of Lemma E.1.Direct computation shows that|BQ(s, a) -F Q(s, a)| =|r(s, a) + γE s ′ ∼P D (s ′ ) {max a ′ Q(s ′ , a ′ )} -r(s, a) -γE s ′ ∼P D (s ′ ) {max a ′ [αQ(s ′ , a ′ ) + (1 -α)Q L (s ′ , a ′ )]}| ≤γE s ′ ∼P D (s ′ ) | max a ′ {Q(s ′ , a ′ ) -[αQ(s ′ , a ′ ) + (1 -α)Q L (s ′ , a ′ )]}| ≤γE s ′ ∼P D (s ′ ) | max a ′ {Q(s ′ , a ′ ) -[αQ(s ′ , a ′ ) + (1 -α)Q L (s ′ , a ′ )]}| ≤γE s ′ ∼P D (s ′ ) max a ′ |Q(s ′ , a ′ ) -[αQ(s ′ , a ′ ) + (1 -α)Q L (s ′ , a ′ )]| =γE s ′ ∼P D (s ′ ) max a ′ |Q(s ′ , a ′ ) -[(α + (1 -α)β)Q(s ′ , a ′ )1 (a ′ / ∈U (Q)) + (α + (1 -α) H Q (a ′ |s ′ ) )Q(s ′ , a ′ )1 (a ′ ∈U (Q)) ]| =γE s ′ ∼P D (s ′ ) max a ′ |(1 -α -(1 -α)β)Q(s ′ , a ′ )1 (a ′ / ∈U (Q)) + (1 -α - (1 -α) H Q (a ′ |s ′ ) )Q(s ′ , a ′ )1 (a ′ ∈U (Q)) | ≤γE s ′ ∼P D (s ′ ) max a ′ {(1 -α)(1 -β))|Q(s ′ , a ′ )|1 (a ′ / ∈U (Q)) + (1 -α)(1 - 1 H Q (a ′ |s ′ ) )|Q(s ′ , a ′ )|1 (a ′ ∈U (Q)) }. (E.5)Then with probability 1 -η,|BQ * (s, a) -F Q * (s, a)| ≤γE s ′ ∼P D (s ′ ) max a ′ {(1 -α)(1 -β))|Q * (s ′ , a ′ )|} ≤γ(1 -α)(1 -β)||Q * (•, •)|| ∞ , (E.6)and with probability η,|BQ * (s, a) -F Q * (s, a)| ≤γ max s ′ max a ′ {(1 -α)(1 - 1 H Q * (a ′ |s ′ ) )|Q(s ′ , a ′ )|} ≤γ(1 -α)(1 - 1 H Q * (a ′ |s ′ ) )||Q * (•, •)|| ∞ . (E.7)Then we finish the proof.Next, we will give a brief proof of Theorem 4.4 with the results of Theorem 4.3 and Lemma E.1. Suppose the stationary point or the optimal Q-value derived based on the QDQ Bellman operator is Q ∆ , which satisfying:Q ∆ = F Q ∆ .Then with probability 1 -η we haveQ ∆ -Q * ∞ = F Q ∆ -F Q * + F Q * -BQ * ∞ ≤ F Q ∆ -F Q * ∞ + ∥F Q * -BQ * ∥ ∞ ≤cγ||Q ∆ -Q * || ∞ + γ(1 -α)(1 -β)||Q * (•, •)|| ∞ .Hence,Q ∆ -Q * ∞ ≤ γ(1 -α)(1 -β)(1 -cγ) -1 ||Q * (•, •)|| ∞ . Set ϵ = γ(1 -α)(1 -β)(1 -cγ) -1 ||Q * (•, •)|| ∞ finish the proof. Remark E.1. The error ϵ = γ(1 -α)(1 -β)(1 -cγ) -1 ||Q * (•, •)|| ∞ canbe 0 by setting β = 1 . In practice, we can ensure that ϵ converges to a small value by appropriately adjusting the parameters α and β. Remark E.2. Our primary focus lies on the scenario where max a′ {(1 -β))|Q(s ′ , a ′ )|1 (a ′ / ∈U (Q)) + (1 - 1 H Q (a ′ |s ′ ) )|Q(s ′ , a ′ )|1 (a ′ ∈U (Q)) } = max a ′ {(1 -β))|Q(s ′ , a ′ )|1 (a ′ / ∈U (Q)) }.This preference stems from the potential minuteness of η. Firstly, we can rely on Theorem 1 in [20] to ensure that the consistency model can converge to ground truth, thus guaranteeing the fidelity of the learned Q-distribution. Consequently, the accuracy of our uncertainty estimation is upheld, enabling us to effectively assess OOD points and pessimistically adjust the Q-value for such occurrences [4] . Secondly, in order to mitigate segmentation errors of the uncertainty set, we introduce the penalty factor β in Eq.8. Finally, since actions in the set U(Q) are always penalized, the Q-value always takes a lower value when a ′ ∈ U(Q). Collectively, these measures reinforce our aim to maintain a small η. Theorem 4.4 shows the optimal Q-value by QDQ can closely approximate the true optimal Q-value Q * over S D × A D . Then we also provide the following corollary to show that substitute the optimal Bellman operator BQ with F Q will introduce controllable error at each step, and give an step-wise analysis of the convergence of QDQ operator F Q. Let ζ k (s, a) = |Q k (s, a) -Q * (s,ζ k (s, a) ≤δ * k (s, a) + ξ + γE s ′ ∼D max a ′ |ζ k-1 (s, a)|Then we introduce our second Lemma E.2 to help the proof of Corollary 1:Lemma E.2. For any s ∈ S D , a ∈ A D , with probability 1 -η, δ k (s, a) ≤ δ * k (s, a) + γ(1 -α)(1 -β)||Q(s ′ , a ′ )|| ∞ . (E.8) With probability η, δ k (s, a) ≤ δ * k (s, a) + γ(1 -α)(1 - 1 H Q (a ′ |s ′ ) )||Q(s ′ , a ′ )|| ∞ . (E.9) Proof of Lemma E.2. δ k (s, a) =|Q k (s, a) -F Q k (s, a)| =|Q k (s, a) -BQ k (s, a) + BQ k (s, a) -F Q k (s, a)| ≤δ * k (s, a) + |BQ(s, a) -F Q(s, a)|δ * k (s, a) + γ(1 -α)(1 -β)||Q(s ′ , a ′ )|| ∞ . (E.10) By Lemma E.1, with probability 1 -η, δ k (s, a) =|Q k (s, a) -F Q k (s, a)| =|Q k (s, a) -BQ k (s, a) + BQ k (s, a) -F Q k (s, a)| ≤δ * k (s, a) + |BQ(s, a) -F Q(s, a)| ≤δ * k (s, a) + γ(1 -α)(1 -β)||Q(s ′ , a ′ )|| ∞ . (E.11) With probability η, δ k (s, a) =|Q k (s, a) -F Q k (s, a)| =|Q k (s, a) -BQ k (s, a) + BQ k (s, a) -F Q k (s, a)| ≤δ * k (s, a) + |BQ(s, a) -F Q(s, a)| ≤δ * k (s, a) + γ(1 -α)(1 - 1 H Q (a ′ |s ′ ) )||Q(s ′ , a ′ )|| ∞ . (E.12)Then we finish the proof.Next, we will give a brief proof of Corollary 1.ζ k (s, a) =|Q k (s, a) -Q * (s, a)| =|Q k (s, a) -F Q k-1 (s, a) + F Q k-1 (s, a) -Q * (s, a)| ≤|Q k (s, a) -F Q k-1 (s, a)| + |F Q k-1 (s, a) -Q * (s, a)| =δ k (s, a) + |F Q k-1 (s, a) -Q * (s, a)| =δ k (s, a) + |F Q k-1 (s, a) -BQ k-1 (s, a) + BQ k-1 (s, a) -BQ * (s, a)| ≤δ k (s, a) + |F Q k-1 (s, a) -BQ k-1 (s, a)| + |BQ k-1 (s, a) -BQ * (s, a)| =δ k (s, a) + |F Q k-1 (s, a) -BQ k-1 (s, a)| + |γE s ′ ∼P D (s ′ ) max a ′ Q k-1 (s, a) -γE s ′ ∼P D (s ′ ) max a ′ Q * (s, a)| ≤δ k (s, a) + |F Q k-1 (s, a) -BQ k-1 (s, a)| + γE s ′ ∼D max a ′ |Q k-1 (s, a) -Q * (s, a)| =δ * k (s, a) + γ(1 -α)(1 - 1 H Q (a ′ |s ′ ) )||Q(s ′ , a ′ )|| ∞ + γE s ′ ∼D max a ′ |ζ k-1 (s, a)| (E.13)By Lemma E.1 and Lemma E.2, we can easily derive, with probability 1 -η,ζ k (s, a) ≤δ * k (s, a) + γ(1 -α)(1 -β)||Q(s ′ , a ′ )|| ∞ + γE s ′ ∼D max a ′ |ζ k-1 (s, a)| (E.14) Set ξ = γ(1 -α)(1 -β)||Q(s ′ , a ′ )|| ∞ ,then we finish the proof. Remark E.3. During the value iteration, there are two error accumulate: δ * k (s, a) and ξ. Given the convergence of optimal Q-value iteration, δ * k (s, a) tends to approach an arbitrarily small value and may even converge to 0 under optimal circumstances. The error ξ = γ(1-α)(1-β)||Q(s ′ , a ′ )|| ∞ induced by using the Bellman operator F Q instead of the optimal Bellman operator can also approach 0 as discussed in Remark E.1.F Gap expanding of the QDQ algorithm.In this section we will show that QDQ also has a Gap expanding property as discussed in CQL [3] . Proposition F.1. The QDQ algorithm is Gap expanding for Q-values within-distribution and Q-values out of distribution.(1) If a / ∈ U(Q) and is indeed in-distribution action, the target Q-value for the in-distribution point is(α + (1 -α)β)Q. The target Q-value for the OOD point is (α + (1-α) H Q (a|s) )Q. And (α + (1 -α)β)Q -(α + (1-α)H Q (a|s) )Q > 0 definitely. And this suggest Q-value will prefer in-distribution actions.(2) If a / ∈ U(Q) and is indeed OOD action. In such cases, the penalty parameter β is introduced to penalize the Q-value, resulting in (α+(1-α)β)Q < Q. Consequently, misclassifications of OOD actions can be handled in a pessimistic manner.While introducing the penalty parameter β may slightly slow down optimization updates for true in-distribution actions, it is crucial to prioritize control over out-of-distribution (OOD) actions due to the potentially severe consequences of exploration errors. In balancing the optimization of Q-learning with a pessimistic approach to OOD actions, the focus should be on reducing the impact of OOD actions. In fact, compared to previous pessimistic Q-value methods, QDQ offers greater flexibility in managing these values. This includes incorporating an uncertainty-aware learning objective and adjusting Q-values based on their uncertainty.G.1 Real Q dataset generation.In Section 3.1, we use a k-step sliding window approach with a length of T to traverse each trajectory in the dataset D and generate the Q-value dataset D Q based on Equation 5. A good Q-value dataset for uncertainty estimation should cover a broad state and action space to accurately characterize the Q distribution and detect high uncertainty in the OOD action region. This coverage helps identify actions with significant Q-value uncertainty in OOD areas.Choosing the right value for the sliding step k and the window length T is crucial. A large value for k may result in a smaller Q dataset, while a small value may lead to excessive homogenization of the Q dataset. Both situations can negatively impact the learning of the Q-value distribution. If the Q-value dataset is too small, the learned Q distribution may not generalize well across the state-action space S × A. Conversely, if the Q-values are too homogeneous, it can hinder feature learning by the distribution learner.To illustrate this, Figure G.1 shows the Q dataset distributions obtained for different values of k using the halfcheetah-medium dataset. When k = 1, the Q distribution is more concentrated, resulting in more homogeneous Q-values. In contrast, with k = 50, the Q distribution becomes sparser, showing a greater inclination towards individual features.In the experiments, we consider various factors when setting the value of k, including the trajectory length, the width of the sliding window, and the derived Q-value dataset size. Throughout all experiments, we set k to 10. Interestingly, we observed that the distribution of Q-values remains robust to minor adjustments in k, indicating that the choice of k does not necessitate overly stringent tuning.k=1，data size=800,000 k=10，data size=80,000 k=50，data size=16,000Figure G.1: The derived Q-value distribution when using difference sliding step and same window width to scan over the trajectory's on halfcheetah-medium dataset.The width of the sliding window is set to 200. The Q-value is scaled to facilitate comparison. When choosing the width of the sliding window, T , we must consider factors such as trajectory length and the resulting size of the Q-value dataset. When T increases, the size of the Q-value dataset decreases. However, if T is too small, it might truncate essential information from the true Q-value. We give the experimental analysis of Q-value distribution for different T on the sparse reward task Antmaze-medium-play in Figure G.2. The Q distribution of the Antmaze-medium-play dataset with varying sliding window widths (100 to 300 steps) is shown in the figure. Widening the sliding window does not change the shape of the Q distribution, even though a larger window covers more information for this sparse reward task with many short trajectories. Instead, enlarging the sliding window decreases the Q value and compresses the size of the derived Q data.In our experiments, we opted for T = 200 across all tasks. This choice considers factors such as the maximum trajectory length (1000), the decay rate of γ m-1 in Eq.5. Based on the analysis provided in Section B, T does not need to be very large. We also found that minor adjustments to T do not significantly affect the Q-value distribution, indicating that strict tuning of this parameter is unnecessary. From the distribution of Antmaze we observe that trajectories for these tasks mainly consist of suboptimal or failed experiences, this underscores the challenging nature of the Antmaze task.Further the narrower data distribution make it easier to take OOD actions during offline training and fewer positive experiences limits the optimisation of Q, these all potentially leading to failure of these kinds of tasks. The uncertainty measure is crucial for guiding the Q-value towards a pessimistic regime within the QDQ algorithm. In this section, we will first verify how uncertainty can assess the overestimation of Q-values in OOD regions. Then, we will discuss how the uncertainty set U(Q) can be shaped using the hyperparameter β.To understand the differences in uncertainty between in-distribution actions and OOD actions from a random policy, we can compare their distributions. In the left graph of Figure G.5, the red line represents the 95% quantile of the standard deviation of sampled Q-values from the learned Q-distribution for in-distribution actions. In the right graph, this 95% quantile corresponds to approximately the 75% quantile of the standard deviation for OOD actions. This indicates that OOD actions contribute to a heavy-tailed distribution of Q-value uncertainty, resulting in larger values compared to in-distribution actions. Although QDQ has three hyperparameters (α, β, and γ) for flexibility, the tuning process is straightforward. For example, as discussed in Theorem 4.4 (Appendix E), theoretically, (1-α)(1-β) should be small. Since β controls the size of the uncertainty set and requires flexibility across different tasks, we typically set α close to 1, tuning it between 0.9 and 0.995. This only requires a few experiments to find the optimal value. Our tuning process involves sequentially fixing parameters while selecting the best α, then β, and finally γ. Based on the characteristics of different datasets, we can set each parameter to an initial value close to its optimal value. QDQ offers evidence-based guidelines for hyperparameter ranges, making tuning manageable. Additionally, QDQ's efficiency (see Table G .6) reduces the tuning burden.In the ablation study, we choose four task which represent different kinds of dataset we analyzed during parameter study:(1) Wide distribution: halfcheetah-medium-v2.(2) Task sensitive: hopper-medium-v2.(3) Demonstration: walker2d-medium-expert-v2.(4) Narrow distribution: umaze-diverse-v0.We perform ablation study for parameters we discussed in Section 5.2. We compare the performances of three different settings for the three parameters respectively.The learning curves for four types of tasks with different uncertainty-aware weights α (Eq. 7) are shown in Figure G.9. For the halfcheetah-medium-v2 and hopper-medium-v2 datasets, decreasing α harms performance. For the walker2d-medium-expert-v2 dataset, a slightly lower α increases training volatility. In contrast, the umaze-diverse-v0 dataset shows high sensitivity to α. This sensitivity occurs because the Antmaze task requires combining different suboptimal trajectories, which challenges algorithms like QDQ that are not fully in-sample but in-support. For instance, an α value that is too high or too low can lead to overly optimistic or pessimistic Q-values, causing the algorithm to favor actions that do not align with the exact suboptimal trajectories.The ablation study of the uncertainty-related parameter β is presented in Figure G.11. For the wide dataset halfcheetah-medium-v2, a higher β is preferred, indicating less control over the uncertainty penalty and a smaller uncertainty set. In contrast, the performance for the hopper-medium-v2 and umaze-diverse-v0 datasets is sensitive to small changes in β due to their task sensitivity and narrow distribution. The walker2d-medium-expert-v2 dataset, however, is robust to small changes in β.We present the performance of different tasks for the entropy parameter γ in Figure G.12. For the wide distribution in halfcheetah-medium-v2, a larger γ negatively impacts performance. The hopper-medium-v2 and umaze-diverse-v0 datasets also show sensitivity to the parameter γ. In the walker2d-medium-expert-v2 dataset, a very small γ may introduce volatility into the training process. However, the final result's convergence is not significantly affected.Furthermore, as discussed in Section 5.2, the gamma term primarily stabilizes the learning of a simple Gaussian policy, especially for action-sensitive and dataset-narrow tasks. We note that Gaussian policies are prone to sampling risky actions because they can only fit a single-mode policy. To verify the impact of uncertainty-aware Q-value optimization in QDQ, we compared the performance of Q-values without uncertainty control (using Bellman optimization like in online RL settings) to the QDQ algorithm on the action-sensitive hopper-medium dataset, using identical gamma settings. Figure ?? shows that introducing an uncertainty-based constraint for the Q-value function in QDQ significantly improves training stability, convergence speed, and overall performance. This supports the effectiveness of QDQ's uncertainty-aware Q-value optimization. We believe that a stronger learning policy will further reduce the need for this stabilizing term. Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).Our code can be found at https://github.com/evalarzj/qdq.We do not provide the performance of UWAC, EDAC, PBRL, and MCQ on the AntMaze task, as they do not offer useful parameter settings for this task.https://github.com/openai/consistency_models_cifar10 


Decision: The paper presents a method for conservative offline RL via consistency models, that reviewers found well-motivated, intuitive, novel, and sound. The paper itself is mostly clear and the empirical evaluation is satisfactory. Some concerns were raised regarding the presentation and correctness of the method's theoretical grounding, as well as its context in the literature. The reviewers and I are satisfied with the authors' explanation of these issues and with their promise to polish the writing.

Reviews: review: paper_summary: paper_summary: The paper proposes a method for estimating Q values in the offline/batch setting by leveraging consistency models. Via these, the uncertainty over the Q function can be estimated and used as a robust penalty to prevent distribution shift in offline RL. The authors provide both experimental validation on the standard D4RL dataset, as well as theoretical insight into the performance of their algorithm.
strength_weakness: strength_weakness: The main weakness at the moment lies in the slightly confusing presentation. While I think that all the ideas are in principle well supported, it is very hard to follow the exact setup and intuition throughout. In addition, I am not fully convinced that the theoretical results are fully informative for the empirical implementation.

The introduction of the consistency model is very terse. For readers who are less deep in the diffusion literature, this presents a large barrier for understanding the rest of the paper. I would recommend that the authors provide a slightly more thorough introduction here.

The sliding window approach to Q estimation seems to rely heavily on a relatively dense informative reward. This should be discussed in the paper. While Theorem 4.1 does (somewhat trivially) hold, it does not suggest that partial reward sums will necessarily be informative. In cases with 0 reward across a chunk (which would happen frequently with sparse rewards) the consistency model would always predict near 0 uncertainty. This issues is currently the main reason I am leaning towards recommending rejection and I am happy to discuss it in the rebuttal in case I am overestimating the importance of the problem.

In general, the theoretical analysis is not tied strongly to the rest of the paper. It is not clear if the authors draw conclusions from it for their method, or simply present it as a justification for its validity (in which case I would expect a slightly more thorough discussion).

The method requires 3 additional hyperparameters, which the authors discuss. Given the enormous difficulty of model selection / hyperparameter tuning in offline RL and the communities lack of coherent standards here, I would like for the authors to discuss how the hyperparameters were tuned (and whether this constitutes test set training). I do concede that this is a wider issue in the community and that I cannot fully fault the authors to adhering to standards here, but I think the issue should be discussed in offline RL papers.

The empirical results do not seem vastly better than MCQ, so i would tone done the writing a small bit. It is unclear whether the authors for example allowed themselves a larger hyperparameter tuning budget.
questions: questions: Why is the set of baseline methods different on the two different benchmarks? Especially the strongest method on D4RL seems to be missing from the Ant-maze comparison.
limitations: limitations: none
paper_summary: paper_summary: This paper proposes a new offline RL method, Q-Distribution guided Q-learning (QDQ), which uses a consistency model to model the distribution of Q-value for uncertainty estimation and then introduces an uncertainty-aware optimization objective for pessimistic Q-learning. This method has theoretical guarantees and exhibits strong performance in the D4RL benchmark.
strength_weakness: strength_weakness: - Personally, there are some confusions regarding the method. The consistency model is trained to fit the distribution of $Q^{\pi{\beta}}$, but it is used to estimate the uncertainty of the updating Q function (as shown in Eq. 8). So, I wonder why the variance of $Q^{\pi{\beta}}$ can estimate the uncertainty of Q. Another confusion is why there is still a policy constraint in Eq. 9. Normally, for pessimistic value offline RL methods like CQL[1], EDAC[2], and PBRL[3], they only impose conservatism on Q-value learning and do not add any constraint on policy learning. I noticed the ablation study on $\gamma$ shows that too small $\gamma$ leads to poor performance on some tasks like hopper-medium-v2. Does this mean that the proposed uncertainty-aware learning objective (Eq. 7) is too weak for some tasks?
- I think some baselines are missing in the main evaluation experiment, like EDAC and PBRL. To the best of my knowledge, EDAC is the SOTA model-free uncertainty-based method so far. So, I suggest the authors include these two algorithms in Table 1 & 2.
- The authors point out that previous uncertainty-based offline RL methods like bootstrap or ensemble impose significant computational burdens (line 181). Although their proposed method does not need ensemble Q-value networks, it additionally brings a consistency model whose training relies on a pretrained diffusion model. So, my concern is whether this will cost more computation compared with ensemble Q-value networks.

[1] Kumar et al. "Conservative Q-Learning for Offline Reinforcement Learning"

[2] An et al. "Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble"

[3] Bai et al. "Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning"
questions: questions: To summarize the main points/questions raised in the weaknesses section:

- Could the authors provide some explanations for the confusions mentioned in the first point of weaknesses?
- Could the authors include EDAC and PBRL in Table 1 & 2?
- Could the authors compare the computational costs (like runtime and GPU memory usage) of their method with other methods?
limitations: limitations: N/A
paper_summary: paper_summary: This paper addresses the well-known issue of OOD (Out-of-Distribution) actions in offline reinforcement learning by proposing the QDQ method, which penalizes the Q-values in regions with high uncertainty. To better estimate uncertainty, QDQ first constructs a truncated Q-value dataset for the behavior policy and learns the distribution of Q-values induced by the behavior policy using a consistency model, allowing subsequent estimation of the uncertainty for each state-action pair through one-step sampling. With this uncertainty estimation method, QDQ can perform  pessimistic value estimation for samples, thus avoiding the overestimation of Q-values for OOD actions. The paper provides a series of theoretical results to support the rationality of the QDQ method and demonstrates the advantages of the QDQ method through a series of experiments.
strength_weakness: strength_weakness: 1. First and foremost, the fundamental assumption of this paper is that a consistency model can learn a distribution of Q-values with high fidelity. However, why do Q-values need to satisfy the property of consistency? If we are learning the distribution of trajectories, it is reasonable to require the trajectory distribution to be consistent, but is this assumption also reasonable for the distribution of Q-values? This is the starting point of the paper, yet it does not argue for its necessity. Therefore, I suspect that for any type of generative model, even Variational Autoencoders (VAEs), the uncertainty of Q(s,a) could be calculated, thereby allowing the subsequent methods of this paper to be applied for learning.

2. Based on my understanding, Equation (6) actually calculates the emperical version $V_{\epsilon}(s,a)$ of uncertainty $Var(Q^{\pi_{\beta}})$  corresponding to the behavior policy $\pi_{\beta}$, rather than the uncertainty $Var(Q^{\pi})$ of  corresponding to the current policy $\pi$. However, it seems to me that in the subsequent use of $V_{\epsilon}(s,a)$ and in the proof of the theorems, $V_{\epsilon}(s,a)$ is directly used as $Var(Q^{\pi})$, which could be mathematically problematic. The authors need to provide an explanation for this point. If indeed $Var(Q^{\pi_{\beta}})$ is used in place of $Var(Q^{\pi})$, please justify the rationale.

3. This paper lacks some comparisons with related work. It could be compared with methods that use ensembles to estimate uncertainty [1] and methods that make pessimistic estimates for Out-of-Distribution (OOD) actions [2].


[1] Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning.
[2] Supported value regularization for offline reinforcement learning.
questions: questions: The same as Weaknesses
limitations: limitations: This paper does not mention any potential negative impacts that may arise from its work. It is recommended that this be supplemented.

Comments: title: Quick reply
Reviewer reply
Follow-up response
Follow-up Questions.
comment: Thank you very much for your response! We sincerely appreciate you pointing out our issues and placing your trust in us. We will do our utmost to polish and revise the paper, ensuring that we live up to your trust and expectations! Regardless of the outcome, we will carefully implement your suggestions in our revisions and will continue to adhere to the rigorous scholarship that you and the other reviewers have demonstrated. Once again, thank you for your time and effort!
If you are confident that you can get the work in shape for a great camera ready than I’m not going to stand in the way of acceptance :) ultimately it’s the AC/SAC/PCs that decide of course, I just wanted to make sure that you are certain you can get this work into a great enough shape so that it can get the recognition it deserves
Thank you very much for your response! We sincerely appreciate your ongoing recognition of our work and its contribution to the field of offline RL. As we mentioned earlier, we hope that QDQ can provide evidence and insights for research on uncertainty estimation in offline reinforcement learning, and on how to effectively use uncertainty to more accurately control the overestimation bias of Q values in areas beyond the knowledge of the dataset. We hope our work can offer some insights and support for future research in offline RL, and even more, shed a light on the usage of uncertainty estimation in the study of exploration in online RL.

We sincerely apologize for the issues with the presentation of our work, which negatively impacted your review experience and took up your valuable time. We are taking this feedback very seriously and will thoroughly revise all presentation-related aspects of the paper. In fact, we have already begun revising our manuscript, and we will incorporate all the issues you mentioned, as well as the content from the rebuttal, into the latest version of our paper. We will refine the motivation, the introduction of relevant background (especially the consistency model), the description of the algorithm, and particularly the integration of theory and algorithm. We are confident that, with the additional time available and the valuable suggestions provided by you and the other reviewers, we can complete these revisions effectively. We are committed to refining our paper and are confident that the next version will meet your expectations.

Once again, thank you for recognizing our work, and we sincerely appreciate your review efforts! We will address and correct all the issues you mentioned, making the entire work clearer and more precise. We also hope that our work can meet your expectations and play a positive role in advancing research in reinforcement learning.
Thanks for your thorough comments. I am satisfied with the answer and I will update my score, but I truly believe that this paper deserves another very thorough editorial pass to make everything more clear to the readers. I do not mean to offend, but the presentation feels very rushed and it might be a good idea to consider re-submission to a future conference after polishing the presentation so that you do not lose potential impact due to low clarity. As I am only a reviewer, this is of course not my final decision and between the AC/SAC and the authors, but I think it might be a good idea to forgo publication now to give yourselves the time to make your work truly shine.
Thank you very much for your prompt response and for raising the score! Your questions and valuable suggestions play a crucial role in improving our paper, and we will incorporate all the content from the rebuttal into the manuscript. Once again, we sincerely appreciate your feedback and your assistance in enhancing the quality of our paper!
Thank you for your reply, my concerns have been mostly resolved. I have already improved my score.
Thank you very much for recognizing our work. We are also deeply grateful for your valuable suggestions, which have helped to further refine and strengthen our research. We will carefully incorporate the points mentioned in the rebuttal and discussion into the paper to make our work clearer and more comprehensive.
Thank you for your reply, my doubts have been mostly resolved. I have already improved my score.
Thank you very much for your quick response. In QDQ, the uncertainty estimation of $H_Q(a'|s') =  \sqrt{V(X_\epsilon|(s',a'))}$ is only used to penalize the Q-value of actions with high uncertainty($a' \in U(Q)$). Its primary purpose is to make the Q target more pessimistic by adjusting the Q target value to $\frac{1}{H_Q(a'|s')}Q(s',a')1_{(a' \in U(Q))}$. The use of $H_Q(a'|s') =  \sqrt{V(X_\epsilon|(s',a'))}$ does not affect the proofs and conclusions of Theorems 4.2 and 4.3, as it only serves to underestimate the Q target value in high uncertainty (OOD) regions. As long as the Q target value in OOD regions is pessimistic, the theorem holds. Indeed, $H_Q(a'|s') =  \sqrt{V(X_\epsilon|(s',a'))}$ is just a penalty factor, and as long as $\frac{1}{H_Q(a'|s')}Q(s',a')1_{(a' \in U(Q))} < Q(s',a') 1_{(a' \in U(Q))}$, Theorems 4.2 and 4.3 are hold, regardless of whether it is the uncertainty estimation of the Q value of the learning policy or the behavior policy. As shown in Figure G.4 (Appendix G.4), the distribution of $H_Q(a'|s') =  \sqrt{V(X_\epsilon|(s',a'))}$ confirms that $\frac{1}{H_Q(a'|s')}Q(s',a')1_{(a' \in U(Q))} < Q(s',a') 1_{(a' \in U(Q))}$ always holds because the value of $H_Q(a'|s') =  \sqrt{V(X_\epsilon|(s',a'))}$ is relatively large in the high uncertainty regions defined by QDQ.
Thank you very much for your response. Most of my questions have been answered, especially as you pointed out, replacing the uncertainty of \(Q\) with that of \(Q^{\beta}\) does indeed seem reasonable. However, I still need to confirm a few points with you. Specifically, do Theorems 4.2 and 4.3 strictly require that \(V(X|(s',a')\) be the uncertainty of \(Q\), or is it permissible to replace it with the uncertainty of \(Q^{\beta}\) in a rigorous sense?