Offline RL with Smooth OOD Generalization in Convex Hull and its Neighborhood
Authors: Qingmao Yao
Zhichao Lei
Tianyuan Chen
Ziyue Yuan
Xuefan Chen
Jianxiang Liu
Faguo Wu
Xiao Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Deep Reinforcement Learning
Offline Reinforcement Learning
Smooth Bellman Operator
Smooth Q-function OOD Generalization
Primary area: reinforcement learning

Abstract: Offline Reinforcement Learning (RL) struggles with distributional shifts, leading to the $Q$-value overestimation for out-of-distribution (OOD) actions. Existing methods address this issue by imposing constraints; however, they often become overly conservative when evaluating OOD regions, which constrains the $Q$-function generalization. This over-constraint issue results in poor $Q$-value estimation and hinders policy improvement. In this paper, we introduce a novel approach to achieve better $Q$-value estimation by enhancing $Q$-function generalization in OOD regions within Convex Hull and its Neighborhood (CHN). Under the safety generalization guarantees of the CHN, we propose the Smooth Bellman Operator (SBO), which updates OOD $Q$-values by smoothing them with neighboring in-sample $Q$-values. We theoretically show that SBO approximates true $Q$-values for both in-sample and OOD actions within the CHN. Our practical algorithm, Smooth Q-function OOD Generalization (SQOG), empirically alleviates the over-constraint issue, achieving near-accurate $Q$-value estimation. On the D4RL benchmarks, SQOG outperforms existing state-of-the-art methods in both performance and computational efficiency. Code is available at <https://github.com/yqpqry/SQOG>.

Published as a conference paper at ICLR 2025
OFFLINE RL WITH SMOOTH OOD GENERALIZATION IN CONVEX HULL AND ITS NEIGHBORHOOD
Qingmao Yao', Zhichao Lei?, Tianyuan Chen Ziyue Yuan', Xuefan Chen', Jianxiang Liu?:4,
Faguo W,3.,,556, Xiao Zhangl .4.5,,777
School of Mathematical Sciences, Beihang University
School of Physics, Beihang University
3 School of Artificial Intelligence, Beihang University
"Key Laboratory of Mathematics, Informatics and Behavioral Semanties, MoE, Beihang University 'Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing, Beihang University @Zhongguancun Laboratory
Hangghou International Innovation Institute Beihang University
ABSTRACT
Offline Reinforcement Learning (RL) struggles with distributional shifts, leading to the Q-value overestimation for methods address this issue by imposing constraints; however, they often become overly conservative when evaluating OOD regions, which constrains the Q-function generalization. This over-constraint issue results in poor Q-value estimation and hinders policy improvement. In this paper. introduce novel approach achieve better Q-value estimation by enhancing Q-function generalization in OOD regions within Convex Hull and its Neighborhood (CHN). Under the safety generalization guarantees of the CHN, we propose the Smooth Bellman Operator (SBO), which updates OOD Q-values by smoothing them with neighboring in-sample Q-values. We theoretically show that SBO approximates true Q-values for both in in-sample and OOD actions within the CHN. Our practical algorithm, Smooth Q-function OOD Generalization (SQOG), empirically alleviates the over-constraint issue, achieving near-ac near-accures accurate Q-value estimation. On the D4RL benchmarks, SQOG outperforms existing state-of-the-art methods in both performance and computational efficiency. Code is available at https: / /ttg//gtttbb//tt com/ yogpr////ggr//00
(OOD) actions. Existing
INTRODUCTION
Reinforcement Learning (RL) offers a powerful framework for control tasks, underpinned by rigorous mathematical principles. In online RL, an agent learns optimal strategies through direct interaction with the environment. However, in many real-world domains (e.g.. robotics, healthcare, autonomous driving), such interactions are infeasible impractical. Offline RI., in contrast, enables the agent to learn optimal policies from pre-collected datasets, eliminating the need for online interaction. Com- bining this data-driven paradigm with deep neural networks (DNNs) anticipated produce robust and generalizable decision-makenn making engines. However, the primary challenge in offline RL lies in the distribution shift between the learned policy and the behavior policy, which leads overestimation of OOD actions. Incorrect evaluation of OOD actions results in extrapolation errors, which are further exacerbated by bootstrapping, ultimately causing severe value function approximation errors and hindering the agent from learning an optimal policy.
Recent model-free offline RI. algorithms addresses this challenge through the following methods: 1) policy constraints, where constraints are added during policy updates to ensure the learned policies
remain close to the behavior policy (Fujimoto et al., 2019 Wu et al. 2019;
Kumar ct al. 2019
Fujimoto & Gui 2021 Kostrikov et al.) 202 Ran al. 2023 Li et al. 2022 Huang et al. 2023). 2)
value penalization, where constraints are incorporated during value updates to enforce conservatism
in the value function (Wu al. 2019 Kostrikov et al., 2021a Kumar al. 2020 Lyu al. 2022
Yang et al. 2022). 3) in-sample learning, where the value function is learned only within the dataset
*Corresponding Authors. Emails: {faguo, xiao. 2 Jbbuaa edu. en.

---

Published as a conference paper at ICLR 2025
to avoid evaluating OOD samples (Wang et al. 2018 Chen al. 2020 Kostrikov et al. b02 Xu iet al. 2023 Garg et al. 2023). Dataset OOD regions are typically regarded as cooormation-dfficiee and potentially hazardous. Avoiding evaluations in these regions helps mitigate the overestimation issue. However, many existing methods tend to be overly conservative handling dataset OOI) regions, which limits the Q-function's 's ability to generalize effectively. This phenomenon, termed the over-constraint issue, results in poor Q-value estimation. This raises an important question: Can we achieve better Q-value estimation by enhancing Q-function generalization in dataset OOD regions?
To address this question, we first introduce the CHN to define a boundary for safety generalization. By providing two safety guarantees, we demonstrate that generalizing the Q-function to OOD regions within the CHN is safe, while doing so outside the CHN is risky. Therefore, we focus on Q-function generalization within the CHN. To enhance this generalization, we propose the SBO, which incorporates a smooth generalization term into the empirical Bellman operator. The key idea is to adjust biased OOD Q-values using neighboring in in-sample Q- vavlle that closely approximate the true values. We provide theoretical justification for the SBO, showing that the smooth generalization term is appropriate. Additionally, we analyze its effects on both in-sample and OOD evaluations and establish its convergence properties. Applying the SBO allows the Q-function to gradually approximate the true OOD Q-values, while minimally affecting ia-sae evaluations. In theory, SBO yields more accurate Q-function for policy evaluation.
Building on SBO, we develop a computationally efficient offline RL algorithm: Smooth Q-function OOD Generalization (SQOG). Empirically, demonstrate that compared to TD3+BC (Fujimoto & Gu 2021), SQOG achieves more accurate Q-value estimation, particularly in OOD regions within the CHN, thereby alleviating the over-constraint issue of the Q-function. Finally, on the D4RL benchmarks, SQOG shows superior performance and computational efficiency compared to existing tfate-o methods.
To summarize, our contributions are as follows:
Under the safety guarantees of the CHN, we propose the Smooth Bellman Operator (SBO), which enhances Q-function generalization in OOD regions and approximates the true Q-values.
Building on SBO, we design an effective algorithm, SQOG, which alleviates the over-constraint issue and obtains SOTA results on benchmark datasets.
PRELIMINARIES
RL is typically modeled as a Markov Decision Process (MDP) (Sutton & Barto 2018), defined as M = (S, A,T, do, r, r). S represents the state space, A denotes the action space, and ' describes the conditional probability of state transitions T (st+1|s,, ax) (simply denoted as T (s'ls, a)). The initial state distribution is defined by do(su), the reward function is r S x A + R, and y E [0, 1) is the discount factor. The objective of RL is to learn an optimal policy æ that maximizes the cumulative expected reward ( = (as.:dd.arra(())))..::r
(2== 1'r (st, 0t)]. The state-action value function Q" (s, a) quantifies the discounted return of trajectory starting from state s and action a, following the policy x. The reward function is bounded, i.e. (r(s,)) S T'ax- Given a policy π, the Bellman operator for the Q function's 's iteration is defined as: B Q (s (s, a)
" [Q as))
Offline RL algorithms based on dynamic programming maintain parametric Q-function Qo(s,a) a) and optionally a parametric policy (a(ss)) The dataset is typically defined as D = {(s,, ai ri, ai,ri,
s/, ) where d, E {0, 1} is the done flag. The dataset is generated according to the behavior policy u(-)s). Given state s' E D. the empirical behavior policy is defined as:
j(aa)ss)== De.Dee 1|s==' ==aa" The Actor-Critic algorithm is widely used in RL, consisting of
Eeep 1[s=s/]
policy evaluation in Eq. ① and policy improvement in Eq. (2)
Loritie (0) 10:=E(aaaa
a) - (r 7E, - - (15) (2 2
(1)
Jactor (0) = EE-DD (Qo (s((sa)))
(2)
Since D typically does not contain all possible transitions (5, a, s'), the policy evaluation step uses an empirical Bellman operator 83T (s( a) = r + B. (+) QQ (s', a')) that only backs up
2

---

Published as a conference paper at ICLR 2025
single sample. The empirical Bellman operator relies a' sampled from learned policy a (-1s). In offline RL, a' may not correspond to any action in the dataset, typically when ((a'ls') = O. We refer to such actions as OOD actions" which are usually overestimated (Kumar al. 2019). Most existing methods introduce a new over-constraint issue when addressing the overestimation of OOD actions. We alleviate this issue by enhancing Q-function generalization in OOD regions within the CHN.
Q-LEARNING WITH SMOOTH OOD GENERALIZATION IN CHN
In this section, we first formally define the CHN and outline its safety guarantees for distinguishing the safer OOD regions (Section B.1). Then, we construct the SBO to improve the Q generalization within the CHN. Theoretically, we provide the justification for using SBO, as well discuss its in-sample and OOD effects (Section 3.2). Finally, we propose our practical algorithm SQOG with a low-computational cost implementation (Section 3.3
3.1 CONVEX HULL AND ITS NEIGHBORHOOD
Definition 1 (CHN, Convex Hull and its Neighborhood). For given dataset D, we define the in-sample state- cate-action set (S, A) = {(s, (s, a) E D CHN
R defined as the union of the convex hull and its neighborhood of (S, A)p: CHN(D) Cone (D) N (CCov((D)), where (D) = = ),zi(>; 2 0, >; {x E (S, A) x & Conw(D), minneComw(D) || yll S rr] is the external neighborhood. The radius r is always chosen to be smaller than or equal to the diameter of CD)
(S, A)p) is corvex hull, and N (Conw(D)) =
Definition 1 presents the formal mathematical definition of CHN, which possesses uniqueness. compactness and connectivity. We then demonstrate two safety guarantees of CHN.
Proposition 1 (Safety guarantee 1: Q-value difference is controlled within CHN). Under the NTK regime, given dataset D, æ1 Conv(D), x2 E (Cooo(()) 23 E A) CHN(D). We have,
[[Q((z1) - Qo (Projn 1
||. Proj V d1 + 2d,) _ M1
(3)
[1Qx(22) Qo (Projn || < ( hat//mmn[[aaa [2 Projp( / d2 + 2d2) S M2
(4)
[Qo(4s) - (z)))))((3))) - C1(v C1(ymin() #ssll (Projp(zz)]1) dz + 2dg)
(5)
z;|| is the projection to the dataset. d1 d2, d3 are the point- to-dataset distances. Both d1 = || - Projp (21)ll < max || x' || < B and d2 |,2 Projp(22)]1 Sr < B are bounded. dg = ||aa - Projp((s))) > where r is the external neighborhood radius and B sup [1|æ yll Jrr, Conv (D)) is the diameter of the convex hull.
where (x): = argmin,,
let r = maXxeConu(D) |x - Proj [I. then r S B. C1, M1, M2 are constants.
We generalize Proposition frrom the analysis of DOGE (Li et 2022) under the NTK regime (see Appendix A). The external neighborhood is crucial augmentation that significantly broadens the scope of safety generalization. For any state-action pair x (inside the convex hull) or 22 (in the external neighborhood), the difference between its Q-value and the in-sample Q-value Qo (p(oo)) can be controlled by the point-to-datenn tatassst distance. Due to the uniqueness of CHN, this distance d, ||, Proj (z:) |l, = 1,2 can be strictly controlled by the longest diameter B of the convex hull. Assuming that deep Q-function is a continuous mapping, keeping the compactness (bounded and closed) and connectivity of the set, then Q is bounded within CHN. Building upon Proposition T we can quantify the bound: Vx E CHN, || ( II S supx,eD (z.)( + max {M1, M2).
Proposition (Safety guarantee 2: Q-function is uniformly continuous within CHN). Assuming that deep Q-uucction is continuous, then deep Q 2-function defined on CHN is uniformly continuous:
Ve > 0, 38 > 0, s.t. VX,, T, E CHN(D), if x, < 6, then [[Qo(z.) - Qo(x))1 <
In practice, actions with ji(a'ls' |s') FO are often treated as OOD actions due their negligible frequency. Herr, (S, A) represents the set of statc-action pairs (s, a) extracted from the dataset D. The dataset 'D consists of tuples (8, a, r, s', d), but only the (s, a) pairs are included in (S, A)D, while (r, are ignored. 'We use both CHN and the CHN in this paper. The italicized form emphasizes the mathematical structure and properties, while the regular font highlights its conceptual and intuitive meaning.
'In this paper, unless otherwise specified, the ||. norm refers to the La norm || .122-
3

---

Published as a conference paper at ICLR 2025
Proposition 2 ensures that small input changes will not lead to drastic changes in the output Q-value, which means that the Q-function of OOD actions within CHN is easy to learn from the neighbor Q-function of in-sample actions which is more accurate (proved in Section B.2) Through the safety guarantees in Proposition T and 2
we can make it clear that the generalization of the Q-function in the OOD regions within CHN is safer and more reliable, without producing excessive high estimates. In the following sections, we only focus on the Q generalization in the OOD regions within CHN.
3.2 SMOOTH BELLMAN OPERATOR WITH OOD GENERALIZATION
In this section, we introduce method to actively improve Q generalization in OOD regions within the CHN. We start by defining the SBO and providing its theoretical justification through Theorem and Proposition 3 Subsequently, we present its in ma-samlle and OOD effects in Theorems 2 and
Definition 2. Given policy n, the Smooth Bellman Operator (SBO) is defined as
BTT (T((s. a) = (G,
(6)
where G1 is the smooth generalization operator:
= =
Q(s, a): ((s, neighbor)
M(ass) 0
Gi((s,a)
p(a|s) and (s,a) E CHN
and B2 is the base Bellman operator:
B)Q(s,a)= = B Qss,a) M(als) ((s.a), p(als) = 0 and (s, a) E CHN
(8)
where ju(als) = 0 and (s,a) E CHN implies that is OOD action within CHN, a denotes a dataset action which is in the neighborhood of the 0OD action a, i.e,, neighbor D and Neohesghbrr - all S 8. B² Q(s, a) denotes the wildly used empirical Bellman operator 1 Q Es,,, (1))) (Q(s,, a²) d']] For simplicity, we omit the network parameters 0, O' and Φb
nnighhor
in
In the SBO, sample Q-values are updated using the empirical Bellman backup in B 8g. while OOD Q-values within CHN are updated using the neighboring in-sample Q-values Q(s, an Reighbor)) in G Qg. Inspired by the MCB operator in MCQ (Lyu et al. 2022). we decompose the operator into 9QQ, and B Qo to address the potential OOD actions generated by T--ss). The smooth
generalization operator G1 conveys the key contribution of the SBO. Through the following Theorem 3 we will provide theoretical justification for G1.
1 and Proposition
Theorem (The empirical Bellman operator 8 Qo is close to 1 go).. Suppose there exist a policy constraint offline RL algorithmm such that the KL-divergence of learned policy π and the behavior policy is optimized to guarantee max K L(T, 1),, K KL(u, T)) S e. Then, under the NTK regime, for all (s,a) D, with high probability 2 5, ô E (0,
Cr.,
||B Qo-B S
+G.C max
V min (nin(Earzzllee a')l Eg ~ al) aa) 2d
sampling bownd
OOD overestimation error bound
where Cr.,, = Cr,8 + Pmrrs/aaa//1 1), <=
3rr and d & ||aman | Jamarll ViD(Rao]
VT- Here,
2
Gr.8 and Cr.s are constants dependent on the concentration properties of ((s,a) a) and (ss'ls, a). [D(s,,) a)) is the dataset size, amin and Omaz denote the minimum and maximum actions, and C is a constant.
Proof sketch. The proof consists of considering two main sources of error: the sampling error (arising from and T), and the OOD overestimation error (generated from Be' (s', d'))). Since By Qo has low OOD overestimation error and μ is close to म. we first analyze the sampling error through || Qa - BuQoll, which can be bounded by C.r. The OOD overestimation error is
V then examined the difference between Eg ( (o) (Q or (s', s') and E (Qo (8', a')] Under
4

---

Published as a conference paper at ICLR 2025
the NTK regime, this difference is controlled by the distance d. Given that both VP and d are small, the difference between 8 Qo and B Qo is expected be small. See Appendix A
Theorem shows that under the policy constraint algorithm framework, for in-sample (s, a."),, the empirical Q7 (s, a² can closely approximate Q% (s, by applying the empirical Bell- man operator 8 Meanwhile, Q3 (s. a²m) ) is close to the true non-parametric Q-value Q Q* a²m) (Ran et al. 2023). Then, Q7 a²t) is a near-accurate estimation for the in-sample (s, a," i.e. (s, R Q" ". However, the OOD Q-value Q7 (8, a dood) may suffer from underestimation due to the over-constraint issue, i.e., for some (s, adod) E CHN, Q7 a,od < <Q= good).
In Eq. 77) the operator G1 is introduced to approximate the true OOD Q-value within the CHN. Although the true OOD Q* (o, and the exact OOD reward r(s, good) are unattainable, we already obtain the nearly accurate in-sample Q7 (s, a In Proposition 3 we show that Q (m) can serve appropriate OOD target when combined with the neighboring condition.
Proposition 3 (Q7 (s, in
neighhee)) is appropriate). Suppose there exist e such that ||Ộ (s, a) (*(ss)) < e/2, for all (s.a) E D. For any OOD actions aood within CHN, by Proposition
2 there exist a small 8, if I|aood iin nniighoor! < then ||Q" (s, a Q" (s, a neeighorr )1 < e/2 we have,
||e (s, 2od) Q (s, a aneighbor) <
(10)
Proposition 3 can be proved directly using the triangle inequality. Subsequently, we propose Theorem 2 3 illustrate the effects of the SBO.
Theorem 2 (Effects on sa-sa evaluation). For the in- in-sample evaluation, 91 introduces negligible
changes the empirical Bellman operator 8D Qo- Under the NTK regime, given (s,a) E D. < 8, we have.
assuming that Va' ~ (-)ss), neighborr: s.t.lla' -neighbor ||
1 Qo(s, a) - 8Ts Qo(s, a)ll S C. yBs [ Wmin(z,w)VB + 26
(11)
where x Eg' lla ain hh | in neigghoor)) )l, y E Bals, al)l,, a constant.
The proof of Theorem is similar to Theorem I See Appendix A
Theorem 3 (Effects OOD evaluation). For the OOD evaluation, Gi helps mitigate underestimation and overestimation. Assuming that all (s, a) E DD Q* R Q" (s. a). Given (8,0) C 'D and (s, a) E CHN, assuming that ||a (s,a) (',,, a oneeghhor))] II > E (underestimation or overestimation), by applying the SBO for gradient descent updates (with infinitesimally small learning rate), we have,
an &neighhor S 8 and (s," Q am Wwiighhor Il < E, if
10"
(s, a)l < 11Q Q²
(12)
111(" a) - Q+ (8, (.s.aatt an
(eiighherr) S then (Q - (x. < 2s.
Proof sketch. If os) < as aiiighorr) descent method, it follows that Q* a) < Q**1(s,,,,)) (s, a) < Q" (s. a). Similarly, the overestimation case can be addressed. The final result can be established using similar approach combined with the triangle inequality. See Appendix
(underestimation), then by applying the gradient
From Theorem 2 and we observe that applying the SBO, BT, enables the Q-function to gradually approximate the true OOD Q-values within the CHN, while incurring negligible side effects on the in-sample evaluation. Finally, we present the convergence of the SBO with its proof in Appendix A
is worth noting that the SBO is designed specifically handle OOD Q Q-values within the CHN.
adhering to the safety guarantees. Extending this approach to learn Q-values for faraway OOD actions, which fall outside the CHN, remains an open challenge in the field.
AAssuming that parametric in-sample Q-value Q7 (s, (s, converged, i.c. Q8(s,a"") N 1B" (s,
5

---

Published as a conference paper at ICLR 2025
Algorithm 1 Smooth Q-function OOD Generalization (SQOG)
Initialize: Actor network parameter @, critic network parameters 01 and 0g. dataset D, target parameters Φ/, 00 02', training step T, smoothing parameter т, actor update frequency m.
2: for step 1 to T do
3: Sample mini-batch of transitions {(s, s', d from D.
4: Update 0,, 02 via minimizing the critic loss Eq. (1).
5: if mod w 0 then
6: Update ø via minimizing the actor loss Eq.
15
7: Update target network parameters by d' +
D) + 0 (1 -)0 70, 1,2
8: end if
9:
end for
3.3 PRACTICAL ALGORITHM
Based on the smooth generalization operator 91 in SBO, we design an OOD generalization loss Log Eq. 133 which can be easily integrated into the objective function of critic network Eq.
LoG (0) an [Q00(s Q(s,, an (nesghbor))
(13)
In practice, we aim to devise a www-computational-costt implementation for a aod and "nighhoor Notably, during each training loop. we randomly sample a batch state-action pairs from the dataset, which naturally yields an in-sample action a'n By adding noise 17 to a² we generate neighboring action aod an . ensuring that ||a 2ood] S and (s, ood) E CHN are satisfied by appropriately controlling the noise scale. This approach allows to sample pairs of a neighber and
in
and with minimal computational cost. Consequently, by combining with Eq. [ and 13 we achieve practical Q-learming objective function of critic networks with low-computational.com -cos::
(SQoa(()) = (((...)))) (oo (r+ 7 m (o)))
(14)
+ (8, - 20, (s,
where Qo (s', a') represents the Q target network outputs, a' (s). Qo, (s, a) the Q network output with the gradient detached, {1,2]. Similar to TD3+BC (Fujimoto & Gu 2021). representative offline Actor-Critic algorithm, we set the objective function of actor network as:
( = -E (s,,a)> IAQo, (s, (s)) (T a}²]
(15)
where À = aN/ L./
a, Qs,, 0z), a is a hyperparameter, N is the batch-size. The pseudo-code of SQOG is in Algorithm Further discussions on SQOG are provided Appendix D and
Es,
EXPERIMENTS
In this section, we first empirically show that compared to TD3+BC, SQOG alleviates the over- constraint issue, leading more accurate Q-value estimation. Second, we highlight the advantages of our algorithm on the D4RL benchmarks (Fu et al. 202T), where SQOG demonstrates superior performance and computational efficiency. Finally, present ablation study to analyze the contributions the key components in our approach.
Sanity check: alleviation of the over-constraint issue. To demonstrate the alleviation of the constraint issue, we construct a dataset using the Mujoco environment "Inverted Double Pendulum" chosen for its one-dimensional action space and appropriate task complexity (see Appendix B.2 for wider evidence on high-dimensional tasks). The dataset is generated by training a policy online using Soft Actor-Critic (Haarnoja et al.) 2018) and subsequently collecting I million samples from the trained policy. We select two key states (the most frequently occurring ones in the dataset) to
"Note that the superscript ood is used to distinguish from the in-dataset real actions. If a ood is in-sample, the added noise will provide robustness for training the in-sample Q. Similar (Lyu et al. 2022

---

Published as a conference paper at ICLR 2025
0.30
0.28
0.30
Value
0.28
0.26
0.24
1031
5000
-0.25
0.00
0.25
0.75
1,00
1.00-0.75-0.30
Actions
Actions
(a) Q-values for state
(b) Q-values for state
Figure I: Q-values estimation for two key states. The color bars show the density of different actions.
Higher density actions correspond darker colors. With the tight constraints of the behavior policy, the Q-values of TD3+BC are overly constrained within [-0.50, 0.50] as shown in Figure [(a) while Figure [[b]] the Q-values are overly constrained within [-0.80, -0.40] and [0.25, 1.00]. However, SQOG consistently achieves accurate estimation of Q-values in most cases.
Table Normalized average score comparison of SQOG against baseline methods on D4RL bench- marks over the final 10 evaluations and 4 random seeds. We bold the highest scores.
Dataset
BC
TD3+BC
CQL
IQL
DOGE MCQ
SQOG
halfcheetah-r
2.210.00
11.011111
17.511.5
13.1411.
17.811.2
23.640.88
25.6=004 3..6+3.3
hopper-r
3.74:0.6
8.5+00.6
7.9+0.4
7.9-0.2
21.1+12.6
31.0
walker2d-r
1.310.1
1.611.7
5.111.3
5.41122
0.9-2.4
10.3:6.8
17.7
halfcheetah-m
43.2.10.6
48.310.3
47.010.5
47.410.2
45.310.66
58.311.3
59.2-2244
hopper-m
54.11338
59.314.2
53.0.128.5
66.21:5.7
98.6122.1
73.6110..
100.610.7
walker2d-m
70.9+11..0
83.74:2.1
73.3
78.3+48.7
86.8++0.8
88
82.9-
halfchectah-m-r
37.6+2.1
44.6+0.5
45.5+0.77
44,2+11.2
42.8+0066
51,5+00,2
46.4=1.2
hopper-m-r
16.614.8
60,9118.8
88.7112.9
94.718.6
76.2217.7
99.511.7
100.9
walker2d-m-r
20.319.8
81.815.5
81.822.7
73.817.1
87.342.3
83.341.9
38.3
balfchectah-m-cc
44.0=t1.6
90.7+4.33
75.6+
86.7+5.3
78.7+8.4
85.4++3.4
92.6=0.4
hopper-m-e
53.9114.7
98.011.4
105.6-12.9
91.5111.33
102.715.2
106.112..3
109.2
walker2d-m-e
90.1+13.2
110.110.5
107.9=1.6
109.611.0
110.4.1.5
110.310.1
109.040.3
Mujoco Average
36.5
58.2
61.8
59.9
64.1
68.4
70.7
Maze2d Average
-2.0
35.0
19.6
37.2
102.2
124.7
Adroit Total
93.9
0.0
93.6
110.7
123.3
149.6
Runtime (h)
0.3
0.4
10.8
0.4
illustrate the estimation of Q-values and use TD3+BC to highlight the over-constraint issue. For each state, compute the Q-values for every 0.01 increment within the action range [-1.0, 1.0], using the critic networks of TD3+BC and SQOG. The true Q-values are obtained by Monte Carlo method, where the discounted return is computed for the same state-action pairs under the same policy. To facilitate comparison, we smooth the values using cubic spline interpolation.
Based on the color bars Figure D can identify the OOD regions. In Figure I(a) the highest true value occurs within the range [-0.50, 0.50], corresponding to OOD regions inside the convex hull. In Figure [(b] the highest true value is located within [0.30, 1.00], representing OOD regions in the neighborhood of the convex hull. However, TD3+BC struggles with the over-constraint issue in these OOD regions, failing to accurately estimate Q-values for policy evaluation. In contrast, SQOG successfully estimates Q-values by smoothly generalizing in the OOD regions within the CHN (inside the convex hull in [(a) and its neighborhood in ((b).. b. These results from our sanity check demonstrate that improving Q-value generalization in the OOD regions within the CHN leads to better policy evaluation, reinforcing our theoretical analysis.
Results on D4RL benchmarks. We evaluate our proposed approach the D4RL benchmarks of OpenAl gym Mujoco locomotion tasks (Brockman et al.) 2016,] Todorov et al.) 2012). For baselines, choose representative offline model-free algorithms of different categories including BC, TD3+BC
7

---

Published as a conference paper at ICLR 2025
(Fujimoto & Gu 2021b D. CQL (Kumar et al. 2020). IQL (Kostrikov al. 2021b), well including DOGE (Li et al. 2022), , MCQ (Lyu et al. 2022 due to their high performance. For fairness, we choose four types of the "-v2" datasets (rrandom, m:medium. m-rrmedium-rrplay, m-c:medium- expert) for all methods, yielding a total of 12 datasets. We conduct additional experiments on Maze2d "-v1" datasets and 8 Adroit (Rajeswaran al. 2017) "-v1" datasets (see Appendix [ The results of BC, TD3+BC, CQL, IQL and MCQ are obtained from MCQ paper (Lyu et al. (0022)
DOGE (Li et al. 2022) is obtained from own paper. All methods run for M gradient steps.
In Table 1 we present the Mujoco results alongside the average scores for Maze2d and total scores for Adroit tasks. For detailed results, please refer to Appendix As demonstrated. SQOG consistently attains the highest scores on most datasets and achieves the highest average scores across the Mujoco, Maze2d, and Adroit tasks. The second-ranking MCQ algorithm approaches ours average score but runs 20 times slower. Compared to the representative method TD3+BC, SQOG demonstrates 70% performance improvement on hopper- v2 dataset and significant improvements in average scores, with minimal increase in computational cost. Based on the benchmark results, SQOG exhibits performance improvement, which aligns with our theoretical analysis indicating that SQOG achieves better evaluation.
Computational cost. Time complexity is a significant challenge in offline RL. We evaluate run time of training each offline RL algorithms for I million time steps, using the author-pree provided implementations or the re-implementations the source code using JAX. The results are reported in Figure In contrast to MCQ and CQL,, approach significantly reduces computational costs by avoiding the use of a generative model for behavior modeling (Figure 3}. Compared to TD3+BC, the supplementary OOD generalization term is computationally-fice due the low-computationallcost implementation of our OOD sampling methods.
Runtime
Key Features
MCQ
SQOG
10L
Loss Modification
Critic
Critic
Generative Model
CVAE
None
DOGE
OOD Sampling
From
Add noise
TD3+BC
Figure The key features of SQOG and MCQ
500G
(Lyu et al. 2022). The use of CVAE makes MCQ contrast, SQOG avoids the use of any generative model, achieving SOTA results with low computational cost.
2
4
7
time consuming.
Time(h)
Figure 2: Average run time on Mujoco lo- comotion tasks.
Ablation study. We conduct ablation studies on hyperparameter 3 and the noise type. The hyper- parameter controls the significance of the OOD generalization term in Qllearning, specifically balancing the learning weight between the OOD Q-values and the iamsam Q-values. We investigate the effects of four different values
understand its impact. Our findings (Figure 4(a) 4(b) [c))
indicate that B = 0.5 generally yields optimal performance. Larger values of make it difficult to achieve accurate in-sample Q-values, while smaller values of /9 hinder the learning of OOD Q-values, leading to reduced performance. Additional study on hyperparameter is provided in Appendix B.3
Additionally, we examined two commonly used types noise, both constrained to the same range [-0.5, 0.5]. As shown in Figures 4(d) straightforward and effective choice for dataset noise. In contrast, uniform noise tends to generate overly random OOD samples that sparsely and evenly distributed, which may limit its ability to focus on critical OOD regions. Further details, including an additional study on noise clipping, are provided in Appendix B.4 and and B.5
4(e) and 4(1) the normal noise with clipping appears to be
8

---

Published as a conference paper at ICLR 2025
Hupper-medium--e
hogper-medium-rees replay
halfchestah medium v2
Normalized Score
Normalized Score
Normalizzdd Score
x
SIN
Trainning Iterations
Trainiing Iterations
Trainning ltterations
(a) Efffctt of
(b) Effects of 3
(c) Effects of
hopper medium-dd x2
hopper-medium
hallcheetah-meddumee
Normalized Score
Normalized
Normalized by
Trainning Iterations
Trainning Iterations
Trainning Iterations
(d) Effects noise type
(e) Effects noise type
(f) Effects of noise type
Figure 4: Hyperparameter study and noise study on hopper-medium--2 popper--edium-replayyee seatheetah-mmeium- v2. The experiments are run for 1M gradient steps over 4 random seeds.
5 RELATED WORK
2000; Tenenbaum et al. 2000; Coifman & Lafon| 2006; Belkin & Niyogi 2001 Wang et al. 2004 Barannikov et al. 202 1 Hegde et al. |007]] Brehmer & Crammer 2020) is a kind of dataset structure learning method aiming to uncover the underlying structure of high-dimensional data. However, manifold learning faces challenges when applied to offline RL datasets due to the complexity of the data and the trajectory information in the high dimensional state-action space. Convex hull is another dataset structure used in dataset analysis and algorithm designation in supervised learning settings such as classification and regression (Fung al. 2005 Khosravani et al.) 2016; Xiong et al., 2014| Nemirko & Dulá, 2021; Xu al.) 2021). Understanding the structure of RL datasets facilitates the design of superior algorithms, enhancing their performance, stability, and generalization capability (Wang et al. 2020 Schweighooer et al. 2022, Hong et al. 2023). DOGE (Li et al., (022) was the first to apply the convex hull in the design offline RL algorithms. SEABO (Lyu al., (024), PRDC (Ran et al., 2023) and (Sun) al. 2023) utilize nearest neighbor techniques to address practical challenges. Building upon these previous works, we propose new dataset structure called CHN and analyze its safety guarantees for generalization.
Dataset structure. Manifold learning Rooweis & Saul
Model-free offline RL. Offline RL algorithms address the challenge of distribution shift by employing
different strategies. Model-free offline RL can be broadly categorized into policy-based based approaches
(Wang et al.
Fujimoto et al., 2019 Peng et al.
2019 Ran et al. 2023 Wu et al. 2019 Fujimoto
& Gu 2021
Kostrikov et al.] [202
Kumar et al. 2019 Li al. [022] Mao et al.
2024) and
value-based approaches (Kumar et al.
2020 Kostrikov et al.
202
Lyu et al. 2022
An et al.
2021
Ghasemipour al.
[022]] Xu et al. [023]] Zhang et 2023; Yang et al. 2024
Lee al.
2024
Geng et al. 2024). Policy-based approaches like BCQ FFjjimooo et al. 2019). BRAC (Wu
al.
2019) and TD3+BC (Fujimoto & Gu 202I) solely rely on the distribution of behavior policy,
leading to overly constrained learned policies. PRDC (Ran et al. (023) relaxes policy constraints by
utilizing the entire dataset, while DOGE (Li al. 2022) introduces a novel policy constraint that enables exploitation in OOD areas within the dataset convex hull. Instead of directly modifying the constraint term during policy improvement, we focus on policy evaluation. Our method, MQOG, achieves better policy evaluation, which indirectly addresses the over-constraint issue arising from policy improvement. Compared to value-basee approaches like IQL (Kostrikov et al., 2021b) and MCQ (Lyu al. 2022), which approximate the optimal in sample Q for Q-learning, MQOG aims to approximate the true Q for policy evaluation. Our theoretical and empirical results demonstrate that improved evaluation leads to enhanced performance.

---

Published as a conference paper at ICLR 2025
CONCLUSION
In this paper, we introduce Smooth Q-function OOD Generalization (SQOG) to achieve better policy evaluation by improving Q generalization in OOD regions within the CHN. We provide the safety guarantees for the CHN, indicating that our approach to OOD generalization is unlikely to adversely affect evaluation. SQOG trains OOD Q-values by constructing appropriate OOD target values with guidance from the Smooth Bellman Operator (SBO). Specifically, we use the neighboring in-sample Q- Q-values update OOD Q-values the SBO. Theoretically, we demonstrate that the SBO is appropriate and enables the Q-function to approximate the true Q-values within the CHN. Furthermore, we conduct sanity check to show that SQOG achieve better Q estimation by improving the generalization in OOD regions, thereby alleviating the over-constraint issue. Experiments the D4RL benchmarks demonstrate that SQOG outperforms baseline methods across most datasets, highlighting the importance of accurate evaluation in OOD regions within the CHN. We anticipate that our work will draw greater attention to Q-value estimation in OOD regions and provide new insights for the offline RL community.
Finally, it is crucial to emphasize that precisely solving the CHN and identifying all OOD regions within CHN for each dataset is impractical and unnecessary. However, various ingenious approaches can be employed to improve the estimation of the OOD Q-values within the CHN.
ACKNOWLEDGEMENT
This work was supported by the National Science and Technology Major Project under Grant 2022ZD01 16401, the National Natural Science Foundation of China under Grant 62141605, and the Fundamental Research Funds for the Central Universities, China. We acknowledge the support of pigh-performance computing resources provided by the School of Mathematical Sciences of Beihang University. We would like to thank Xiaotie Deng, Xu Chu, Yurong Chen, Zhijian Duan and the anonymous reviewers for their insightful comments on improving the paper.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty- offline reinforcement learning with diversified q-ensemble. Advances in neural information processing systems, 34:7436-7447, 2021.
Peter Auer, Thomas Jaksch, and Ronald Ortner. Near-optimal regret bounds for reinforce- ment learning. In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou (eds.), Ad- vances in Neural Information Processing Systems, volume 21. Curran Associates, Inc., 2008. URL F12/448822cbb3377700000022222222
wwwrips. wwwp://ii......euu.............e cc/paper files//paerr2208/ pdf
Serguei Barannikov, Ilya Trofimov, Grigorii Sotnikov, Ekaterina Trimbach, Alexander Korotin, Alexander Filippov, and Evgeny Burnaev. Manifold topology divergence: a framework for comparing data manifolds. Advances in neural information processing systems, 34:7294-7305, 2021.
Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps and spectral techniques for embedding and clustering. Advances in neural information processing systems, 14, 2001.
Alberto Bietti and Julien Mairal. On the inductive bias of neural tangent kernels, 2019. URL attps:://arixv org/ abs /tts//199..1...
Johann Brehmer and Kyle Cranmer. Flows for simultancous manifold learning and density estimation. Advances in Neural Information Processing Systems, 33:442 453, 2020.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. CoRR, abs/1606.01544 2016.
abs/1606.0154 01540
Xinyue Chen, Zijian Zhou, Zheng Wang, Che Wang, Yanqiu Wu, and Keith Ross. Bail: Best- action imitation learning for batch deep reinforcement learning. Advances in Neural Information Processing Systems, 13:18353-18363. 2020.
Ronald R Coifman and Stéphane Lafon. Diffusion maps. Applied and computational harmonic analysis, 21(1):5-30, 2006.
Onno Eberhard, Jakob Hollenstein, Cristina Pinneri, and Georg Martius. Pink noise is all you need: Colored noise exploration in deep reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023. URL hetLhhtps://ooo http:tpss///peereeeww..e.ee mermffrum?
id=ho9V5QN2 1e8
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2021.
Scott Fujimoto and Shixiang (Shane) Gu. A minimalist approach to offline reinforcement learn- ing. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang. and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 20132-20145. Curran Asso- ciates, Inc., 2021. URL.httpss:/prrceeeeee cepppp..eeppppp..ppppprpr.... ff files/paper/ 2021/file/at http:///0010..000000000000000010
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference Machine Learning, pp. 2052-2062, 2019.
Glenn Fung, Romer Rosales, and Balaji Krishnapuram. Learning rankings via convex hull separation. Advances in Neural Information Processing Systems, 18, 2005.
Divyansh Garg, Jocy Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learning: Maxent rl without entropy. arXiv preprint arXiv:2301.02322 2023.
Sinong Geng, Aldo Pacchiano, Andrey Kolobov, and Ching Cheng. Improving offline rl by blending heuristics, 2024. Utl.https::/arxii..oo
/ 00321]
11

---

Published as a conference paper at ICLR 2025
Kamyar Ghasemipour, Shixiang Shane Gu, and Ofir Nachum. Why pessimistic? estimating uncertainties for offline rl through ensembles, and why their independence matters, Advances in Neural Information Processing Systems, 35:18267-18281, 2022.
Tuomas Haarojj,, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pp. 1861-1870 PMLR, 2018.
Chinmay Hegde, Michael Wakin, and Richard Baraniuk. Random projections for manifold learning. Advances in neural information processing systems, 20, 2007.
Zhang-Wor Wei Hong, Aviral Kumar, Sathwik Karnik, Abhishek Bhandwaldar, Akash Srivastava, Joni Pajarinen, Romain Laroche, Abhishek Gupta, and Pulkit Agrawal. Beyond uniform sampling: Of- fline reinforcement learning with imbalanced datasets. Advances in Neural Information Processing Systems, 36:4985-5009, 2023.
Longyang Huang, Botao Dong, Jinhui Lu, and Weidong Zhang. Mild policy evaluation for offline actor-critic. IEEE Transactions on Neural Networks and Learning Systems, 2023.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. CoRR, abs/1806.07572, 2018. URL http://arxiv.org/
abs/ 18066075600 07572
Hamid Reza Khosravani, AE Ruano, and Pedro M Ferreira. A convex hull-based data selection method for data driven models. Applied Soft Computing, 47:515-533, 2016.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua
Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. http
7/arxiv. org/ abs 1412. 6980
Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pp. 5783- PMLR, 2021a.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXhv:2110.06166 2021b.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Sta- bilizing off-policy q-learning via bootstrapping eгтor reduction. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bue, E. Fox, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL htttps::/proceeeiiiii copp ifrffill www.ss/pppeeee...0///
nwwrins. pdf
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances Neural Information Processing Systems, 33:1 179-1191,, 2020.
Haanvid Lee, Tri Wahyu Guntara, Jongmin Lee, Yung-Kyun Noh, and Kee-Eung Kim. Kernel metric learning for in-sample off-policy evaluation of deterministic rl policies, 2024. URL
Jianxiong Li, Xianyuan Zhan, Haoran Xu, Xiangyu Zhu, Jingjing Liu, and Ya-Qin Zhang. When data geometry meets deep function: Generalizing offline reinforcement learning. arXiv preprint arXiv:2205.11027 2022.
Jiafei Lyu, Xiaoteng Ma. Xiu Li, and Zongqing Lu. Mildly conservative g-learning for offline rein- forcement learning. In S. Koyejo. S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 1711-1724. Curran Asso- ciates, Inc., 2022.
neur ps cc http:/aaeer__ee wttpapprreeeeleeppppeerpeee wttp:ppppppppppppppppppep
0b5 2222:226655555000022222222681663
Jiafei Lyu, Xiaoteng Ma. Le Wan, Runze Liu, Xiu Li, and Zongging Seabo: A scanhh method for offline imitation learning, 2024. URL www.hhttps///arri.ror///////22..
Seabo
12

---

Published as a conference paper at ICLR 2025
Chenjie Mao, Qiaosheng Zhang, Zhen Wang, and Xuelong Li. On the role of general function approx- imation in offline reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024. URL httthhts::/// htttps::/openreveeet net/ http:///www.......oo.orooddaaaa.
AP Nemirko and Dulá. Machine learning algorithm based on convex hull analysis. Procedia Computer Science, 386:3113866 2021.
Ian Osband and Benjamin Van Roy. Why is posterior sampling better than optimism for reinforcement learning?, 2017. URL hwwphhttp:/ataae httpp/pppp/w.....og/////o/ abs/ 1607. 00215
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv: 19000 2019.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. CoRR, 10087
2017. httt:://arxav.crrrabb////1
Yuhang Ran, Yi-Chen Li, Fuxiang Zhang, Zongzhang Zhang, and Yang Yu. Policy regularization with dataset constraint for offline reinforcement learning. International Conference on Machine Learning, 2023.
Sam T Roweis and Lawrence K Saul. Nonlinear dimensionality reduction by locally linear embedding. science, 290(55000:2333-22338 2000.
Kajetan Schweighofer. Marius-constantin Dinu, Andreas Radler, Markus Hofmarcher, Vihang Prakash Patil, Angela Bitto-Nemling, Hamid Eghbal-zadeh, and Sepp Hochreiter. dataset perspective on offline reinforcement learning. In Conference Lifelong Learning Agents, pp. 470-517. PMLR, 2022.
Hao Sun, Alihan Hüyük, Daniel Jarrett, and Mihaela van der Schaar. Accountability in offline reinforcement learning: Explaining decisions with a corpus of examples, 2023. URL https
//arxiv. www./arrrraaa. abaaaba///aa/ /et.//31..0000 07747
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. Bradford
Book, Cambridge, MA, USA, 2018. ISBN 0262039249.
Joshua B Tenenbaum, Vin de Silva, and John C Langford. A global geometric framework for nonlinear dimensionality reduction. science, 290/5500):231--2333, 2000.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, 2012. doi: 1..109//ROS..012.66666
Jing Wang, Zhenyue Zhang, and Hongyuan Zha. Adaptive manifold learning. Advances in neural information processing systems, 17, 2004.
Kaixin Wang, Bingyi Kang, Jie Shao, and Jiashi Feng. Improving generalization in reinforcement learning with mixture regularization. Advances in Neural Information Processing Systems, 33: 7968-7978, 2020.
Qing Wang, Jiechao Xiong, Lei Han, Han Liu, Tong Zhang, et al. Exponentially weighted imitation learning for batched historical data. Advances in Neural Information Processing Systems, 31, 2018.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. CoRR, 110/19911.116661 2019. htt.:httpp//arxi..rrr///////
Yuanjun Xiong, Wei Liu, Deli Zhao, and Xiaoou Tang. Zeta hull pursuits: Learning nonconvex data hulls. Advances in Neural Information Processing Systems, 27, 2014.
Hailong Xu, Longyue Li, Pengsong Guo, and Changan Shang. Uncertainty active learning algorithm based on convex hull and sample distance. In 2021 33rd Chinese Control and Decision Conference (CCDC), pp. 6815-6822. IEEF, 2021.
13

---

Published as a conference paper at ICLR 2025
Haoran Xu, Li Jiang. Jianxiong Li, Zhuoran Yang. Zhaoran Wang, Victor Wai Kin Chan, and Xianyuan Zhan. Offline with no ood actions: In-sample learning via implicit value regularization. arXiv preprint arXhv:2303.15810 2023.
Rui Yang. Chenjia Bai, Xiaoteng Ma, Zhaoran Wang, Chongjie Zhang, and Lei Han. Rorl: Robust offline reinforcement learning via conservative smoothing. Advances in neural information processing systems, 35:23851-23806 2022.
Rui Yang. Han Zhong, Jiawei Xu, Amy Zhang, Chongjie Zhang, Lei Han, and Tong Zhang. Towards robust offline reinforcement learning under diverse data corruption, 2024. URL https://
arxiv.org/abs//330. 12955
Hongchang Zhang, Yixiu Mao, Boyuan Wang, Shuncheng He, Yi Xu, and Xiangyang Ji. In-sample ac- tor critic for offline reinforcement learning. In The Eleventh International Conference Learning Representations, 2023, URL
14

---

Published as a conference paper at ICLR 2025
A ADDITIONAL THEORETICAL RESULTS AND MISSING PROOFS
Before providing the missing proofs, we briefly introduce the Neural Tangent Kernel (NTK) (Jacot et al. 2018). NTK is widely used in the analysis of the generalization, the convergence and optimality of deep RL. In this paper, we complete our proof of Proposition and Theorem and under the
NTK regime.
Following DOGE eti al. 2022), we introduce Assumption I and Lemma and
We assume the function upproximators discussed in our paper are two-layer fully- connected ReLU neural networks with infinity width and are trained with infinitesimally small learning rate unless otherwise specified.
Assumption 1.
Lemma (Smoothness of the kernel map of two-layer ReLU networks). Let be the kernel map of the neural langent kernel induced by a two-layer ReLU neural network, x and be two imputs, then satisfies the following smoothness property.
[1x(z) <(v))1 S /min(] |1ylD|[ yl + 2||.r vll
(16)
For the proof of Lemmal we refer the reader to (Li et al. 2022).
Lemma (Smoothness for deep Q-function). Given two inputs x and x', the distance between these two data points is d = || 1ız-z0 - C' is finite constant. Then the difference between the output and the output at can be bounded by:
[1Qo(z) Q+(x')) S CCCCmCC///llll// | DVv - 2d
(17)
For the proof of Lemma 2
refer the reader to (Bietti & Mairal 2019).
We obtain this theorem by generalizing DOGE's (Li et al. 2022) Theorem which proved under Neural Tangent Kernel (NTK) regime. We recall the main results in our theorem: Given a point within convex hull #1 E (D). a point in the convex hull neighborhood x2 E (Conn((D)). and an external point æ3 (8, A) CHN(D), we have,
Proof of Proposition I
[1Qa(11) - (z(Projn(z)))I < C1 min x |l. (rojp( /d + 2dz) $ M1
(18)
[[Qa(12) (2)))))n(2)))1 < Ci(v min( [2 (12)) V de + 2d2) S M2
(19)
- (xpppi)))p )) s Ci(y min[m 23]1 [|rroj [I) V dg + 2ds)
(20)
where Projp(x) (x)): argmin,, | -; || is the projection to the dataset. d1, d2, d3 are the point- to-dataset distances. Both d1 = [ - Projp (01) S ma ||z| x'|l < B and dy = x2 Projp(Z2) < < B are bounded. d3 = 23 (roj) >> where is the neighborhood radius and B = sup {|| - yl x, E Conv (D)} the diameter of the convex hull. let r maX,eCone(P)) || Proj (+) then S B. C1, M1, M2 are constants.
DOGE classify data points into Zin (the interpolate data in convex hull) and Tout (the extrapolate data outside convex hull). However, Tout be further divided into 2a (the data in the neighborhood of convex hull) and 2g (the data outside CHN).
With Lemma 2 we can derive the following bound:
[[Qa(z;) (zroi < C1(v mm/mmnnnaaa (Prjj( [P /d + 2d4)
< C1(( min
[Projo l.
2B) s Mr
Similarly, we have:
[1Q)(z2) (Projn _ C1 min( [2
Proj
122
dg + 2d2)
< C. min(
[22)) [Poojo
2r) s M2
Since d3 = ||ا - (r>) > r, Eq. (0) is no longer bounded, indicating that the error grows uncontrollably for data points far outside the neighborhood of the convex hull. However, we demonstrate that the approximation error within the convex hull's neighborhood can still be effectively controlled. To up, expand the safe generalization boundary of Q-function.
15

---

Published as a conference paper at ICLR 2025
Proof of Proposition 2 The Q-function defined CHN is uniformly continuous:
Ye > 0, 30 O, s.t. Vx,, 2. E CHN(D), if || zyy < 6, then [[Q(z,) - Q(xx)) E.
Proof. To simplify the notation, let X = CHN, which is compact. For any arbitrary & > 0, for every x E X, we can find 6z > such that for all x' E B(x, lI@(+) - Q(a')) < 8/2 (since Q is continuous). The collection B = [B(x, 5x) x E X) is an open cover of X. Define B, = [B(z, 62/2) æ E X}; j; this is also an open cover of X. Since X is compact, there exists a finite subcover of B1, denoted by B3 (B ong /2) 2) MKm
Let = min {ōm1, Se.........., J/2. For any y. E X, if Ilv - =|| < 6, without loss of generality, assume ||y (iy-20 rxll dza for some k. By the triangle inequality, || zx|l S ||2 yl lly xx < 5+8gg/2 /2 < 82. Therefore, y, 2 E Bx ) and by the properties of Q and the triangle inequality,
[1Q(y) Q(=)]l <.
Thus, for any 6 > 0, there exists > 0 such that for all y. E CHN(D), if ly - =1l < 6, then [1Q(v) Q(=)l| <.
Proof of Theorem I Assuming that max [ L(T, uu) K kL), S e. Then, under the NTK
regime, for all (s,a) D, with high probability 2 1 8,8 (0,1).
Cr..5 VID(s,a]
||BT Qo s
+ -mxx
V min((Eas || ( a')ll E Ear a') 2d
sampling bound
OOD overestination error bound
(21)
where Cr.1,5 = Cr,6 + CCr Raaz/ 5
3Crs and d S [|amen 1P+ 1 mlamax
V2- Cr.6
2 and Cr,5 is constants dependent the concentration properties of r(s,a) a) and T(s'|s, a). (D(s, a)] is the dataset size, Omin and amar denote the minimum and maximum actions. C is a constant. Similar to (Kumar al., 2020 Auer al., 2008 Osband & Roy 2017), we assume concentration properties of the reward function and the transition dynamics.
viD(s.o)
Assumption 2. v(s,a) E D, with high probability 2 1 8, we have,
Cr.
Cr.5 VID(s,a)
||- S
|| T(s'ss,
(22)
aiD
where Cr,s and CT,s is constants dependent the concentration properties of r(s, a) and ( [D(s, a)) is the datasel size.
Proof. From Assumption 2 we have,
||BT TT II(r - r(s, +7
T a) Qo a')ll
s - r(s, + +/11"(//1,,,)---1111/) T(s'(s, max Ex Q Qo a'',
Crs
CI.5 max E. Qa (5', a')ll
+1 V))(s
VID(s,a) a)
where max, |E 2r (s', a')l S maxx Qorss -a) Qa a'), max a'>N 112 (s', a')ll For simplicity, we denote Ex Qu a') as E, Qo (s', a'). From Lemma 2
we have,
a') - (s a d')ll (iin(Eaa (ss',, (o'",")) " 2d
For distance d, by applying the Pinsker's Inequality, we have,
d | -x [a'] (ae 11 f a' a'(a(a')ss) - u(a'|s') da'll
A
s/) - -(d''s'') S ||ansirl! 112 + ||amaz 112 E
||a'll da'- sup a'EA
2
Total variation distance
16

---

Published as a conference paper at ICLR 2025
Where amnn and amaz denotes the min and max action, C' is constant. The reward function is bounded, then mäXs' ,e'Nu QQ a')ll I Rmaz/ - 7). Therefore, we have,
_ Qo - 18. Q0ll & Cns + 7OT, Rman/ (1-7)
(D((s, a)
sampling error bound
TT.
a')ll, Rat (dt /ol 2d
max
ViD(s,a)
OOD overestimation bound
I- The reason is that π may generate OOD actions for Qø (s', a'). and the neural network could overestimate the Q-values for these OOD actions, making it impossible simply bound them. However, since p does not produce OOD actions, we can bound max,' ,a-u QQ (s',a')) aa)ll by Russ Therefore, is not
Remark. One might ask why we can't directly set max, [en (s','')))) a')l I Bun.
sufficiently close nt, cannot bound | Qo B Qoll
Proof of Theorem For the ia-ple evaluation, 91 introduces negligible changes to the empirical Bellman operator 1n Qo- Under the NTK regime, given (s, a) E DD assuming that in neigghbrrll < 6, we have,
Va' ~ (-1ss),, En neighbor s.t.|la'
1 - BTT (s(ss S C 7B. Vmin(x, v)Vo 26
(23)
where x = B. Ba.'non.la' - aetghoo -l lssl
a neightbor 11, y Eo' (1(s,, a')ll, C' is constant.
Proof. Similar to the proof of Theorem | we can derive the inequality from Lemma in this case is bounded by 8 from the assumption.
The distance
Assuming that for all (s, a) E D, (s, = Q= a). Given (s, a) $ D and as CHN, assuming that ||a neighhorr |I I 5 and [Q" Q" in neighbor)) <
Proof of Theorem 3
am
if (1 (*1ss, neigghtor IIl > by applying the SBO B for gradient descent updates (with infinitesimally small learning rate), we have.
1Q"(s, a) 1 (s,a)) < (s," Q* If I[ (s. - (s, neighberr )l S e, then I (s,a) Q" (s, a)ll 2E.
(24)
Proof. Given (s, a) & D. (s, < Q an meighbbrr) e, then (* < (s, From the SBO, we define the MSE loss as:
L 1Q (s, a) - -Q (8, in aneightor)) )]2
for gradient descent updates. Then,
Q++1 (x, a) = Q" (s. a) + 20(Q" (s, in where the learning rate is infinitesimally small. Consequently, (s, (s,t (s,aa) and we have,
(zeighoor) (=)
(IQT(s.)) Q a)ll < 11 Q* a)) The proof for the case (s, > (s, abeighhor) + follows a similar approach.
For (s) - Q+ neighhbrr)) I s. if (*, a then by applying the gradient descent method, we have, (=, Qu+2 Q* an neightor) Similarly, if (* an neighhor) I (*(s, a) I (" ahiighor)) Qst Q* (s. Therefore, || (s.a) - Q² (s, ain onesghber)) < always holds. By the triangle inequality, we have 11, - (s, < 28.
(s, < Q* ath ateshhbor
+ e, then Q* neighbor) S
17

---

Published as a conference paper at ICLR 2025
Proposition 4 (Convergence of SBO). Assume that the OOD actions generated by a(-ss') lie within the CHN. Then the SBO is a " contraction operator in the & norm. Any initial Q-function can converge to unique fixed point by repeatedly applying the SBO.
We prove the 7-contraction property of the SBO operator under the assumption that OOD actions generated by (--ss') lie within the CHN. This assumption can be satisfied with high probability if we incorporate a regularization term into the actor loss to constrain и closer to μ4, such as using a behavior cloning (BC) loss.
Proof of Proposition
The SBO is 7 contraction operator in the Lo Any initial Q-function can converge to a unique fixed point by repeated application of the SBO.
We first recall the definition of smooth Bellman operator,
(²)(ssa) (G)
(25)
where
Q(s,a), Q(, a neighbor))
A(als) 0
G1Q(s,a) =
(26)
M(aas) = and (s, a) E CHN
EsanswDlr + ylB ( (Q(s d')ll, p(als) > a)s
()Q(s, 1
(27)
((ass) 0 and (s, a) CHN
Proof. Given policy n, let Q1 and Q2 be two arbitrary Q-functions. Since D, have,
||8 Q1 - B" @2 = || Q1 9 221
max 9i |r + yB. a' Q1 a') - - yB at Q2(5', d')ll
8.a
max و E
01 (s', - Q2(5',a')l
(28)
9.8
max
/.a 19 91Q2('',
8,a
If ((ala(ss)) > 0, then |19 0') 9 (5 d')l 110 a'(s','')l,, we have,
|18 Q1 121o2 = max 7 - 101 Q2(s','))l
S 7 max 1| 2211o
(29)
810
>1111 - Q2
So we can find that [B - B S >11Q1 - Q21loo.
If A(a'lss) 0 and (8,a) a) CHN, then Q a') 9 = (s', tn
oeighhor)
Q2 an aeeightor) we have.
118T -3 Q2llb
max B a neightorr Q2(s', a neighborr
8,0
S max QQ Q211o0
(30)
>||Q1 Q21loo
Combining the results together, conclude that the smooth Bellman operator is 7/- contraction operator in the L
18

---

Published as a conference paper at ICLR 2025
B ADDITIONAL EXPERIMENTS AND EXPERIMENTAL DETAILS
B.I RESULTS ON ADDITIONAL DATASETS
Table 2: Normalized average score comparison of SQOG against baseline methods on Maze2d "-v1" and Adroit "-v1" datasets over the final 10 evaluations and 4 random seeds. We bold the highest scores.
Dataset
BC
CQL
BCQ
TD3+BC
IQL
MCQ
SQOG
maze2d-umaze
-3.2
18.9
49.1
25.7.1..1
63.3.113.4
81.5.123.7
226.4.110.4
maze2d-umaze-dense
-6,9
14.4
48.4
39.713.8
12.8
107,8:13.2
100.4.1..9
maze2d-medium
-0.5
14.6
17.1
19.5.442
23.541111
106.8:138.4
1.9.4
maze2d-medium-ddese
2.7
30.5
41.1
54.916.4
28.1+16..8
112.7+555
122.7-00.
Maze2d Average
-2.0
19.6
38.9
35.0
37.2
102.2
124.7
pen-human
34.4
37.5
68.9
0.0+-0.0
68.7+8.66
68.5+6.5
80.0+44.7
door-human
0.5
9.9
0.0
0.0+0.0
3,3+1.3
2.3+2.2
1.0+1.1
relocate-human
0.0
0.2
-0.1
0,0+0.0
0.0+0.0
0.1+0.1
0.1+0.0
hammer-human
1.5
4.4
0.5
0.0+0.0
1.4+0.66
0.3+0.1
1.4+0..7
pen-eloned
56.9
39.2
44.0
0.0+00.0
35.3+7.3
49.4+-..3
66.7+33.4
door-cloned
-0.1
0.4
0.0
0.0+00.0
0.5+0.6
1.3+0.4
-0.1
relocate-cloned
-0.1
-0.1
-0.3
0.0+00.0
-0.2+0.0
0.0+0.0
-0.1
hammer-cloned
0.8
2.1
0.4
0.0+0.0
1.7+10
1,4
0.6+0.3
Adroit Total
93.9
93.6
113.4
0.0
110.7
123.3
149.6
To further illustrate the effectiveness of the SQOG algorithm, we conduct more experiments on Maze2d "-v1" datasets and 8 Adroit "-v1" datasets from D4RL benchmarks. The Maze2d tasks involve navigating 2D maze environment from an initial point to a designated goal, presenting intricate pathfinding challenges due the maze's 's structural complexity and the presence obstacles. Conversely, the Adroit environment engages agents in object manipulation within physics- based simulation, demanding adept control and adaptability across diverse scenarios. The Maze2d datasets feature non-Marke Mon-Markovinn policies, yielding undirected and multitask data, while Adroit datasets exhibit heightened realism characterized by non-representable policies, narrow data distributions, and sparse rewards.
We compare MCQ against BC, CQL Kumar al. 2020). BCQ (Fujimoto et al. 2019). TD3+BC
(Fujimoto & Gu 2021 IQL (Kostrikov et al. 2021b) and MCQ (Lyu et (2022). We take the results on these datasets from Lyu et al. (022) directly. From T we observe that SQOG consistently outperforms existing algorithms on the Maze2d datasets, while demonstrating competitiveness with prior methods on Adroit tasks. Remarkably, SQOG achieves the highest average score across all Maze2d and Adroit datasets, underscoring its superior performance. Additional results further corroborate SQOG's efficacy across diverse dataset types, thus attesting to its robust generalization capabilities across varied environments. Moreover, SQOG demonstrates consistent and low runtime on Maze2d and Adroit tasks, with an average runtime of 0.4 hours. Therefore, we believe that its strong performance and computational efficiency make SQOG noteworthy contribution the offline RI. community.
B.2 WIDER EVIDENCE ON Q-VALUE ESTIMATION OF SQOG
We have presented intuitive sanity check to demonstrate that SQOG alleviates the over-constraint issue and achieves more accurate Q-value estimation. This sanity check is conducted on the Inverted Double Pendulum task, which features a one-dimensional action space and a suitable level of task complexity. To further validate SQOG's effecttiveness, perform additional experiments locomotion tasks to show that SQOG maintains accurate Q-value estimation in higher-dimensions. environments.
The results, shown in Figure are obtained by estimating true Q-values using the Monte Carlo method, similar the sanity check. The critic Q-values computed using SQOG's critic network, and both sets of Q-values are normalized for ease of comparison. We compute the difference between these normalized Q-values. Across all datasets, the value difference consistently decreases
19

---

Published as a conference paper at ICLR 2025
hopper medium v2
hopper medium replay
halfthertah medium
Normalzed
Normal leed
Normal feeed
Cosiis Vakue Ofference
O Wiee offurance
O
Vite Offerance
Training lteraaioon
Training Iterations
Training Iterations
Figure 5: Q-value difference between critic Q-value and true Q-value on hopper-mediume hopper- medium-replay-v22 halfcheetah-meddine medium-v2. The experiments are run for 1M gradient steps over random seeds.
training iterations, indicating that SQOG achieves accurate Q-value estimation. Notably, SQOG demonstrates the most stable value estimation on the hopper-medium-VV dataset, aligning with its stable, SOTA performance on the same task.
B.3 TRADE-OFF BETWEEN GENERALIZATION AND CONSERVATIVE BEHAVIORS
The trade-off between generalization and conservative behaviors is primarily influenced by two hyperparameters: B and a. 3 controls the significance of the OOD generalization term in the critic loss, as demonstrated our ablation study. a governs the relative intensity of the behavioral cloning penalty in the actor loss. Smaller values of encourage more conservative behaviors. To investigate the impact of o, we conduct a series of experiments with varying a values, fixing 3 = 0.5. The results summarized in Table
Table 3: Normalized average score of SQOG over different choices of different values (with = 0.5) on Mujoco "-v2". The results are averaged over different random seeds. We bold the highest scores.
Dataset
300
a 200
a 150
100
50
halfchectah-meddun
59.3
57.711.
59.242.4
54.1
51.1100.4
hopper-medium
70.9=7.7
91.6:16.0
100.610.7
94.1
74.9+4.11
malker2d-medine
83.6+7.55
88.2+1..8
82.9+0.8
81.7+10.8
81.1+0..
halfcheetah medium-replay
41.4=00.6
42.5#22.3
46.4+:12
37.8+1.6
36.1=0055
hopper-mediun-replag
74.14:5.5
86.9-18.5
100.9
84.0+19.4
82.8
walkerrdd-mediun-rrplay
39.0+12.3
42,4477.4
88.3
73.1+8.0
68.5++5.3
Mujoco Average
61.4
68.2
79.7
70.8
65.8
datasets like halfcheetah-medeee and walker2d-medium larger a values (e-g., o 300 or a 200) tend to perform better, suggesting that these tasks benefit from more aggressive generalization. The dynamics of these environments may not require highly conservative strategies, allowing the agent to explore beyond the dataset effectively. Across tasks, the results indicate that moderate value of 150 provides a strong balance between generalization and conservative behaviors. This balance likely ensures sufficient exploration without risking the instability of over-generallization making it a robust choice across diverse scenarios.
B.4 ADDITIONAL EXPERIMENTS ON NOISE SCALE AND CLIP
To further study the effects of the noise scale and clipping range, we conduct additional experiments by systematically varying the scale and clipping parameters observe their influence performance across multiple datasets. We present the results in Table
Across most datasets, scale of 0.6 and clip of 0.5 consistently achieves strong performance, suggesting that moderate noise levels effectively balance exploration and stability. While noise promotes generalization, excessive noise can lead sampling outside the CHN boundary, resulting
20

---

Published as a conference paper at ICLR 2025
Table 4: Normalized average score of SQOG over different choices of Gaussian noise scale and clip on Mujoco "-v2" and Adroit "-v1" datasets. The results are averaged over 4 different random seeds. We bold the highest scores.
Dataset
scale=0, clip=0
scale=0.2, clip=0.3
scale=0.6, clip=0.5
scale= clip=0.7
scale=2.0,
clip=1.0
malfcheetah-medium
51.2.1399
6..610.5
59,2:12.4
53.1
51.7=1.5
hopper-medium walker2d-medee
1.5
6..2
100.6:10.7
94.511.66
10.5
48.3+1.2
58.9+2.3
82.9+0.8
86.0=11.0
82.5+1.1
saffcheeeta-mediimne
0..
47.
46.4+1.2
36.6#1111
35.8=0.8
hopper-mediunn-eplla
22.3:16.6
19.4
100.9
24.3:12.7
27.1
wallerr2d-mediun-replay
12.24:5.5
56.8
88.343.5
90.1
60.6:16.9
Mujoco Average
30.8
51.8
79.7
64.1
56.2
pen-human pen-cloned
23.316.7
46.714.88
80.0
64.9
55.8=5.9
9.5
41.247.66
66.7133.4
43.2
42.3+7.0
Adroit Average
16.4
44.0
73.4
54.1
49.1
in OOD Q-values that violate safety guarantees and degrade performance. Properly scaled noise ensures effective learning while respecting safety constraints.
The sensitivity to the noise distribution parameters (scale and clip) varies across datasets. In haffcheeual-mediun-rrpaaaa the baseline configuration (scale=0, clip=0) yields the best perfor- mance, indicating that the effect of noise on in-sample Q-value estimation cannot be ignored. The dataset may already provide sufficient diversity, and adding noise introduces harmful uncertainty, leading to performance degradation. For such datasets, crucial to keep noise parameters conser- vative to avoid disrupting in-sample Ql-eaaniig and maintain stable performance. In contrast, for the optimal configuration is (scale=0.2, clip=0.3), and for walker2d meddim (scale=1.0, clip=0.7) achieves the best results. These differences
and malker2d-medium-ree medium--epplaa highlight that there room for improvement in fine-tuning noise parameters across various datasets. Combined with the results of experiments on the hyperparameter we find that larger noise clipping and larger a achieve better performance in the walker2d-medium dataset. This implies that the task benefits more from Q-value generalization and aggressive policy extrapolation within the CHN.
The baseline configuration (scale=0, clip=0) significantly underperforms in most datasets, underscor- ing the necessity of noise injection to enhance exploration and overall performance. Noise injection essential for improving the Q-function's 's ability to generalize to previously unexplored regions and optimize learning.
B.5 ADDITIONAL EXPERIMENTS ON NOISE TYPE
To better analyze the performance differences among noise types, we conducted additional experi- ments using three different noise settings: normal noise with a scale of 0.6 and clip [-0.5, 0.5], normal noise scaled through a tanh transformation to [-0.5, 0.5], and uniform noise within [-0.5, 0.5]. Results across 4 random seeds are presented in T
From Table 5 it is evident that uniform noise significantly underperforms on Mujoco datasets, with higher variance compared normal noise settings. This is likely due uniform noise generating overly random OOD samples that are distributed sparsely and equally across the entire range of [-0.5, 0.5). While this ensures coverage across the OOD region, fails to focus sufficiently on some key areas that are critical for Q-learning Consequently, uniform noise may lead to insufficient training in each region, resulting in unstable and inconsistent performance.
Normal noise (both scaled+clip and tanh-transformed) performs better due to its concentrated sampling behavior. For normal+scale+clipp noise, samples are densely concentrated at the clip boundaries (e.g., -0.5, 0.5). For normal+tanh noise, most samples concentrated near the boundaries (e.g., -0.5, 0.5). This hundary--ooussee sampling behavior is likely to provide relatively adequate training samples and encourage the generalization in critical OOD regions.
21

---

Published as a conference paper at ICLR 2025
Table 5: Normalized average score of SQOG with different noise types Mujoco "-v2" and Adroit " "-vI" datasets. The results are averaged over 4 different random seeds. We bold the highest scores.
Dataset
normal+0.6 scale
normal+tanh
uniform
haffcheetah-medium
59.2++2.4
59.3+0044
47.4+11.1
hopper-medium
100.6:10.7
89.7=7.4
62.5+36.5
walker2d-medium
82.910.88
82.8#3.2
65.5=17.9
saffcheetah-medium--ee
46,4+11.2
39.9#1.66
40.1
hopper--medium-rplla
100.9:5..1
94.5#5.9
37.9=38.3
walker2d-meddiu--repay
88.3
62.5#17.4
46.8
Mujoco Average
79.7
71.5
49.9
pen-human pen-cloned
30.0.1:4.7
75.3+5.8
75.4
66.7+3.44
64.5
61.9+4.7
Adroit Average
73.4
69.9
68.7
Adroit datasets, differences between noise types are less pronounced, suggesting that the clip range [-0.5, 0.5] plays more critical role than the specific noise type. Within this range, all noise types perform reasonably well.
Our observations suggest that the poor performance of uniform noise may result from its overly sparse and evenly spread sampling. which appears to limit its ability to provide sufficient coverage of some critical OOD regions for Q-value generalization. In contrast, normal noise with clipping or tanh transformations demonstrates potential advantages due to its boundary-focused sampling, which may facilitate sufficient learning in those critical OOD regions. While experiments demonstrate the superiority of normal noise with clipping in this context, we acknowledge that noise type and its influence on OOD action sampling is important topic deserving deeper exploration. Our primary contribution lies in introducing SQOG, which effectively addresses over-constraint issues in offline RL by improving OOD Q- value generalization, delivering superior performance and computational efficiency. In this work, we focused empirically validating the feasibility of normal noise (with clipping) and conducted preliminary analyses on the effects noise type and range.
As future work, we plan to conduct more systematic investigation into the role of noise in OOD sampling and Q-function generalization, aiming establish a clearer theoretical understanding of its impact.
B.6 ADDITIONAL EVALUATIONS WITH MORE SEEDS
We extended the evaluation of SQOG Mujoco "-v2" datasets by running it with another 4 random seeds, bringing the total to 8. We believe this is a sufficiently robust number for reliable evaluation. The summarized results presented in Table 6 where SQOG demonstrates consistent and reliable performance across different settings.
B.7 COMPUTE INFRASTRUCTURE
In Table 7 we list the compute infrastructure that we use to run all of the baseline algorithms and SQOG experiments.
B.8 EXPERIMENTAL DETAILS
The true Q-values sanity check. sanity check, we calculate the true Q-values using Monte Carlo estimation method ensure accuracy. Specifically, reset the environment to given state s and execute the action a. Starting from (s, a). we simulate full trajectories and calculate the discounted return for each trajectory. To approximate the expected return, repeat this process for
In this context, the term "true Q-value" refers to the ground truth Q-values computed through Monte Carlo estimation. These values serve reference standard for evaluating the accuracy of the predicted Q-values generated by the critic networks.
22

---

Published as a conference paper at ICLR 2025
Table 6: Normalized average score of SQOG on Mujoco "-v2" datasets. The results are averaged over 4 and different random seeds
Dataset
SQOG (4 seeds)
SQOG (8 seeds)
halfcheetab-random
25.6 = 0.4
25.7 + 1.8
hopper-random
15.6 3.3
15.4 + 4.1
walker2d-random
17.7
16.9 = 3.1
mulfchectaahmeddiee
2
6..1
hopper-medium
107
100.8
walker2d-medium
82.3
4
4
hopper-meeduum-replay
100
99.5
walkerr2d-mediuun-rrpaayy
3.5
5.8
halfcheetah- wwwchhh-mmiiiiieeeeeeeee expert
92.6
91.6
hopper-medium weetoe--miiideppeeeeeeee expertt
109.2
110.1 = 2.5
walker2d-medium--ee
109.0 0.3
109.1 + 0.3
Mujoco Average
70.7
70.3
Table 7: Compute infrastructure.
CPU
GPU
Memory
Intel(R) Xeon(R) CPU E5-2698
Tesla V100-DGXS-32GH x 8
251G
Intel(R) Xeon(R) Silver 4216
GPU GeForce RTX 3090
62G
1000 sampled trajectories and take the average. We compute the Q-values for every 0.01 increment within the action range [-1.0, 1.0] and smooth the values using cubic spline interpolation. Finally, normalize the values to keep them between and by multiplying by an appropriate constant. Given the computational intensity and rigorous sampling, this process provides robust approximation of the true Q-values.
The true Q-function in the Mujoco environment could be irregular or even stepwise. However, note that the smooth appearance of the Q-values in Figure arises from the use of cubic spline interpolation applied densely sampled data points. The interpolation is used solely for visual clarity and does not affect the underlying accuracy of the Q-value computation. Without this interpolation, the individual sampled points would still demonstrate the high accuracy of our SQOG method in estimating Q-values while effectively alleviating the issue of over-constraint.
D4RL benchmarks. In the main paper, we evaluate SQOG on the D4RL Gym-Mujoco task (Fu et all]]2022 2021), which contains three environments (halfcheetah, hopper, walker2d), and five types of datasets (random, medium, medium-replay, medium-expert, expert). Random datasets are gathered by random policy. Medium is generated by first training policy online using Soft Cettr-Critic (Haarnoja et al. 2018), carly-stopping the training, and collecting 1M samples from this partially- trained policy. Medium-replay datasets are collected during the training process of the "medium" SAC policy. Medium-exper datasets are formed by combining the suboptimal samples and the expert samples. Expert datasets are made up of expert trajectories.
D4RL offers a metric called normalized score to evaluate the performance the offline RL algorithm, which is calculated by:
score random score
normalized score 100 expert score random score
(31)
If the normalized score equals to 0, that indicates that the learned policy has a similar performance as the random policy, while 100 corresponds to an expert policy. The final results are obtained using a standard evaluation procedure. Specifically, for each experiment, calculate the average normalized score over the last ten evaluation episodes. We then compute the mean and standard deviation of these scores across different random seeds.
23

---

Published as a conference paper at ICLR 2025
Table 8: Experimental hyperparameters of SQOG.
Hyperparameter
Value
Optimizer
Adam (Kingma & Ba 2015
Actor network learning rate
x 10
Critic network learning rate
3 x 10
Discount factor y
0.99
Total training step T
x 10
Policy noise clipping parameter
0.5
Mini-batch size
256
Target network smoothing parameter 7
0.005
Actor network update frequency m
a (TD3+BC (Fujimoto & Gu 2021) Hyperparameter)
150 (25 for Adroit tasks)
B
0.5 (2.5 for Adroit tasks)
Noise type
Normal distribution
Noise scale
0.6
Noise clip
0.5
Hyperparameters. All experimental hyperparameters of SQOG are documented in Table The hyperparameter a of TD3+BC governs the efficacy of the behavior cloning constraint, while in our algorithm, introduce 3 to regulate the strength of OOD generalization. Notably, trade-off exists between a and 33, where a larger 3 corresponds to stronger generalization, while smaller a implies more stringent constraints. Through empirical exploration, that yield favorable performance: (150, 0.5) for Mujoco (excluding saffcheetah-mediium-eee and Maze2d tasks, and (25, 2.5) for Adroit tasks and wwwfchheetahmmddu--repeeee
identify two sets of hyperparameters
in Mujoco. The Gaussian distribution is good choice for dataset noise in OOD sampling, but there could be better alternatives such pink noise (Eberhard et al. 2023). which is left for future work. We did not finely tune the noise scale and noise clip parameters, leaving room for improvement in their optimization.
C ADDITIONAL DISCUSSION ON CHN
Why the state space needs to be included in the CHN definition? We study the Q-function in this paper, which is also called the state-action value function. The state is necessary when discussing OOD actions, as the state-action pair defines the context in which the action is taken.
An alternative definition of the CHN can be given follows:
Definition 3. Given state the dataset DD let Conu, (D) denotes the convex hull of the in-sample actions corresponding to the state 8, Conv, = =Ei >jaz(A 2 0, E A = (s, E {a E A A &
D}. Define the external neighborhood of the convex hull as: Ns(Conv, (D) Conv, (D), ming'eCone (D) I|a' all S ra where T is a radius chosen to be less than or equal to the diameter of Conv, (D). Finally, the CHN can be defined as: CHN Ctaciiion (D) = USe where (D),(D) Conv,(D) U N₂ (Conw,(D)).
While the alternative definition provided above introduces more detailed structures, it is less elegant and concise compared to the definition of the CHN in the main text. Furthermore, the alternative definition may lack connectivity, as the CHN, for different states may be disjoint, failing to form a cohesive and unified region. Importantly, the alternative Definition is encompassed within the broader Definition provided in the main text. (Given state E D, Va E A. if a E CHN,(D) then (s, a) E CHN(D).) The original definition encompasses a broader region and enables us to offer more comprehensive safety guarantees due to its more general formulation.
D CONNECTION BETWEEN THEORY AND PRACTICE
In theory, introduce the CHN and the Smooth Bellman Operator (SBO). In practice, design SQOG based on the insights from CHN and SBO. The connection between SBO and SQOG's critic loss is summarized in Table 9 For simplicity, we MSE (mean squared error) represent the
24

---

Published as a conference paper at ICLR 2025
loss and omit the expectation operator for readability. The complete definition of SBO is provided in Definition 2
while the detailed formulation of SQOG's critic loss can be found in Eq. 113 and (14.
Table 9: Connections between SBO and SQOG's critic loss.
Connections
In-sample action (a)
OOD action within CHN (aood)
SBO
Q a) + -r y (Q
Q(s, aood) + Q(s, a (neighbor)
MSE (o((s,a), (r 22 (s', (s', ( MSE QQo(s, aood), Q(3, a
Loss
(neighbor)
For the Q-values of ia-sample actions, SBO applies the base Bellman operator B5. Similarly, in practice, SQOG samples batches of data from the dataset, and the first term of its critic loss updates the Q-values of in-sample actions using an empirical Bellman operator. This closely aligns with the in-sample component of SBO.
For the Q-values of OOD actions within the CHN, SBO uses the smooth generalization operator G to generalize these OOD actions to their in-sample neighbors. Correspondingly, in practice, SQOG actively generates OOD actions from in-sample actions and trains their Q-values by leveraging the Q-values of their neighboring in-sample actions. This approach aligns well with the OOD component of SBO, enabling smooth generalization for OOD Q- Q-aauuss.
However, there is difference. In SBO, if the policy's 's action (s') ) is identified as OOD, Oj updates its Q-value, Q(s', (s)).. based on the Q-value of neighboring na-aamll action, Q(s', neighbor)) an
In practice, however, we leave (s' (s((mT= unchanged, as accurately determining whether an action OOD challenging. Nevertheless, as the OOD Q-values are iteratively trained through the OG loss term, we expect Qe (s', Th (s')) to become increasingly accurate over time, mitigating the overestimation and underestimation issue of Qa TT(6") as training progresses (empirically shown in Appendix (B.2).
Additionally, the BC loss applied to the actor network keeps the learned policy not far from the behavior policy. This ensures that the actions TT(s') generated from To are likely to remain close to the dataset. Consequently, Qe (s', (s',TT(s')) TT(s') is less prone to overestimation. further stabilizing the learning process.
Finally, the role of CHN is critical in providing safety guarantees and guiding the selection of practicable OOD regions. Without CHN, would be challenging to identify feasible OOD regions for generalization. Attempting to handle all OOD regions indiscriminately could result in degraded performance or instability. Similarly, without the theoretical analysis provided by SBO, it would be challenging conceive the idea of detaching the gradient of in-sample Q-values, which is essential for constructing meaningful targets for OOD Q-value updates.
E
THE RELATIONSHIP BETWEEN SBO AND BC LOSS
Our main contribution lies in alleviating the over-constraint issue by generalizing Q-function esti- mation to OOD regions within the CHN. While "pessimism" or "constraint" is critical for policy improvement in offline RI., excessive constraints can lead to inaccurate Q-value estimation and suboptimal performance. SQOG mitigates this issue by enabling more accurate Q-value estimation, which directly benefits the policy improvement step.
To demonstrate this, we adopt TD3+BC as baseline. However, we emphasize that SQOG's effectiveness is not limited to this specific framework or the use of behavior cloning (BC) loss.
Theoretical justification. As shown in Theorem I SBO approximates accurate OOD Q-values within CHN under the assumption that the learned policy π is close to the behavior policy μ. This assumption KL(H, r)) < c. Therefore, the conclusion that SBO enables gradual approximation of true OOD Q-values within CHN is generalizable to other policy constraint methods beyond TD3+BC.
is satisfied by any policy constraint offline RL algorithm that enforces max max(KL (π, n)),
Empirical validation. To demonstrate SBO's generalizability, we replace the BC loss in TD3+BC with the KI. divergence penalty used in BRAC's actor loss (Wu al. 2019). Since the true behavior
25

---

Published as a conference paper at ICLR 2025
Table 10: Normalized average comparison of BRAC+SBO against baseline method BRAC on Mujoco "-v2" and Adroit "-v1" datasets. The results are averaged over 4 different random seeds. We bold the highest scores.
Dataset
BRAC
BRAC+SBO
halfchectah-men mediim
19.
54.3=1.2
hopper-medium
3.6:3.11
90.9=2.9
walker2d-medium
7.8+8.1
85.6==.33
halfcheetah-
halfchhetaa-mmeiium-reeg medium-r-aamuuaa
41.8:16.2
47.8=2.0
hopper-meddium-reply
28.8+420.3
61.1
walker2ddmeedum--rppay
8.5+3..0
67.6+11.0
Mujoco Average
23.4
67.9
Improvement
190.2%
pen-human pen-cloned
19.2+-16.3
69.7
28.44:3.4
69.0
Adroit Average
23.8
69.4
Improvement
191.6%
policy is inaccessible, adopt behavior cloning (a common and simple method) approximate it. We then re-imm immpleeett BRAC and add SBO to it. Table presents the performance of BRAC+SBO across various datasets. We observe significant performance improvement when SBO added to BRAC, confirming that SBO is a versatile plug-in for policy constraint methods.
Although BRAC+SBO performs well, its performance does not surpass that of SQOG. This is likely because behavior cloning only provides an approximate behavior policy, and inaccuracies in this approximation weaken the KL penalty's ability ensure и is close to μ. which may hurt the effectiveness of SBO. While advanced methods like generative models could improve hehavior policy estimation, they are computationally expensive. contrast, TD3+BC with BC loss strikes favorable balance between simplicity, computational efficiency, and effectiveness.
In summary, SBO is not restricted to TD3+BC BC loss. Our empirical results demonstrate its potential to generalize across other policy constraint methods, addressing the over-constraint issue and achieving superior performance.
Further discussion on the contribution to policy constraint methods. Policy constraint methods, such as TD3+BC (Fujimoto & Gu 2021], aim to align the learned policy with the behavior policy to mitigate Q-value overestimation. However, these methods often result in Q-value underestimation due to limited exploration beyond the behavior policy, leading to the neglect of OOD actions that might correspond to higher Qavalus. In contrast, SBO mitigates both Q-value underestimation and overestimation. particularly in OOD regions, as proven in Theorem 3 This improvement allows the policy to make more informed decisions by considering OOD actions with higher Q-values, addressing key limitation of policy constraint methods. This capability is the primary driver behind the observed experimental improvements.
Furthermore, addressing the offline RL community context, the over-constraint issue remains significant challenge for policy constraint methods. Theoretical analysis and experiments often neglect the "value part" in critic network, as it risks conflicting with policy constraints in actor network. However, SBO serves as a valuable complement policy constraint methods. On one hand, SBO overcomes this bottleneck by accurately learning OOD Q-values for policy improvement and exploration (proved Theorem 3 shown the sanity check and Appendix [..2). On the other hand, SBO is really smooth when integrating into those policy constraint methods, primarily because treat policy constraints as a precondition (as demonstrated by the significant improvements in Tables and 10 We firmly believe that SBO (SQOG) is valuable supplement, combining theoretical ngor, strong performance, and computational efficiency.
Finally, while prior value-basee based offline RL methods often treat OOD regions inherently risky due to information deficiencies (Xu et al. 2023) Kumarr et al.) 2020; Kostrikov et al., 2021b). Methods like MCQ (Lyu et al. 2022) explore mild but enough conservatism for offline learning while not
26

---

Published as a conference paper at ICLR 2025
harming generalization. Building this direction, work demonstrates the insight that extending Q-function generalization to OOD regions within CHN can be beneficial. We believe this approach will attract more attention to Q-value estimation in OOD regions and offer new insights for the offline RL community.
27


Reviews:
[{"review_id": "PnaCuvU01f", "reviewer": "Reviewer_Fqh7", "summary": "This paper introduces the method of using the convex hull and a smooth Bellman operator to solve the challenge of over-conservative behaviors in offline reinforcement learning algorithms. \nBoth theoretical analysis and empirical justifications of the proposed method are provided.\nThe results look promising on the reported tasks.", "questions": "Limitations and potential pitfalls of the proposed method are not discussed. Could the authors explain more about the potential challenges of the proposed method? e.g., how do the choices of distance metric affect the results? what would be the trade-off between generalization and conservative behaviors? Is there any task that would benefit more from a conservative estimation or more aggressive extrapolation? Having analyses of some case studies would greatly improve the clarity and impact of the paper.\n\nHow does the proposed method scale with different numbers of training samples (i.e., such that convex hulls differ)? Could the authors provide some analysis of the computational cost (time/memory usage), and compare different methods in those settings?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well written and the structure is clear. The idea is supported by both theoretical justification and relatively sufficient empirical study.\n\n\n----\n\nThe authors made a great effort in the rebuttal process and addressed all my concerns.", "weaknesses": "Using convex hulls and nearest neighbors in reinforcement learning is not a new technique, yet the current paper does not discuss those methods. Giving credit to the existing work will not hurt the novelty of this paper but will help readers have a clearer understanding of the advancements in the field.\n\n[1] Sun, Hao, et al. \"Accountability in offline reinforcement learning: Explaining decisions with a corpus of examples.\" Advances in Neural Information Processing Systems 37 (2023).\n\n[2] Lyu, Jiafei, et al. \"SEABO: A Simple Search-Based Method for Offline Imitation Learning.\" arXiv preprint arXiv:2402.03807 (2024).\n\n\nFigure 3 (the table) is not very clear. The authors may want to re-organize the information in a different format.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wHeqzLEPHF", "reviewer": "Reviewer_cDBr", "summary": "This paper addresses the problem of generalization in offline RL. Most existing techniques tend to severely constrain the policy to produce actions that remain close to the data distribution, to avoid known nefarious effects of OOD generalization in unexplored parts of the state-action space. This paper suggests that these constraints can be alleviated. In particular, they identify a region close to the dataset (the CHN) where controlled estimation of the Q-values is possible, whereas this region is usually treated as completely OOD by existing techniques. After deriving a variant of the Bellman operator that leverages the existence of this region and showing some of its properties, the authors propose a concrete regularization term to add to the classic critic loss in offline RL. The authors first showcase the existence of the over-constraining problem on a classic offline RL algorithm, and that their method alleviates this issue. They then evaluate their method on the D4RL benchmark, showing improved performance compared to baselines while having a quick runtime.", "questions": "- Can you please precise the different definitions, most importantly of CHN? I believe there is a mistake in its current form. Please also confirm what $(S,A)_D$ is, as it is not properly defined (in mathematical terms).\n- Could you please write a mathematical definition that confirms the notion of OOD that was chosen in this work? Is it the one of Definition 2?\n- Could you comment on the link between the SBO and the OG loss of Eq.13? Are these two objectives equivalent? If not, what are the differences? What is captured by one but not the other?\n- How did you decide on the scale (and the clip) of the noise distribution, indicated at p.19? Are they sensible hyperparameters for the performance of the algorithm?\n- Regarding Eq. 14: for s’ that are at the “boundary” of D, and a’ potentially far from the dataset: the OOD backprop effect will not be countered by the new smoothing term, is that correct? \n- $a_{ood}$ seems to implicitly belong to CHN in Prop. 3. $a_{ood}$ was introduced in the main text of Sec 3.2, was it already part of CHN then or could it also be outside of CHN?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- First, I find that the problem tackled is important and very well motivated. I found in particular that the introduction did a great job of exposing to the reader the problem of over-constraining the policy. The simple but logical explanation is rooted in the design of existing policies that completely avoid to generalize OOD. Having described the problem, the authors are then clear in their ambition: leveraging the part of the state space where neural networks actually are able to generalize to improve the approximation of Q, and possibly of the learned policy. I think this idea is very sensible, and sounds like a natural and promising follow-up to the existing literature following the introduction. I expect a work in this direction to be very interesting.\n- The practical algorithm is simple to implement, as it is just adding a smoothing term to the loss. This is a strong point of the empirical part of the proposed approach, since practitioners can easily add this term to their training loss. This also makes the algorithm very quick to run, which is a great advantage compared to certain competitors (such as MCQ) that are much slower.\n- I find the sanity check (in particular Fig 1.b) to be a very convincing illustration of the problem that motivated the paper. The authors show how a classic baseline, TD3+BC, tends to keep its high Q-value predictions close to the regions of high data density. I find this is a great illustration of the over-constraining problem. At the same time, this figure shows that SQOG does not suffer from the same limitations.\n- The ablation study illustrated in Fig. 4 was a welcome addition that shows the sensitivity of the method to one of its main hyperparameters. \n- Finally, the algorithm seems to perform well according to the results in Table 1. In Fig. 1, the authors show that their method is indeed more robust to OOD and approximates well the true action values, while TD3+BC tends to keep its predictions well within the data distribution. This is consistent with the story of the paper.", "weaknesses": "Unfortunately, I found that the paper lacked clarity and mathematical rigor at several crucial moments, which severely hurt my understanding of the contribution.\n- Most importantly, I believe there is a mistake with the definition of perhaps the most central object of the paper, CHN. As it is defined, by construction, $N(Conv(D)) \\subset (S,A)_D \\subset Conv(D)$, therefore $CHN(D) = Conv(D)$. It is not clear how to change the existing definition to get to the one that seems intended by the authors (on this note, a figure of $(S,A)_D$, $Conv(D)$, and $N(Conv(D)$) would have been enlightening, I encourage you to add one). \n- In Definition 1, the definition of $(S,A)_D$ is confusing: it is mentioned to be a space, using the space notation of the preliminaries, but it seems from the rest of the definition that it is the set of state-action tuples from D. Moreover, r is named as the neighborhood radius and seems like a free parameter to set, but is then defined in Proposition 1 as a function of D and its convex hull: what definition should we follow?\n- The definition of OOD was another major point of confusion to me. It is indicated that actions that are OOD are actions that are outside the support of the behavior policy $\\mu$ (which is restated in Definition 2). But as far as I can tell there are no constraints on the behavior policy $\\mu$ that generated the dataset. What if this policy put a non-zero probability on every action? This is far from an edge case, since a behavior policy parametrized as a Gaussian distribution would have support on the full action space. Then, what is the relationship with D? The text pushes me to think that what is OOD is what is far enough from D, but within the CHN (L153): then is OOD what is far from the empirical distribution derived from D? To illustrate the confusion, the authors write at L317 that the [-0.5, 0.5] region in Fig.1 is OOD, but the density is clearly non-zero for almost all of this range, contradicting how OOD was defined earlier. Overall, my impression is that two notions of OOD are colliding in this paper: the formal one from Definition 2 stating that every action outside of the support of $\\mu$ is OOD, and the intuitive one from Figure 1 that regions of low-density are OOD.\n- The confusion inherited from the unclear definitions of the spaces of Sec 3.1 is transfered to Sec 3.2. For instance, in Definition 2, the authors refer to “neighborhood”. why not use a precise mathematical object that you defined earlier, likely in Definition 1? On that note, the neighborhood was defined w.r.t state-action tuples in Definition 1, but now refers to the action only, which does not help with the clarity of exposition. If that is normal, then I think it would be useful to comment on why the state space needs to be included in the CHN definition. As a consequence, neighbor/neighborhood is used several times (L170/175/214) but since it is not clear to me where the exact definition is, it is difficult to follow the text more than at an intuitive level.\n\n_Other weaknesses_\n- I find that a discussion of the link between the SBO and the proposed loss would have been helpful for the flow of the paper. As it stands, Section 3.2 introduces the SBO and some of its properties, while Sec 3.3 switches direction and proposes a regularization term, a bit out of the blue. I think a smoother transition between the two section, justifying the key elements to keep of the SBO, and how they motivated the new regularization term, would have helped the exposition.\n- One of the key aspects of the empirical loss is the noise added to the in sample action to possibly get an artificial OOD sample. The authors did an ablation on the type of noise, but I would have been curious to see a discussion on the scale of the noise instead, as this should control the amount of “OOD-ness” and might influence the performance of the algorithm.\n- In proposition 1, the authors refer to the NTK regime without explaining the acronym nor introducing the general framework. I think the framework should be at least introduced.\n- I am also unclear about the last sentence of the paragraph preceding proposition 2 (l. 138-139): how did you derive this statement?\n- The norm used in all mathematical statements besides Theorem 4 is unclear. \n- at L170, the authors write “the dataset action […]”: this assumes that this action exists and that it is unique, but it is not clear to me if this is true.\n\n_Minor weaknesses that did not affect the score but could be addressed anyway_\n- A suggestion: Fig. 1 (1.b in particular) does a great job of illustrating the overconstraint issue of TD3+BC. It would be interesting to consider showing this figure earlier in the paper. I am OK with where the figure is right now and this did not impact my score, but I think this could convince the reader very early on that the problem you are tackling is real and substantial.\n- In the experiments, I found confusion the exact definition of the true Q values (L314): are you using Monte Carlo estimation of a (s,a) pair, resetting the environment at this state? Similarly, what do you mean exactly by standardized and smoothed (L315)?\n- Besides the comments above, I find that the mathematical notation was imprecise or inconsistent at a couple other spots. For instance, replacing “noise” in Eq. 14 by an actual variable; the action $a^{in}_{neighbor}$ is indicated sampled from a support (L170); the expectations are taken w.r.t $\\pi$ (L.85) or $\\pi(\\cdot | s’)$. \n- “Longest diameter” -> you can drop the “longest”, diameter is already the supremum of the distances within the set.\n- L135: “Due to the uniqueness of CHN”: this fact is never stated before, but since it is referred to like this, I would have expected to have a sentence stating that the CHN is unique (maybe at L115-116) with the other properties.\n- In the appendix, you indicate in the proof “we can easily derive” (L744), but in a proof I would expect for this derivation to be there.\n- The authors ran several algorithms on several algorithms, but unfortunately on a rather small number of seeds (4). A higher number of seeds could have gotten these confidence interval smaller and non intersecting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PgHVgm0pJC", "reviewer": "Reviewer_etFc", "summary": "This paper introduces the Smooth Bellman Operator (SBO), aimed at addressing the over-constraint issues prevalent in popular offline RL algorithms. Building on SBO, the authors propose SQOG, a practical algorithm that demonstrates superior performance over well-known baselines like CQL, IQL, and MCQ, on D4RL benchmark. The paper also includes ablation studies examining the impact of the hyperparameter \\beta, which controls the out-of-distribution (OOD) generalization loss, and different types of noise. Additionally, the paper provides theoretical analysis that supports the efficacy of the SBO.", "questions": "See above", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow.\n2. SQOG achieves strong performance on the D4RL benchmark, remaining relatively simple, straightforward, and computationally efficient.\n3. The sanity check on the Inverted Double Pendulum task empirically shows SBO’s effectiveness in alleviating the over-constraint issue, which is a valuable addition.", "weaknesses": "1. My key concern is the lack of discussion on the relationship between SBO and behavior cloning loss. While the paper claims that SBO alleviates the over-constraint issue in existing offline methods, it’s unclear to me that whether SBO’s effectiveness is limited to use with TD3+BC. Further discussion on this point would be beneficial.\n2. The Inverted Double Pendulum task is relatively simple and differs from locomotion tasks, raising questions about whether SQOG can maintain accurate Q-value estimates in higher-dimensional environments. While visualizing Q-values in high-dimensional spaces may not be as straightforward as with the Inverted Double Pendulum, including experiments on a locomotion task to validate SQOG’s Q-estimation accuracy would strengthen this claim.\n3. All experiments were conducted with only 4 seeds; additional seeds would improve result robustness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LFU5W6O7v3", "reviewer": "Reviewer_CHBA", "summary": "This paper introduces a novel offline RL algorithm, Smooth Q-function OOD Generalization (SQOG), aimed at improving Q-value estimation by enhancing the generalization of the Q-function within the Convex Hull and its Neighborhood (CHN). The authors provide theoretical guarantees, showing that the proposed algorithm approximates true Q-values for both in-sample and out-of-distribution (OOD) actions. On the D4RL benchmarks, SQOG demonstrates superior performance compared to state-of-the-art methods.", "questions": "Q1: In Figure 1, I am curious about how to calculate the true Q values for action in the range $[0,1]$. Also, it would be more appropriate to refer to these as \"ground truth\" Q-values. Furthermore, the \"true Q-values\" are shown as a smooth line due to the use of neural network function approximation. However, as mentioned earlier, the true Q function (which is not precisely known in the MuJoCo environment) could be quite irregular or even stepwise.  This makes the toy example less convincing in demonstrating the accuracy of the approximation. \n\nQ2: In the conclusion, the authors state that solving the CHN and identifying all OOD regions within it is both impractical and unnecessary. While I partially agree, I also hold a different viewpoint. The authors mention that one advantage of CHN is its safety guarantees, as measured by the neighborhood radius. I noticed that in the code, the noise is clipped within the range [−0.5,0.5]. Have the authors experimented with tuning this parameter? If so, providing more experimental details would be beneficial, as the clipped range directly affects the neighborhood radius associated with the safety guarantees.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper is well-organized. The proposed algorithm SQOG significantly reduces computational costs compared to existing algorithms.", "weaknesses": "- **Novelty:** The proposed SQOG method builds on the concepts of the Convex Hull and its Neighborhood (CHN) and the Smooth Bellman Operator (SBO). The use of CHN was first introduced in DOGE [1], and SBO for in-sample Q-value updates is a common practice in offline RL. These techniques are not new, which makes the contribution appear incremental. It would be better to clarify the difference or advantage of these techniques compared to existing methods.\n- **Neighborhood**: While the introduction of the neighborhood is intended to alleviate the over-constraining issue of OOD actions, it is unclear how this concept is reflected in Theorem 1 and informs practical implementation. Additionally, although the authors theoretically demonstrate that SQOG can achieve good OOD evaluation, the underlying assumptions—such as the continuity of the Q-function—are idealistic. For example, in sparse reward environments like Montezuma's Revenge, where only a few actions yield rewards, the Q-function is likely to be discontinuous or stepwise.\n- **Connection between theory and practice:** There appears to be a disconnect between the theoretical and experimental sections. The theory suggests using CHN and SBO, but even without these components, a researcher familiar with offline RL could still design an algorithm based on principles like pessimism and in-sample updates. The writing should better emphasize the unique advantages of CHN to make its value more apparent.\n- **The quality of the dataset:** In my opinion, CHN essentially augments the data by adding state-action pairs near the original dataset, making the quality of the dataset crucial for performance. In high-dimensional state and action spaces, the main challenge is often limited sample size. It would help to discuss this issue further or use toy examples to illustrate the benefits of CHN.\n\n[1] When data geometry meets deep function: Generalizing offline reinforcement learning. ICLR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)